Skip to content

BUG: resolve divide by 0 error when uploading empty dataframe #252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Feb 26, 2019
Merged

BUG: resolve divide by 0 error when uploading empty dataframe #252

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
42ccab2
resolve divide by 0 error when uploading empty dataframe
bwanglzu Feb 22, 2019
3864d46
reformat with black
bwanglzu Feb 22, 2019
272aa7b
add unit test when uploading empty dataframe
bwanglzu Feb 22, 2019
8bfde50
add empty data upload system test
Feb 23, 2019
59a7950
remove empty df unit test
Feb 23, 2019
87ae125
update empty df
Feb 23, 2019
02c897a
add 0.10.0 release note
bwanglzu Feb 25, 2019
a9ec115
Merge branch 'bug/divide-by-zero' of https://github.com/bwanglzu/pand…
bwanglzu Feb 25, 2019
922707c
Merge branch 'master' into bug/divide-by-zero
bwanglzu Feb 25, 2019
1fe7c13
update release note version number to 0.11.0
bwanglzu Feb 25, 2019
d62c36e
update empty dataframe bug fix in change log
bwanglzu Feb 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/source/changelog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
Changelog
=========

.. _changelog-0.11.0:

0.11.0 / 2019-02-25
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should still be in 0.10.0!

------------------

- This fixes a bug where pandas-gbq could not upload an empty database. (:issue:`237`)


.. _changelog-0.10.0:

0.10.0 / TBD
Expand Down
4 changes: 2 additions & 2 deletions pandas_gbq/gbq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -518,8 +518,8 @@ def load_data(
chunks = tqdm.tqdm(chunks)
for remaining_rows in chunks:
logger.info(
"\rLoad is {0}% Complete".format(
((total_rows - remaining_rows) * 100) / total_rows
"\r{} out of {} rows loaded.".format(
total_rows - remaining_rows, total_rows
)
)
except self.http_error as ex:
Expand Down
22 changes: 22 additions & 0 deletions tests/system/test_gbq.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -924,6 +924,28 @@ def test_upload_data(self, project_id):
)
assert result["num_rows"][0] == test_size

def test_upload_empty_data(self, project_id):
test_id = "data_with_0_rows"
test_size = 0
df = DataFrame()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we might have an additional problem when the DataFrame contains no columns.

In the conda build (https://circleci.com/gh/tswast/pandas-gbq/276) I'm getting:

E           google.api_core.exceptions.BadRequest: 400 POST https://www.googleapis.com/upload/bigquery/v2/projects/pandas-gbq-tests/jobs?uploadType=resumable: Empty schema specified for the load job. Please specify a schema that describes the data being loaded.

Since we still create a table in pandas-gbq before running the load job, we can probably avoid doing the load job altogether when a DataFrame contains no rows.


gbq.to_gbq(
df,
self.destination_table + test_id,
project_id,
credentials=self.credentials,
)

result = gbq.read_gbq(
"SELECT COUNT(*) AS num_rows FROM {0}".format(
self.destination_table + test_id
),
project_id=project_id,
credentials=self.credentials,
dialect="legacy",
)
assert result["num_rows"][0] == test_size

def test_upload_data_if_table_exists_fail(self, project_id):
test_id = "2"
test_size = 10
Expand Down