Skip to content

docs: add samples from bigquery/cloud-client #174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 103 commits into from
Jul 22, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
fdb0218
Add veneer samples for BigQuery. [(#444)](https://github.com/GoogleCl…
tswast Aug 10, 2016
3f11046
Add data loading samples
Aug 18, 2016
7579980
Add bigquery snippets
Aug 19, 2016
14eb646
Remove unnecessary fixture
Aug 19, 2016
63c97fa
Address review comments
Aug 19, 2016
8f2a7fb
Fix lint
Aug 19, 2016
dd58710
Address post-review comments for BigQuery samples
Aug 22, 2016
63b1b0a
Add bigquery create and copy table examples [(#514)](https://github.c…
Sep 12, 2016
9dd12c1
Auto-update dependencies. [(#540)](https://github.com/GoogleCloudPlat…
dpebot Sep 23, 2016
74d55b4
Auto-update dependencies. [(#542)](https://github.com/GoogleCloudPlat…
dpebot Sep 26, 2016
e85cb76
Move to google-cloud [(#544)](https://github.com/GoogleCloudPlatform/…
Sep 30, 2016
4e64955
Add new "quickstart" samples [(#547)](https://github.com/GoogleCloudP…
jmdobry Oct 5, 2016
6ce03fc
BigQuery - use new QueryJob.results() method. [(#555)](https://github…
tswast Oct 5, 2016
2deb40e
Quickstart tests [(#569)](https://github.com/GoogleCloudPlatform/pyth…
Oct 12, 2016
644bfec
Generate readmes for most service samples [(#599)](https://github.com…
Oct 24, 2016
e1f91be
Generate most non-appengine readmes
Oct 25, 2016
a65e7d9
Add missing bigquery samples [(#622)](https://github.com/GoogleCloudP…
Oct 27, 2016
1375731
Update samples to support latest Google Cloud Python [(#656)](https:/…
Nov 15, 2016
d63164c
Update readmes
Nov 15, 2016
cd24127
Fix bigquery load from file sample
Nov 30, 2016
c594be1
Fix lint
Nov 30, 2016
261033f
Adjusted error handling based on Googler feedback [(#693)](https://gi…
ryanmats Nov 30, 2016
f566774
Changed error handling for all other big query samples as well [(#694…
ryanmats Nov 30, 2016
442652b
Add bigquery shakespeare example [(#604)](https://github.com/GoogleCl…
Dec 6, 2016
9c63f3f
BigQuery parameterized query sample [(#699)](https://github.com/Googl…
tswast Dec 6, 2016
b1b2697
Auto-update dependencies. [(#715)](https://github.com/GoogleCloudPlat…
dpebot Dec 13, 2016
4409c77
Remove cloud config fixture [(#887)](https://github.com/GoogleCloudPl…
Apr 4, 2017
028699b
Remove resource [(#890)](https://github.com/GoogleCloudPlatform/pytho…
Apr 5, 2017
4558c6b
BigQuery: named parameter query samples
tswast Dec 19, 2016
130fdb7
Rename query file and use async method.
tswast Apr 21, 2017
4ae86d6
Indent queries.
tswast Apr 24, 2017
e34d16a
Auto-update dependencies. [(#916)](https://github.com/GoogleCloudPlat…
dpebot Apr 25, 2017
b9d9429
Re-generate all readmes
Apr 27, 2017
4494da5
BigQuery: user credentials to run a query. [(#925)](https://github.co…
tswast May 3, 2017
093a11f
Auto-update dependencies. [(#942)](https://github.com/GoogleCloudPlat…
dpebot May 12, 2017
d3b5681
Fix README rst links [(#962)](https://github.com/GoogleCloudPlatform/…
waprin May 24, 2017
b230508
Auto-update dependencies. [(#1004)](https://github.com/GoogleCloudPla…
dpebot Jun 27, 2017
40ba003
BigQuery: add auth samples for service accounts.
tswast May 1, 2017
5527c56
Fix SQL query to return 10 rows & simplify [(#1041)](https://github.c…
PicardParis Jul 31, 2017
29768c4
Fix test randomness [(#1043)](https://github.com/GoogleCloudPlatform/…
PicardParis Aug 1, 2017
193e449
Fix test randomness [(#1044)](https://github.com/GoogleCloudPlatform/…
PicardParis Aug 1, 2017
c19e0d0
Auto-update dependencies. [(#1055)](https://github.com/GoogleCloudPla…
dpebot Aug 7, 2017
bd74d23
Update readme references. Resolves 1059
Aug 8, 2017
e81f489
Use futures API to wait for jobs to finish.
tswast Aug 8, 2017
de6ba2c
BQ: Use futures API for quickstart.
tswast Aug 8, 2017
554a1f9
BigQuery : max_results changes page size not full list size
tswast Aug 21, 2017
5f13468
Remove max_results argument from table.fetch_data() call.
tswast Aug 22, 2017
7b4e489
Remove unnecessary list() call
tswast Aug 22, 2017
cce95c8
Auto-update dependencies. [(#1093)](https://github.com/GoogleCloudPla…
dpebot Aug 29, 2017
a6a1e16
BigQuery : add sample for writing query results to a destination tabl…
tswast Sep 19, 2017
f523eea
Auto-update dependencies. [(#1128)](https://github.com/GoogleCloudPla…
dpebot Sep 20, 2017
95c1438
BigQuery: Updates samples for BigQuery Beta 2 (do not merge until rel…
alixhami Oct 31, 2017
45f1ad7
Auto-update dependencies. [(#1186)](https://github.com/GoogleCloudPla…
dpebot Nov 1, 2017
8e9adfd
Auto-update dependencies. [(#1227)](https://github.com/GoogleCloudPla…
dpebot Nov 16, 2017
7f3eda1
/s/buckets/datasets in BigQuery auth sample [(#1242)](https://github.…
tswast Dec 1, 2017
b4c7856
Fix broken link to core Client service account helper. [(#1256)](http…
tswast Dec 7, 2017
a8fc6a2
Added "Open in Cloud Shell" buttons to README files [(#1254)](https:/…
Dec 7, 2017
ab25746
Move imports into bigquery samples
Dec 11, 2017
4c9b751
BigQuery: rewrite simple app tutorial.
tswast Dec 2, 2017
52eac30
Auto-update dependencies. [(#1272)](https://github.com/GoogleCloudPla…
dpebot Dec 12, 2017
75f2aeb
Auto-update dependencies. [(#1307)](https://github.com/GoogleCloudPla…
dpebot Jan 9, 2018
57e88ba
Auto-update dependencies. [(#1355)](https://github.com/GoogleCloudPla…
dpebot Feb 9, 2018
617cb29
Auto-update dependencies. [(#1377)](https://github.com/GoogleCloudPla…
dpebot Mar 5, 2018
5612ae1
Regenerate the README files and fix the Open in Cloud Shell link for …
Apr 7, 2018
9f8d118
BigQuery: Remove unused samples [(#1455)](https://github.com/GoogleCl…
alixhami Apr 17, 2018
c5418a7
BigQuery: Removes unused API samples [(#1459)](https://github.com/Goo…
alixhami Apr 18, 2018
d9f336d
Update READMEs to fix numbering and add git clone [(#1464)](https://g…
frankyn Apr 26, 2018
f50b0d2
BigQuery: Moves BigQuery tutorial for Dataproc to python-docs-samples…
alixhami May 18, 2018
3a79a38
BigQuery: Adds authorized view tutorial [(#1535)](https://github.com/…
alixhami Jun 21, 2018
13ae53a
Region tag end typo [(#1536)](https://github.com/GoogleCloudPlatform/…
alixhami Jun 21, 2018
f572de4
Code snippets for Jupyter data visualization tutorial [(#1560)](https…
alixhami Jul 12, 2018
cec846e
updates last query per Steve's feedback [(#1578)](https://github.com/…
alixhami Jul 17, 2018
0c0d2e0
updates tutorial to add datasets to to_delete before running [(#1579)…
alixhami Jul 17, 2018
3b63b31
Remove ipython Python 2 modifier from requirements.txt [(#1675)](http…
tswast Aug 31, 2018
5a4dfdd
Auto-update dependencies. [(#1846)](https://github.com/GoogleCloudPla…
dpebot Nov 20, 2018
1aec4eb
Refactor BQ user credentials sample to use region tags [(#1952)](http…
tswast Jan 2, 2019
81d66e4
Make TODO clearer what action to take. [(#1963)](https://github.com/G…
tswast Jan 8, 2019
ba622f7
Add missing explanation about local_server callback URL
tswast Jan 9, 2019
e8d212b
Auto-update dependencies. [(#1980)](https://github.com/GoogleCloudPla…
dpebot Feb 6, 2019
b906c86
BigQuery: Add sample for explicitly creating client from service acco…
tswast Aug 5, 2019
9ca1bc7
Adds updates including compute [(#2436)](https://github.com/GoogleClo…
gguuss Oct 23, 2019
340c4fc
Simplify noxfile setup. [(#2806)](https://github.com/GoogleCloudPlatf…
kurtisvg Apr 2, 2020
706a39a
fix: address issues with concurrent BigQuery tests [(#3426)](https://…
shollyman Apr 17, 2020
ed50bd9
chore(deps): update dependency pytz to v2019.3 [(#3190)](https://gith…
renovate-bot Apr 18, 2020
e49124d
chore(deps): update dependency matplotlib to v3.2.1 [(#3180)](https:/…
renovate-bot Apr 18, 2020
e605e38
chore(deps): update dependency ipython to v7.13.0 [(#3177)](https://g…
renovate-bot Apr 18, 2020
18a2346
chore(deps): update dependency google-cloud-bigquery to v1.24.0 [(#30…
renovate-bot Apr 18, 2020
f4ad717
chore(deps): update dependency pytz to v2020 [(#3566)](https://github…
renovate-bot Apr 28, 2020
0e31768
chore(deps): update dependency ipython to v7.14.0 [(#3670)](https://g…
renovate-bot May 4, 2020
c1c87bd
[bigquery] opt in to use the build specific projects [(#3883)](https:…
May 26, 2020
239ee4f
chore(deps): update dependency ipython to v7.15.0 [(#3917)](https://g…
renovate-bot May 30, 2020
f97d6ce
Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. [(#4022)](https://g…
kurtisvg Jun 9, 2020
22375a3
Update dependency google-cloud-bigquery to v1.25.0 [(#4024)](https://…
renovate-bot Jun 11, 2020
d3cf8a9
chore(deps): update dependency matplotlib to v3.2.2 [(#4118)](https:/…
renovate-bot Jun 19, 2020
d78170e
Update dependency ipython to v7.16.1 [(#4189)](https://github.com/Goo…
renovate-bot Jun 27, 2020
9a1cb6c
chore(deps): update dependency pytest to v5.4.3 [(#4279)](https://git…
renovate-bot Jul 12, 2020
092370c
chore(deps): update dependency mock to v4 [(#4287)](https://github.co…
renovate-bot Jul 13, 2020
f244bea
chore(deps): update dependency matplotlib to v3.3.0 [(#4310)](https:/…
renovate-bot Jul 17, 2020
dadc62a
chore: update templates
plamut Jul 21, 2020
660407e
chore: change project env variable to use in tests
plamut Jul 21, 2020
25b636f
chore: fix samples tests dependencies
plamut Jul 21, 2020
6ca8cc6
chore: exclude orphan samples doc files from TOC
plamut Jul 21, 2020
a42eff1
chore: skip samples/snippets in top level snippets test
plamut Jul 22, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Code owners file.
# This file controls who is tagged for review for any given pull request.
#
# For syntax help see:
# https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners#codeowners-syntax


/samples/ @shollyman @googleapis/python-samples-owners

7 changes: 6 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,12 @@

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ["_build"]
exclude_patterns = [
"_build",
"samples/AUTHORING_GUIDE.md",
"samples/CONTRIBUTING.md",
"samples/snippets/README.rst",
]

# The reST default role (used for this markup: `text`) to use for all
# documents.
Expand Down
4 changes: 3 additions & 1 deletion noxfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,8 +113,10 @@ def snippets(session):
session.install("-e", ".[all]")

# Run py.test against the snippets tests.
# Skip tests in samples/snippets, as those are run in a different session
# using the nox config from that directory.
session.run("py.test", os.path.join("docs", "snippets.py"), *session.posargs)
session.run("py.test", "samples", *session.posargs)
session.run("py.test", "samples", "--ignore=samples/snippets", *session.posargs)


@nox.session(python="3.8")
Expand Down
1 change: 1 addition & 0 deletions samples/AUTHORING_GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
See https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md
1 change: 1 addition & 0 deletions samples/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
See https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/CONTRIBUTING.md
2 changes: 2 additions & 0 deletions samples/snippets/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
client_secrets.json
service_account.json
166 changes: 166 additions & 0 deletions samples/snippets/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@

.. This file is automatically generated. Do not edit this file directly.

Google BigQuery Python Samples
===============================================================================

.. image:: https://gstatic.com/cloudssh/images/open-btn.png
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=bigquery/cloud-client/README.rst


This directory contains samples for Google BigQuery. `Google BigQuery`_ is Google's fully managed, petabyte scale, low cost analytics data warehouse. BigQuery is NoOps—there is no infrastructure to manage and you don't need a database administrator—so you can focus on analyzing data to find meaningful insights, use familiar SQL, and take advantage of our pay-as-you-go model.




.. _Google BigQuery: https://cloud.google.com/bigquery/docs


Setup
-------------------------------------------------------------------------------



Authentication
++++++++++++++

This sample requires you to have authentication setup. Refer to the
`Authentication Getting Started Guide`_ for instructions on setting up
credentials for applications.

.. _Authentication Getting Started Guide:
https://cloud.google.com/docs/authentication/getting-started




Install Dependencies
++++++++++++++++++++

#. Clone python-docs-samples and change directory to the sample directory you want to use.

.. code-block:: bash

$ git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git

#. Install `pip`_ and `virtualenv`_ if you do not already have them. You may want to refer to the `Python Development Environment Setup Guide`_ for Google Cloud Platform for instructions.

.. _Python Development Environment Setup Guide:
https://cloud.google.com/python/setup

#. Create a virtualenv. Samples are compatible with Python 3.6+.

.. code-block:: bash

$ virtualenv env
$ source env/bin/activate

#. Install the dependencies needed to run the samples.

.. code-block:: bash

$ pip install -r requirements.txt

.. _pip: https://pip.pypa.io/
.. _virtualenv: https://virtualenv.pypa.io/






Samples
-------------------------------------------------------------------------------


Quickstart
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

.. image:: https://gstatic.com/cloudssh/images/open-btn.png
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=bigquery/cloud-client/quickstart.py,bigquery/cloud-client/README.rst




To run this sample:

.. code-block:: bash

$ python quickstart.py




Simple Application
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

.. image:: https://gstatic.com/cloudssh/images/open-btn.png
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=bigquery/cloud-client/simple_app.py,bigquery/cloud-client/README.rst




To run this sample:

.. code-block:: bash

$ python simple_app.py




User Credentials
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

.. image:: https://gstatic.com/cloudssh/images/open-btn.png
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=bigquery/cloud-client/user_credentials.py,bigquery/cloud-client/README.rst




To run this sample:

.. code-block:: bash

$ python user_credentials.py


usage: user_credentials.py [-h] [--launch-browser] project

Command-line application to run a query using user credentials.

You must supply a client secrets file, which would normally be bundled with
your application.

positional arguments:
project Project to use for BigQuery billing.

optional arguments:
-h, --help show this help message and exit
--launch-browser Use a local server flow to authenticate.









The client library
-------------------------------------------------------------------------------

This sample uses the `Google Cloud Client Library for Python`_.
You can read the documentation for more details on API usage and use GitHub
to `browse the source`_ and `report issues`_.

.. _Google Cloud Client Library for Python:
https://googlecloudplatform.github.io/google-cloud-python/
.. _browse the source:
https://github.com/GoogleCloudPlatform/google-cloud-python
.. _report issues:
https://github.com/GoogleCloudPlatform/google-cloud-python/issues



.. _Google Cloud SDK: https://cloud.google.com/sdk/
31 changes: 31 additions & 0 deletions samples/snippets/README.rst.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# This file is used to generate README.rst

product:
name: Google BigQuery
short_name: BigQuery
url: https://cloud.google.com/bigquery/docs
description: >
`Google BigQuery`_ is Google's fully managed, petabyte scale, low cost
analytics data warehouse. BigQuery is NoOps—there is no infrastructure to
manage and you don't need a database administrator—so you can focus on
analyzing data to find meaningful insights, use familiar SQL, and take
advantage of our pay-as-you-go model.

required_role: BigQuery Admin

setup:
- auth
- install_deps

samples:
- name: Quickstart
file: quickstart.py
- name: Simple Application
file: simple_app.py
- name: User Credentials
file: user_credentials.py
show_help: true

cloud_client_library: true

folder: bigquery/cloud-client
39 changes: 39 additions & 0 deletions samples/snippets/authenticate_service_account.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import os


def main():
key_path = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")

# [START bigquery_client_json_credentials]
from google.cloud import bigquery
from google.oauth2 import service_account

# TODO(developer): Set key_path to the path to the service account key
# file.
# key_path = "path/to/service_account.json"

credentials = service_account.Credentials.from_service_account_file(
key_path, scopes=["https://www.googleapis.com/auth/cloud-platform"],
)

client = bigquery.Client(credentials=credentials, project=credentials.project_id,)
# [END bigquery_client_json_credentials]
return client


if __name__ == "__main__":
main()
33 changes: 33 additions & 0 deletions samples/snippets/authenticate_service_account_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import google.auth

import authenticate_service_account


def mock_credentials(*args, **kwargs):
credentials, _ = google.auth.default(
["https://www.googleapis.com/auth/cloud-platform"]
)
return credentials


def test_main(monkeypatch):
monkeypatch.setattr(
"google.oauth2.service_account.Credentials.from_service_account_file",
mock_credentials,
)
client = authenticate_service_account.main()
assert client is not None
Loading