Skip to content

Commit 8c43898

Browse files
Ricardo Mendestswast
Ricardo Mendes
authored and
tswast
committed
Add samples for Data Catalog lookup_entry [(#2148)](GoogleCloudPlatform/python-docs-samples#2148)
* Add samples for Data Catalog lookup_entry * Add tests for Data Catalog lookup_entry * Add samples for lookup_entry by SQL Resource * Add README.rst * Improve command line interface * Removed the "lookup-" prefix from commands * Handle the --sql-resource optional argument by subparsers * Refer to GCP public assets in tests
0 parents  commit 8c43898

File tree

5 files changed

+348
-0
lines changed

5 files changed

+348
-0
lines changed

‎samples/snippet/README.rst

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
.. This file is automatically generated. Do not edit this file directly.
2+
3+
Google Cloud Data Catalog Python Samples
4+
===============================================================================
5+
6+
.. image:: https://gstatic.com/cloudssh/images/open-btn.png
7+
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=datacatalog/cloud-client/README.rst
8+
9+
10+
This directory contains samples for Google Cloud Data Catalog. `Google Cloud Data Catalog`_ is a fully managed and scalable metadata management service that empowers organizations to quickly discover, manage, and understand all their data in Google Cloud.
11+
12+
13+
14+
15+
.. _Google Cloud Data Catalog: https://cloud.google.com/data-catalog/docs
16+
17+
Setup
18+
-------------------------------------------------------------------------------
19+
20+
21+
Authentication
22+
++++++++++++++
23+
24+
This sample requires you to have authentication setup. Refer to the
25+
`Authentication Getting Started Guide`_ for instructions on setting up
26+
credentials for applications.
27+
28+
.. _Authentication Getting Started Guide:
29+
https://cloud.google.com/docs/authentication/getting-started
30+
31+
Install Dependencies
32+
++++++++++++++++++++
33+
34+
#. Clone python-docs-samples and change directory to the sample directory you want to use.
35+
36+
.. code-block:: bash
37+
38+
$ git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git
39+
40+
#. Install `pip`_ and `virtualenv`_ if you do not already have them. You may want to refer to the `Python Development Environment Setup Guide`_ for Google Cloud Platform for instructions.
41+
42+
.. _Python Development Environment Setup Guide:
43+
https://cloud.google.com/python/setup
44+
45+
#. Create a virtualenv. Samples are compatible with Python 2.7 and 3.4+.
46+
47+
.. code-block:: bash
48+
49+
$ virtualenv env
50+
$ source env/bin/activate
51+
52+
#. Install the dependencies needed to run the samples.
53+
54+
.. code-block:: bash
55+
56+
$ pip install -r requirements.txt
57+
58+
.. _pip: https://pip.pypa.io/
59+
.. _virtualenv: https://virtualenv.pypa.io/
60+
61+
Samples
62+
-------------------------------------------------------------------------------
63+
64+
Lookup entry
65+
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
66+
67+
.. image:: https://gstatic.com/cloudssh/images/open-btn.png
68+
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/GoogleCloudPlatform/python-docs-samples&page=editor&open_in_editor=datacatalog/cloud-client/lookup_entry.py,datacatalog/cloud-client/README.rst
69+
70+
71+
72+
73+
To run this sample:
74+
75+
.. code-block:: bash
76+
77+
$ python lookup_entry.py
78+
79+
usage: lookup_entry.py [-h]
80+
project_id
81+
{bigquery-dataset,bigquery-table,pubsub-topic} ...
82+
83+
This application demonstrates how to perform basic operations on entries
84+
with the Cloud Data Catalog API.
85+
86+
For more information, see the README.md under /datacatalog and the
87+
documentation at https://cloud.google.com/data-catalog/docs.
88+
89+
positional arguments:
90+
project_id Your Google Cloud project ID
91+
{bigquery-dataset,bigquery-table,pubsub-topic}
92+
bigquery-dataset Retrieves Data Catalog entry for the given BigQuery
93+
Dataset.
94+
bigquery-table Retrieves Data Catalog entry for the given BigQuery
95+
Table.
96+
pubsub-topic Retrieves Data Catalog entry for the given Pub/Sub
97+
Topic.
98+
99+
optional arguments:
100+
-h, --help show this help message and exit
101+
102+
103+
104+
105+
106+
The client library
107+
-------------------------------------------------------------------------------
108+
109+
This sample uses the `Google Cloud Client Library for Python`_.
110+
You can read the documentation for more details on API usage and use GitHub
111+
to `browse the source`_ and `report issues`_.
112+
113+
.. _Google Cloud Client Library for Python:
114+
https://googlecloudplatform.github.io/google-cloud-python/
115+
.. _browse the source:
116+
https://github.com/GoogleCloudPlatform/google-cloud-python
117+
.. _report issues:
118+
https://github.com/GoogleCloudPlatform/google-cloud-python/issues
119+
120+
121+
.. _Google Cloud SDK: https://cloud.google.com/sdk/

‎samples/snippet/README.rst.in

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# This file is used to generate README.rst
2+
3+
product:
4+
name: Google Cloud Data Catalog
5+
short_name: Data Catalog
6+
url: https://cloud.google.com/data-catalog/docs
7+
description: >
8+
`Google Cloud Data Catalog`_ is a fully managed and scalable metadata
9+
management service that empowers organizations to quickly discover, manage,
10+
and understand all their data in Google Cloud.
11+
12+
setup:
13+
- auth
14+
- install_deps
15+
16+
samples:
17+
- name: Lookup entry
18+
file: lookup_entry.py
19+
show_help: true
20+
21+
cloud_client_library: true
22+
23+
folder: datacatalog/cloud-client

‎samples/snippet/lookup_entry.py

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
#!/usr/bin/env python
2+
3+
# Copyright 2019 Google Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
"""This application demonstrates how to perform basic operations on entries
18+
with the Cloud Data Catalog API.
19+
20+
For more information, see the README.md under /datacatalog and the
21+
documentation at https://cloud.google.com/data-catalog/docs.
22+
"""
23+
24+
import argparse
25+
26+
27+
def lookup_bigquery_dataset(project_id, dataset_id):
28+
"""Retrieves Data Catalog entry for the given BigQuery Dataset."""
29+
from google.cloud import datacatalog_v1beta1
30+
31+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
32+
33+
resource_name = '//bigquery.googleapis.com/projects/{}/datasets/{}'\
34+
.format(project_id, dataset_id)
35+
36+
return datacatalog.lookup_entry(linked_resource=resource_name)
37+
38+
39+
def lookup_bigquery_dataset_sql_resource(project_id, dataset_id):
40+
"""Retrieves Data Catalog entry for the given BigQuery Dataset by
41+
sql_resource.
42+
"""
43+
from google.cloud import datacatalog_v1beta1
44+
45+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
46+
47+
sql_resource = 'bigquery.dataset.`{}`.`{}`'.format(project_id, dataset_id)
48+
49+
return datacatalog.lookup_entry(sql_resource=sql_resource)
50+
51+
52+
def lookup_bigquery_table(project_id, dataset_id, table_id):
53+
"""Retrieves Data Catalog entry for the given BigQuery Table."""
54+
from google.cloud import datacatalog_v1beta1
55+
56+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
57+
58+
resource_name = '//bigquery.googleapis.com/projects/{}/datasets/{}' \
59+
'/tables/{}'\
60+
.format(project_id, dataset_id, table_id)
61+
62+
return datacatalog.lookup_entry(linked_resource=resource_name)
63+
64+
65+
def lookup_bigquery_table_sql_resource(project_id, dataset_id, table_id):
66+
"""Retrieves Data Catalog entry for the given BigQuery Table by
67+
sql_resource.
68+
"""
69+
from google.cloud import datacatalog_v1beta1
70+
71+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
72+
73+
sql_resource = 'bigquery.table.`{}`.`{}`.`{}`'.format(
74+
project_id, dataset_id, table_id)
75+
76+
return datacatalog.lookup_entry(sql_resource=sql_resource)
77+
78+
79+
def lookup_pubsub_topic(project_id, topic_id):
80+
"""Retrieves Data Catalog entry for the given Pub/Sub Topic."""
81+
from google.cloud import datacatalog_v1beta1
82+
83+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
84+
85+
resource_name = '//pubsub.googleapis.com/projects/{}/topics/{}'\
86+
.format(project_id, topic_id)
87+
88+
return datacatalog.lookup_entry(linked_resource=resource_name)
89+
90+
91+
def lookup_pubsub_topic_sql_resource(project_id, topic_id):
92+
"""Retrieves Data Catalog entry for the given Pub/Sub Topic by
93+
sql_resource.
94+
"""
95+
from google.cloud import datacatalog_v1beta1
96+
97+
datacatalog = datacatalog_v1beta1.DataCatalogClient()
98+
99+
sql_resource = 'pubsub.topic.`{}`.`{}`'.format(project_id, topic_id)
100+
101+
return datacatalog.lookup_entry(sql_resource=sql_resource)
102+
103+
104+
if __name__ == '__main__':
105+
parser = argparse.ArgumentParser(
106+
description=__doc__,
107+
formatter_class=argparse.RawDescriptionHelpFormatter
108+
)
109+
110+
parser.add_argument('project_id', help='Your Google Cloud project ID')
111+
112+
subparsers = parser.add_subparsers(dest='command')
113+
114+
bigquery_dataset_parser = subparsers.add_parser(
115+
'bigquery-dataset', help=lookup_bigquery_dataset.__doc__)
116+
bigquery_dataset_parser.add_argument('dataset_id')
117+
bigquery_dataset_parser.add_argument('--sql-resource', action='store_true',
118+
help='Perform lookup by SQL Resource')
119+
120+
bigquery_table_parser = subparsers.add_parser(
121+
'bigquery-table', help=lookup_bigquery_table.__doc__)
122+
bigquery_table_parser.add_argument('dataset_id')
123+
bigquery_table_parser.add_argument('table_id')
124+
bigquery_table_parser.add_argument('--sql-resource', action='store_true',
125+
help='Perform lookup by SQL Resource')
126+
127+
pubsub_topic_parser = subparsers.add_parser(
128+
'pubsub-topic', help=lookup_pubsub_topic.__doc__)
129+
pubsub_topic_parser.add_argument('topic_id')
130+
pubsub_topic_parser.add_argument('--sql-resource', action='store_true',
131+
help='Perform lookup by SQL Resource')
132+
133+
args = parser.parse_args()
134+
135+
entry = None
136+
137+
if args.command == 'bigquery-dataset':
138+
lookup_method = lookup_bigquery_dataset_sql_resource \
139+
if args.sql_resource else lookup_bigquery_dataset
140+
entry = lookup_method(args.project_id, args.dataset_id)
141+
elif args.command == 'bigquery-table':
142+
lookup_method = lookup_bigquery_table_sql_resource \
143+
if args.sql_resource else lookup_bigquery_table
144+
entry = lookup_method(args.project_id, args.dataset_id, args.table_id)
145+
elif args.command == 'pubsub-topic':
146+
lookup_method = lookup_pubsub_topic_sql_resource \
147+
if args.sql_resource else lookup_pubsub_topic
148+
entry = lookup_method(args.project_id, args.topic_id)
149+
150+
print(entry.name)

‎samples/snippet/lookup_entry_test.py

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env python
2+
3+
# Copyright 2019 Google Inc. All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
import lookup_entry
18+
19+
BIGQUERY_PROJECT = 'bigquery-public-data'
20+
BIGQUERY_DATASET = 'new_york_taxi_trips'
21+
BIGQUERY_TABLE = 'taxi_zone_geom'
22+
23+
PUBSUB_PROJECT = 'pubsub-public-data'
24+
PUBSUB_TOPIC = 'taxirides-realtime'
25+
26+
27+
def test_lookup_bigquery_dataset():
28+
assert lookup_entry.lookup_bigquery_dataset(
29+
BIGQUERY_PROJECT, BIGQUERY_DATASET)
30+
31+
32+
def test_lookup_bigquery_dataset_sql_resource():
33+
assert lookup_entry.lookup_bigquery_dataset_sql_resource(
34+
BIGQUERY_PROJECT, BIGQUERY_DATASET)
35+
36+
37+
def test_lookup_bigquery_table():
38+
assert lookup_entry.lookup_bigquery_table(
39+
BIGQUERY_PROJECT, BIGQUERY_DATASET, BIGQUERY_TABLE)
40+
41+
42+
def test_lookup_bigquery_table_sql_resource():
43+
assert lookup_entry.lookup_bigquery_table_sql_resource(
44+
BIGQUERY_PROJECT, BIGQUERY_DATASET, BIGQUERY_TABLE)
45+
46+
47+
def test_lookup_pubsub_topic():
48+
assert lookup_entry.lookup_pubsub_topic(PUBSUB_PROJECT, PUBSUB_TOPIC)
49+
50+
51+
def test_lookup_pubsub_topic_sql_resource():
52+
assert lookup_entry.lookup_pubsub_topic_sql_resource(
53+
PUBSUB_PROJECT, PUBSUB_TOPIC)

‎samples/snippet/requirements.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
google-cloud-datacatalog==0.1.0

0 commit comments

Comments
 (0)