Skip to content

Commit 3f60989

Browse files
t-karasovapartheagcf-owl-bot[bot]release-please[bot]
authored
chore: add scripts to setup and teardown test resources (#137)
* chore: update .repo-metadata.json (#125) * chore: use python-samples-reviewers (#128) * chore: use gapic-generator-python 0.58.4 (#127) * chore: use gapic-generator-python 0.58.4 fix: provide appropriate mock values for message body fields committer: dovs PiperOrigin-RevId: 419025932 Source-Link: googleapis/googleapis@73da669 Source-Link: https://github.com/googleapis/googleapis-gen/commit/46df624a54b9ed47c1a7eefb7a49413cf7b82f98 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDZkZjYyNGE1NGI5ZWQ0N2MxYTdlZWZiN2E0OTQxM2NmN2I4MmY5OCJ9 * 🦉 Updates from OwlBot See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> * chore(python): update release.sh to use keystore (#130) build: switch to release-please for tagging * chore(main): release 1.3.0 (#131) Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> * ci(python): run lint / unit tests / docs as GH actions (#132) * ci(python): run lint / unit tests / docs as GH actions Source-Link: googleapis/synthtool@57be0cd Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:ed1f9983d5a935a89fe8085e8bb97d94e41015252c5b6c9771257cf8624367e6 * add commit to trigger gh actions Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <[email protected]> * add setup/cleanup test resources scripts * Update samples/snippets/remove_test_resources.py Co-authored-by: Anthonios Partheniou <[email protected]> * review comments fix Co-authored-by: Anthonios Partheniou <[email protected]> Co-authored-by: gcf-owl-bot[bot] <78513119+gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
0 parents  commit 3f60989

File tree

4 files changed

+555
-0
lines changed

4 files changed

+555
-0
lines changed

‎generated_samples/snippets/TEST_RESOURCES_SETUP_CLEANUP.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# How to set up/ tear down the test resources
2+
3+
## Required environment variables
4+
5+
To successfully import the catalog data for tests, the following environment variables should be set:
6+
- PROJECT_NUMBER
7+
- BUCKET_NAME
8+
These values are stored in the Secret Manager and will be submitted as
9+
docker environment variables before the test run.
10+
11+
The Secret Manager name is set in .kokoro/presubmit/common.cfg file, SECRET_MANAGER_KEYS variable.
12+
13+
## Import catalog data
14+
15+
There is a JSON file with valid products prepared in the `product` directory:
16+
`resources/products.json`.
17+
18+
Run the `create_test_resources.py` to perform the following actions:
19+
- create the GCS bucket <BUCKET_NAME>,
20+
- upload the product data from `resources/products.json` file,
21+
- import products to the default branch of the Retail catalog.
22+
23+
```
24+
$ python create_test_resources.py
25+
```
26+
27+
In the result 316 products should be created in the test project catalog.
28+
29+
30+
## Remove catalog data
31+
32+
Run the `remove_test_resources.py` to perform the following actions:
33+
- remove all objects from the GCS bucket <BUCKET_NAME>,
34+
- remove the <BUCKET_NAME> bucket,
35+
- delete all products from the Retail catalog.
36+
37+
```
38+
$ python remove_test_resources.py
39+
```

‎generated_samples/snippets/create_test_resources.py

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Copyright 2021 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import os
16+
import re
17+
import time
18+
19+
from google.cloud.storage.bucket import Bucket
20+
21+
from google.cloud import storage
22+
from google.cloud.retail import GcsSource, ImportErrorsConfig, \
23+
ImportProductsRequest, ProductInputConfig
24+
from google.cloud.retail_v2 import ProductServiceClient
25+
26+
project_number = os.getenv('PROJECT_NUMBER')
27+
bucket_name = os.getenv('BUCKET_NAME')
28+
storage_client = storage.Client()
29+
resource_file = "resources/products.json"
30+
object_name = re.search('resources/(.*?)$', resource_file).group(1)
31+
default_catalog = "projects/{0}/locations/global/catalogs/default_catalog/branches/default_branch".format(
32+
project_number)
33+
34+
35+
def create_bucket(bucket_name: str) -> Bucket:
36+
"""Create a new bucket in Cloud Storage"""
37+
print("Creating new bucket:" + bucket_name)
38+
bucket_exists = check_if_bucket_exists(bucket_name)
39+
if bucket_exists:
40+
print("Bucket {} already exists".format(bucket_name))
41+
return storage_client.bucket(bucket_name)
42+
else:
43+
bucket = storage_client.bucket(bucket_name)
44+
bucket.storage_class = "STANDARD"
45+
new_bucket = storage_client.create_bucket(bucket, location="us")
46+
print(
47+
"Created bucket {} in {} with storage class {}".format(
48+
new_bucket.name, new_bucket.location, new_bucket.storage_class
49+
)
50+
)
51+
return new_bucket
52+
53+
54+
def check_if_bucket_exists(new_bucket_name):
55+
"""Check if bucket is already exists"""
56+
bucket_exists = False
57+
buckets = storage_client.list_buckets()
58+
for bucket in buckets:
59+
if bucket.name == new_bucket_name:
60+
bucket_exists = True
61+
break
62+
return bucket_exists
63+
64+
65+
def upload_data_to_bucket(bucket: Bucket):
66+
"""Upload data to a GCS bucket"""
67+
blob = bucket.blob(object_name)
68+
blob.upload_from_filename(resource_file)
69+
print("Data from {} has being uploaded to {}".format(resource_file,
70+
bucket.name))
71+
72+
73+
def get_import_products_gcs_request():
74+
"""Get import products from gcs request"""
75+
gcs_bucket = "gs://{}".format(bucket_name)
76+
gcs_errors_bucket = "{}/error".format(gcs_bucket)
77+
78+
gcs_source = GcsSource()
79+
gcs_source.input_uris = ["{0}/{1}".format(gcs_bucket, object_name)]
80+
81+
input_config = ProductInputConfig()
82+
input_config.gcs_source = gcs_source
83+
84+
errors_config = ImportErrorsConfig()
85+
errors_config.gcs_prefix = gcs_errors_bucket
86+
87+
import_request = ImportProductsRequest()
88+
import_request.parent = default_catalog
89+
import_request.reconciliation_mode = ImportProductsRequest.ReconciliationMode.INCREMENTAL
90+
import_request.input_config = input_config
91+
import_request.errors_config = errors_config
92+
93+
print("---import products from google cloud source request---")
94+
print(import_request)
95+
96+
return import_request
97+
98+
99+
def import_products_from_gcs():
100+
"""Call the Retail API to import products"""
101+
import_gcs_request = get_import_products_gcs_request()
102+
gcs_operation = ProductServiceClient().import_products(
103+
import_gcs_request)
104+
print(
105+
"Import operation is started: {}".format(gcs_operation.operation.name))
106+
107+
while not gcs_operation.done():
108+
print("Please wait till operation is completed")
109+
time.sleep(5)
110+
print("Import products operation is completed")
111+
112+
if gcs_operation.metadata is not None:
113+
print("Number of successfully imported products")
114+
print(gcs_operation.metadata.success_count)
115+
print("Number of failures during the importing")
116+
print(gcs_operation.metadata.failure_count)
117+
else:
118+
print("Operation.metadata is empty")
119+
120+
print(
121+
"Wait 2 -5 minutes till products become indexed in the catalog,\
122+
after that they will be available for search")
123+
124+
125+
created_bucket = create_bucket(bucket_name)
126+
upload_data_to_bucket(created_bucket)
127+
import_products_from_gcs()

‎generated_samples/snippets/remove_test_resources.py

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Copyright 2021 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import os
16+
17+
from google.api_core.exceptions import PermissionDenied
18+
from google.cloud.storage.bucket import Bucket
19+
20+
from google.cloud import storage
21+
from google.cloud.retail import DeleteProductRequest, ListProductsRequest, \
22+
ProductServiceClient
23+
24+
project_number = os.getenv('PROJECT_NUMBER')
25+
bucket_name = os.getenv('BUCKET_NAME')
26+
27+
default_catalog = "projects/{0}/locations/global/catalogs/default_catalog/branches/default_branch".format(
28+
project_number)
29+
30+
storage_client = storage.Client()
31+
32+
33+
def delete_bucket():
34+
"""Delete bucket"""
35+
try:
36+
bucket = storage_client.get_bucket(bucket_name)
37+
except:
38+
print("Bucket {} does not exists".format(bucket_name))
39+
else:
40+
delete_object_from_bucket(bucket)
41+
bucket.delete()
42+
print("bucket {} is deleted".format(bucket_name))
43+
44+
45+
def delete_object_from_bucket(bucket: Bucket):
46+
"""Delete object from bucket"""
47+
blobs = bucket.list_blobs()
48+
for blob in blobs:
49+
blob.delete()
50+
print("all objects are deleted from GCS bucket {}".format(bucket.name))
51+
52+
53+
def delete_all_products():
54+
"""Delete all products in the catalog"""
55+
product_client = ProductServiceClient()
56+
list_request = ListProductsRequest()
57+
list_request.parent = default_catalog
58+
products = product_client.list_products(list_request)
59+
delete_count = 0
60+
for product in products:
61+
delete_request = DeleteProductRequest()
62+
delete_request.name = product.name
63+
try:
64+
product_client.delete_product(delete_request)
65+
delete_count += 1
66+
except PermissionDenied:
67+
print(
68+
"Ignore PermissionDenied in case the product does not exist at time of deletion")
69+
print(f"{delete_count} products were deleted from {default_catalog}")
70+
71+
72+
delete_bucket()
73+
delete_all_products()

0 commit comments

Comments
 (0)