From b5109a76b8ade75636ddb34ae46f29a221a73693 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Thu, 31 Oct 2024 12:20:31 -0400 Subject: [PATCH 01/16] docs: modify hermetic build docs --- .../{library_generation => }/DEVELOPMENT.md | 0 .../{library_generation => }/README.md | 131 +++++++++++------- 2 files changed, 81 insertions(+), 50 deletions(-) rename hermetic_build/{library_generation => }/DEVELOPMENT.md (100%) rename hermetic_build/{library_generation => }/README.md (82%) diff --git a/hermetic_build/library_generation/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md similarity index 100% rename from hermetic_build/library_generation/DEVELOPMENT.md rename to hermetic_build/DEVELOPMENT.md diff --git a/hermetic_build/library_generation/README.md b/hermetic_build/README.md similarity index 82% rename from hermetic_build/library_generation/README.md rename to hermetic_build/README.md index 0b4208ac3e..ee219615a0 100644 --- a/hermetic_build/library_generation/README.md +++ b/hermetic_build/README.md @@ -1,44 +1,43 @@ +> [!IMPORTANT] +> All scripts/examples assume you are inside the repository root folder. + # Generate a repository containing GAPIC Client Libraries -The script, `entry_point.py`, allows you to generate a repository containing -GAPIC client libraries (a monorepo, for example, google-cloud-java) from a -configuration file. +Running the docker image built from `hermetic_build/library_generation` +directory, you can generate a repository containing GAPIC client libraries (a +monorepo, for example, google-cloud-java) from a configuration file. + +Instead of running docker image, if you prefer running the underlying python +script directly, please refer to [development guide](DEVELOPMENT.md) for +additional instructions. ## Environment - OS: Linux -- Java runtime environment (8 or above) -- Python (3.12 or above) -- Docker -- Git +- Docker ## Prerequisite In order to generate a version for each library, a versions.txt has to exist -in `repository_path`. -Please refer to [Repository path](#repository-path--repositorypath---optional) for more information. - -## Parameters to generate a repository using `entry_point.py` - -### Baseline generation configuration yaml (`baseline_generation_config`) +in `repository-path`. +Please refer to [Repository path](#repository-path--repositorypath---optional) +for more information. -An absolute or relative path to a generation_config.yaml. -This config file is used for computing changed libraries, not library -generation. +## Parameters to generate a repository using the docker image -### Current generation configuration yaml (`current_generation_config`) +### Generation configuration yaml (`generation-config-path`) An absolute or relative path to a configuration file containing parameters to generate the repository. Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository) for more information. -### Repository path (`repository_path`), optional +### Repository path (`repository-path`), optional The path to where the generated repository goes. The default value is the current working directory when running the script. -For example, `cd google-cloud-java && python entry_point.py ...` without +For example, `cd google-cloud-java && python /path/to/entry_point.py ...` without specifying the `--repository_path` option will modify the `google-cloud-java` repository the user `cd`'d into. @@ -47,28 +46,37 @@ right version for each library. Please refer [here](go/java-client-releasing#versionstxt-manifest) for more info of versions.txt. -### Api definitions path (`api_definitions_path`), optional +### A list of library names (`library-names`), optional + +A list of library names that will be generated, separated by comma. +The library name of a library is the value of `library_name` or `api_shortname`, +if `library_name` is not specified, in the generation configuration. + +If `library_names` is not specified, all libraries in the generation +configuration will be generated. + +### Api definitions path (`api-definitions-path`), optional The path to where the api definition (proto, service yaml) resides. The default value is the current working directory when running the script. -Note that you need not only the protos defined the service, but also the transitive -dependencies of those protos. +Note that you need not only the protos defined the service, but also the +transitive dependencies of those protos. Any missing dependencies will cause `File not found` error. -For example, if your service is defined in `example_service.proto` and it imports -`google/api/annotations.proto`, you need the `annotations.proto` resides in a -folder that has the exact structure of the import statement (`google/api` in this -case), and set `api_definitions_path` to the path contains the root folder (`google` -in this case). +For example, if your service is defined in `example_service.proto` and it +imports `google/api/annotations.proto`, you need the `annotations.proto` resides +in a folder that has the exact structure of the import statement (`google/api` +in this case), and set `api_definitions_path` to the path contains the root +folder (`google` in this case). -## Output of `entry_point.py` +## Output ### GAPIC libraries -For each module (e.g. `google-cloud-java/java-asset`), the following files/folders -will be created/modified: +For each module (e.g. `google-cloud-java/java-asset`), the following +files/folders will be created/modified: | Name | Notes | |:------------------------------------|:-------------------------------------------------------------------------| @@ -185,32 +193,53 @@ libraries: - proto_path: google/cloud/asset/v1p7beta1 ``` -# Local Environment Setup before running `entry_point.py` +# Build the image from source -1. Assuming Python 3 is installed, follow official guide from [Python.org](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments) to create a virtual environment. -The virtual environment can be installed to any folder, usually it is recommended to be installed under the root folder of the project(`sdk-platform-java` in this case). -2. Assuming the virtual environment is installed under `sdk-platform-java`. -Run the following command under the root folder of `sdk-platform-java` to install `library_generation` and its dependencies. +1. Run the following command to build the image from source - ```bash - python -m pip install --require-hashes -r hermetic_build/common/requirements.txt - python -m pip install hermetic_build/common - python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt - python -m pip install hermetic_build/library_generation + ```shell + docker build \ + -f .cloudbuild/library_generation/library_generation.Dockerfile \ + -t local:image-tag \ + . ``` -3. Download api definition to a local directory +2. Download api definition to a local directory + ```shell + api_def_dir=/path/to/api_definition + ``` -## An example to generate a repository using `entry_point.py` +3. Set the version of gapic-generator-java + ```shell + LOCAL_GENERATOR_VERSION=$(mvn \ + org.apache.maven.plugins:maven-help-plugin:evaluate \ + -Dexpression=project.version \ + -pl gapic-generator-java \ + -DforceStdout \ + -q) + ``` -```bash -python hermetic_build/library_generation/cli/entry_point.py generate \ - --baseline-generation-config-path=/path/to/baseline_config_file \ - --current-generation-config-path=/path/to/current_config_file \ - --repository-path=path/to/repository \ - --api-definitions-path=path/to/api_definition -``` -If you run `entry_point.py` with the example [configuration](#an-example-of-generation-configuration) +4. Run the docker image + ```shell + # Assume you want to generate the library in the current working directory + # and the generation configuration is in the same directory. + docker run \ + --rm \ + --quiet \ + -u "$(id -u):$(id -g)" \ + -v "$(pwd):/workspace" \ + -v "${api_def_dir}:/workspace/googleapis" \ + -e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}" \ + local:image-tag \ + --generation-config-path=/workspace/generation_config_file \ + --library-names=apigee-connect,asset \ + --repository-path=/workspace \ + --api-definitions-path=/workspace/googleapis + ``` + +## An example to generate a repository using the docker image + +If you run the docker image with the example [configuration](#an-example-of-generation-configuration) shown above, the repository structure is: ``` $repository_path @@ -284,6 +313,8 @@ $repository_path |_versions.txt ``` +# Generate release note from library changes + # Owlbot Java Postprocessor We have transferred the From 52386c7076f63bae0edfb91f089e73b2716594fe Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Thu, 31 Oct 2024 15:46:48 -0400 Subject: [PATCH 02/16] change comments in cli --- hermetic_build/README.md | 49 +++++++++++++++++-- .../cli/generate_release_note.py | 5 +- 2 files changed, 48 insertions(+), 6 deletions(-) diff --git a/hermetic_build/README.md b/hermetic_build/README.md index ee219615a0..947a93713b 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -113,7 +113,7 @@ They are shared by library level parameters. | gapic_generator_version | No | set through env variable if not specified | | protoc_version | No | inferred from the generator if not specified | | grpc_version | No | inferred from the generator if not specified | -| googleapis_commitish | Yes | | +| googleapis_committish | Yes | | | libraries_bom_version | No | empty string if not specified | ### Library level parameters @@ -137,7 +137,7 @@ They are shared by all GAPICs of a library. | distribution_name | No | `{group_id}:google-{cloud_prefix}{library_name}` if not specified | | excluded_poms | No | | | excluded_dependencies | No | | -| googleapis_commitish | No | use repository level `googleapis_commitish` if not specified. | +| googleapis_committish | No | use repository level `googleapis_committish` if not specified. | | group_id | No | `com.google.cloud` if not specified | | issue_tracker | No | | | library_name | No | `api_shortname` is not specified. This value should be unique among all libraries. | @@ -313,9 +313,50 @@ $repository_path |_versions.txt ``` -# Generate release note from library changes +# Generate release note from api definition changes -# Owlbot Java Postprocessor +The script, `hermetic_build/release_note_generation/cli/generate_release_note.py` +allows you to generate release note from api definition changes in +[googleapis](https://github.com/googleapis/googleapis) GitHub repository. + +## Environment + +- OS: Linux +- Python (3.12 or above) + +## Parameters to generate a release note + +### Baseline generation configuration path (`baseline-generation-config-path`) + +Absolute or relative path to a generation configuration. +Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository) +for more information. + +Note that the `googleapis_commitish` in this configuration is used to retrieve +the first commit, exclusively, to generate the release note. + +### Current generation configuration path (`current-generation-config-path`) + +Absolute or relative path to a generation configuration. +The release note will be generated from commits that related to the libraries +specified in this configuration. +Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository) +for more information. + +Note that the `googleapis_commitish` in this configuration is used to retrieve +the last commit, inclusively, to generate the release note. + +### Repository path (`repository-path`), optional + +The path to which the file, `pr_description.txt` containing the release note +will be sent. +If not specified, the file will be generated to the current working directory. + +### An example of generating release note + + + +# OwlBot Java Postprocessor We have transferred the [implementation](https://github.com/googleapis/synthtool/tree/59fe44fde9866a26e7ee4e4450fd79f67f8cf599/docker/owlbot/java) diff --git a/hermetic_build/release_note_generation/cli/generate_release_note.py b/hermetic_build/release_note_generation/cli/generate_release_note.py index 294f4baf50..57113a8ff1 100644 --- a/hermetic_build/release_note_generation/cli/generate_release_note.py +++ b/hermetic_build/release_note_generation/cli/generate_release_note.py @@ -54,8 +54,9 @@ def main(ctx): default=".", show_default=True, help=""" - The repository path to which the generated files will be sent. - If not specified, the repository will be generated to the current working + The path to which the file, `pr_description.txt` containing the release note + will be sent. + If not specified, the file will be generated to the current working directory. """, ) From 0a3641ad14333b8ce6a04ca9dbf714fe5749f3e6 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Thu, 31 Oct 2024 16:00:03 -0400 Subject: [PATCH 03/16] add instructions on how to generate release note --- hermetic_build/README.md | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/hermetic_build/README.md b/hermetic_build/README.md index 947a93713b..e6f3c4fb44 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -322,7 +322,7 @@ allows you to generate release note from api definition changes in ## Environment - OS: Linux -- Python (3.12 or above) +- Python (3.12.0 or above) ## Parameters to generate a release note @@ -352,9 +352,27 @@ The path to which the file, `pr_description.txt` containing the release note will be sent. If not specified, the file will be generated to the current working directory. -### An example of generating release note +## Generate a release note in local environment +1. Install python (>= 3.12.0), it is recommended to create a python virtual +environment through [official guide](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments). +2. Running the following commands to install python packages + ```shell + cd /path/to/sdk-platform-java + pip install --require-hashes -r hermetic_build/common/requirements.txt + pip install hermetic_build/common + pip install --require-hashes -r hermetic_build/release_note_generation/requirements.txt + pip install hermetic_build/release_note_generation + ``` +3. Running the following commands to generate a release note + ```shell + cd /path/to/sdk-platform-java + python hermetic_build/release_note_generation/cli/generate_release_note.py generate \ + --baseline-generation-config-path=/path/to/baseline_generation_config \ + --current-generation-config-path=/path/to/current_generation_config \ + --repository-path=/path/to/send/release_note + ``` # OwlBot Java Postprocessor From 425ba11faefb8af2a7034d0b5048cf9cebf6920d Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Thu, 31 Oct 2024 16:02:48 -0400 Subject: [PATCH 04/16] restore param --- hermetic_build/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hermetic_build/README.md b/hermetic_build/README.md index e6f3c4fb44..7ee87c799a 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -113,7 +113,7 @@ They are shared by library level parameters. | gapic_generator_version | No | set through env variable if not specified | | protoc_version | No | inferred from the generator if not specified | | grpc_version | No | inferred from the generator if not specified | -| googleapis_committish | Yes | | +| googleapis_commitish | Yes | | | libraries_bom_version | No | empty string if not specified | ### Library level parameters @@ -137,7 +137,7 @@ They are shared by all GAPICs of a library. | distribution_name | No | `{group_id}:google-{cloud_prefix}{library_name}` if not specified | | excluded_poms | No | | | excluded_dependencies | No | | -| googleapis_committish | No | use repository level `googleapis_committish` if not specified. | +| googleapis_commitish | No | use repository level `googleapis_committish` if not specified. | | group_id | No | `com.google.cloud` if not specified | | issue_tracker | No | | | library_name | No | `api_shortname` is not specified. This value should be unique among all libraries. | From 55c2b4053ab31d6a7c737a375058673042962407 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Fri, 1 Nov 2024 10:21:40 -0400 Subject: [PATCH 05/16] add notes in docker run --- hermetic_build/DEVELOPMENT.md | 163 +++++++++++++++------------------- hermetic_build/README.md | 27 +++++- 2 files changed, 93 insertions(+), 97 deletions(-) diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md index df7843aeaa..80e79476a5 100644 --- a/hermetic_build/DEVELOPMENT.md +++ b/hermetic_build/DEVELOPMENT.md @@ -1,5 +1,5 @@ > [!IMPORTANT] -> All examples assume you are inside the `hermetic_build` folder. +> All examples assume you are inside the repository root folder. # Linting @@ -8,80 +8,92 @@ When contributing, ensure your changes to python code have a valid format. ``` python -m pip install black -black . +black {source_file_or_directory} ``` -# Running the integration tests +# Install package dependencies + +```shell +python -m pip install --require-hashes -r hermetic_build/common/requirements.txt +python -m pip install hermetic_build/common +python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt +python -m pip install hermetic_build/library_generation +python -m pip install --require-hashes -r hermetic_build/release_note_generation/requirements.txt +python -m pip install hermetic_build/release_note_generation +``` + +# Run the integration tests The integration tests build the docker image declared in `.cloudbuild/library_generation/library_generation.Dockerfile`, pull GAPIC repositories, generate the libraries and compares the results with the source code declared in a "golden branch" of the repo. -It requires docker and python 3.x to be installed. +It requires docker and python (>= 3.12.0) to be installed. -``` -python -m pip install --require-hashes -r library_generation/requirements.txt -python -m pip install library_generation -python -m unittest library_generation/tests/integration_tests.py +```shell +python -m unittest hermetic_build/library_generation/tests/integration_tests.py ``` -# Running the unit tests +# Run the unit tests The unit tests of the hermetic build scripts are contained in several scripts, corresponding to a specific component. Every unit test script ends with `unit_tests.py`. To avoid them specifying them individually, we can use the following command: -```bash -python -m unittest discover -s library_generation/tests/ -p "*unit_tests.py" +```shell +python -m unittest discover -s hermetic_build -p "*unit_tests.py" ``` > [!NOTE] > The output of this command may look erratic during the first 30 seconds. > This is normal. After the tests are done, an "OK" message should be shown. -# Running the scripts in your local environment +# Run the library generation scripts in your local environment Although the scripts are designed to be run in a Docker container, you can also run them directly. This section explains how to run the entrypoint script -(`library_generation/cli/entry_point.py`). +(`hermetic_build/library_generation/cli/entry_point.py`). ## Assumptions made by the scripts + ### The Hermetic Build's well-known folder Located in `${HOME}/.library_generation`, this folder is assumed by the scripts to contain certain tools. Developers must make sure this folder is properly configured before running the scripts locally. -Note that this relies on the `HOME` en var which is always defined as per -[POSIX env var definition](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html). +Note that this relies on the `HOME` environment variable which is always defined +as per [POSIX env var definition](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html). #### Put the gapic-generator-java jar in its well-known location -Run `cd sdk-platform-java && mvn install -DskipTests -Dclirr.skip --Dcheckstyle.skip`. -This will generate a jar located in -`~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar` +1. Run the following command to install gapic-generator-java. -Then `mv` the jar into the well-known location of the jar. -The generation scripts will assume the jar is there. + ```shell + mvn install -B -ntp -DskipTests -Dclirr.skip -Dcheckstyle.skip + ``` + This will generate a jar located in `~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar` -```shell -mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar" -``` +2. Move the jar into the well-known location of the jar. + The generation scripts will assume the jar is there. + + ```shell + mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar" + ``` #### Put the java formatter jar in its well-known location -Download google-java-format-{version}-all-deps.jar from [Maven Central](https://central.sonatype.com/artifact/com.google.googlejavaformat/google-java-format) +1. Download google-java-format-{version}-all-deps.jar from [Maven Central](https://central.sonatype.com/artifact/com.google.googlejavaformat/google-java-format) or [GitHub releases](https://github.com/google/google-java-format/releases). -Then `mv` the jar into the well-known location of the jar. -The generation scripts will assume the jar is there. +2. Move the jar into the well-known location of the jar. + The generation scripts will assume the jar is there. -```shell -mv /path/to/jar "${HOME}/.library_generation/google-java-format.jar" -``` + ```shell + mv /path/to/jar "${HOME}/.library_generation/google-java-format.jar" + ``` ## Installing prerequisites @@ -91,12 +103,12 @@ need to install beforehand. ### Install the owl-bot CLI Requires node.js to be installed. -Check this [installation guide](https://github.com/nvm-sh/nvm?tab=readme-ov-file#install--update-script) -for NVM, Node.js's version manager. +Check this [installation guide](https://github.com/nvm-sh/nvm?tab=readme-ov-file#install--update-script) for NVM, Node.js's version manager. After you install it, you can install the owl-bot CLI with the following commands: -```bash + +```shell git clone https://github.com/googleapis/repo-automation-bots cd repo-automation-bots/packages/owl-bot npm i && npm run compile && npm link @@ -106,81 +118,46 @@ owl-bot copy-code --version The key step is `npm link`, which will make the command available in you current shell session. - -## Running the script -The entrypoint script (`library_generation/cli/entry_point.py`) allows you to -generate a GAPIC repository with a given api definition (proto, service yaml). +## Run the script +The entrypoint script (`hermetic_build/library_generation/cli/entry_point.py`) +allows you to generate a GAPIC repository with a given api definition (proto, +service yaml). ### Download the api definition -For example, googleapis -``` +For example, from googleapis + +```shell git clone https://github.com/googleapis/googleapis export api_definitions_path="$(pwd)/googleapis" ``` ### Download the repo For example, google-cloud-java -``` +```shell git clone https://github.com/googleapis/google-cloud-java export path_to_repo="$(pwd)/google-cloud-java" ``` ### Install the scripts -``` -python -m pip install . -``` - -### Run the script -``` -python library_generation/cli/entry_point.py generate \ - --repository-path="${path_to_repo}" \ - --api-definitions-path="${api_definitions_path}" -``` - - -# Running the scripts using the docker container image -This is convenient in order to avoid installing the dependencies manually. -> [!IMPORTANT] -> From now, the examples assume you are in the root of your sdk-platform-java -> folder. +You can skip this step if you've installed the packages in [Install package dependencies](#install-package-dependencies). -## Build the docker image -```bash -docker build --file .cloudbuild/library_generation/library_generation.Dockerfile --iidfile image-id . +```shell +python -m pip install --require-hashes -r hermetic_build/common/requirements.txt +python -m pip install hermetic_build/common +python -m pip install --require-hashes -r hermetic_build/library_generation/requirements.txt +python -m pip install hermetic_build/library_generation ``` -This will create an `image-id` file at the root of the repo with the hash ID of -the image. - -## Run the docker image -The docker image will perform changes on its internal `/workspace` folder, -to which you need to map a folder on your host machine (i.e. map your downloaded -repo to this folder). +### Run the script -To run the docker container on the google-cloud-java repo, you must run: -```bash -docker run \ - -u "$(id -u)":"$(id -g)" \ - -v /path/to/google-cloud-java:/workspace \ - -v /path/to/api-definition:/workspace/apis \ - $(cat image-id) \ - --api-definitions-path=/workspace/apis +```shell +python hermetic_build/library_generation/cli/entry_point.py generate \ + --repository-path="${path_to_repo}" \ + --api-definitions-path="${api_definitions_path}" ``` - * `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating - yourself. This avoids folder ownership changes since it runs as root by - default. - * `-v /path/to/google-cloud-java:/workspace` maps the host machine's - google-cloud-java folder to the /workspace folder. - The image is configured to perform changes in this directory. - * `-v /path/to/api-definition:/workspace/apis` maps the host machine's - api-definition folder to /workspace/apis folder. - * `$(cat image-id)` obtains the image ID created in the build step. - * `--api-definitions-path=/workspace/apis` set the API definition path to - `/workspace/apis`. - -## Debug the created containers +## Debug the library generation container If you are working on changing the way the containers are created, you may want to inspect the containers to check the setup. It would be convenient in such case to have a text editor/viewer available. @@ -188,18 +165,18 @@ You can achieve this by modifying the Dockerfile as follows: ```docker # install OS tools -RUN apt-get update && apt-get install -y \ - unzip openjdk-17-jdk rsync maven jq less vim \ - && apt-get clean +RUN apk update && apk add \ + unzip curl rsync openjdk11 jq bash nodejs npm git less vim ``` We add `less` and `vim` as text tools for further inspection. You can also run a shell in a new container by running: -```bash +```shell docker run \ - --rm -it \ + --rm \ + -it \ -u $(id -u):$(id -g) \ -v /path/to/google-cloud-java:/workspace \ --entrypoint="bash" \ diff --git a/hermetic_build/README.md b/hermetic_build/README.md index 7ee87c799a..73a2b31027 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -8,7 +8,7 @@ directory, you can generate a repository containing GAPIC client libraries (a monorepo, for example, google-cloud-java) from a configuration file. Instead of running docker image, if you prefer running the underlying python -script directly, please refer to [development guide](DEVELOPMENT.md) for +script directly, please refer to [development guide](DEVELOPMENT.md#run-the-script) for additional instructions. ## Environment @@ -52,7 +52,7 @@ A list of library names that will be generated, separated by comma. The library name of a library is the value of `library_name` or `api_shortname`, if `library_name` is not specified, in the generation configuration. -If `library_names` is not specified, all libraries in the generation +If not specified, all libraries in the generation configuration will be generated. ### Api definitions path (`api-definitions-path`), optional @@ -228,15 +228,34 @@ libraries: --quiet \ -u "$(id -u):$(id -g)" \ -v "$(pwd):/workspace" \ - -v "${api_def_dir}:/workspace/googleapis" \ + -v "${api_def_dir}:/workspace/apis" \ -e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}" \ local:image-tag \ --generation-config-path=/workspace/generation_config_file \ --library-names=apigee-connect,asset \ --repository-path=/workspace \ - --api-definitions-path=/workspace/googleapis + --api-definitions-path=/workspace/apis ``` +* `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating + yourself. + This avoids folder ownership changes since it runs as root by default. +* `-v "$(pwd):/workspace"` maps the host machine's current working directory to + the /workspace folder. + The image is configured to perform changes in this directory. +* `-v "${api_def_dir}:/workspace/apis"` maps the host machine's api-definition + folder to /workspace/apis folder. +* `-e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}"` set environment variable + `GENERATOR_VERSION` in the docker container to use the generator we built in + the previous step. +* `--generation-config-path=/workspace/generation_config_file` set the + generation configuration to `/workspace/generation_config_file`. +* `--api-definitions-path=/workspace/apis` set the API definition path to + `/workspace/apis`. + +To debug the image, please refer to [development guide](DEVELOPMENT.md#debug-the-library-generation-container) +for more info. + ## An example to generate a repository using the docker image If you run the docker image with the example [configuration](#an-example-of-generation-configuration) From 7524b65dbb0f11d21959dbe4d2aa1dae5d92c185 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Mon, 4 Nov 2024 09:43:45 -0500 Subject: [PATCH 06/16] rephase --- hermetic_build/DEVELOPMENT.md | 17 ++++---- hermetic_build/README.md | 43 ++++++++++--------- .../cli/generate_release_note.py | 4 +- 3 files changed, 32 insertions(+), 32 deletions(-) diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md index 80e79476a5..722cfa05bb 100644 --- a/hermetic_build/DEVELOPMENT.md +++ b/hermetic_build/DEVELOPMENT.md @@ -26,7 +26,7 @@ python -m pip install hermetic_build/release_note_generation The integration tests build the docker image declared in `.cloudbuild/library_generation/library_generation.Dockerfile`, pull GAPIC -repositories, generate the libraries and compares the results with the source +repositories, generate the libraries and compare the results with the source code declared in a "golden branch" of the repo. It requires docker and python (>= 3.12.0) to be installed. @@ -40,7 +40,7 @@ python -m unittest hermetic_build/library_generation/tests/integration_tests.py The unit tests of the hermetic build scripts are contained in several scripts, corresponding to a specific component. Every unit test script ends with `unit_tests.py`. -To avoid them specifying them individually, we can use the following command: +To avoid specifying them individually, we can use the following command: ```shell python -m unittest discover -s hermetic_build -p "*unit_tests.py" @@ -52,7 +52,7 @@ python -m unittest discover -s hermetic_build -p "*unit_tests.py" # Run the library generation scripts in your local environment -Although the scripts are designed to be run in a Docker container, you can also +Although the scripts are designed to run in a Docker container, you can also run them directly. This section explains how to run the entrypoint script (`hermetic_build/library_generation/cli/entry_point.py`). @@ -77,8 +77,7 @@ as per [POSIX env var definition](https://pubs.opengroup.org/onlinepubs/96999197 ``` This will generate a jar located in `~/.m2/repository/com/google/api/gapic-generator-java/{version}/gapic-generator-java-{version}.jar` -2. Move the jar into the well-known location of the jar. - The generation scripts will assume the jar is there. +2. Move the jar into its well-known location. ```shell mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar" @@ -88,8 +87,7 @@ as per [POSIX env var definition](https://pubs.opengroup.org/onlinepubs/96999197 1. Download google-java-format-{version}-all-deps.jar from [Maven Central](https://central.sonatype.com/artifact/com.google.googlejavaformat/google-java-format) or [GitHub releases](https://github.com/google/google-java-format/releases). -2. Move the jar into the well-known location of the jar. - The generation scripts will assume the jar is there. +2. Move the jar into its well-known location. ```shell mv /path/to/jar "${HOME}/.library_generation/google-java-format.jar" @@ -102,8 +100,9 @@ need to install beforehand. ### Install the owl-bot CLI -Requires node.js to be installed. -Check this [installation guide](https://github.com/nvm-sh/nvm?tab=readme-ov-file#install--update-script) for NVM, Node.js's version manager. +This requires node.js to be installed. +Check this [installation guide](https://github.com/nvm-sh/nvm?tab=readme-ov-file#install--update-script) +for NVM, Node.js's version manager. After you install it, you can install the owl-bot CLI with the following commands: diff --git a/hermetic_build/README.md b/hermetic_build/README.md index 73a2b31027..f9e16b8990 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -7,16 +7,16 @@ Running the docker image built from `hermetic_build/library_generation` directory, you can generate a repository containing GAPIC client libraries (a monorepo, for example, google-cloud-java) from a configuration file. -Instead of running docker image, if you prefer running the underlying python -script directly, please refer to [development guide](DEVELOPMENT.md#run-the-script) for -additional instructions. +Instead of running the docker image, if you prefer running the underlying python +scripts directly, please refer to the [development guide](DEVELOPMENT.md#run-the-script) +for additional instructions. ## Environment - OS: Linux - Docker -## Prerequisite +## Prerequisites In order to generate a version for each library, a versions.txt has to exist in `repository-path`. @@ -29,7 +29,7 @@ for more information. An absolute or relative path to a configuration file containing parameters to generate the repository. -Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository) +Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository) for more information. ### Repository path (`repository-path`), optional @@ -204,7 +204,7 @@ libraries: . ``` -2. Download api definition to a local directory +2. Download the API definitions to a local directory ```shell api_def_dir=/path/to/api_definition ``` @@ -332,10 +332,10 @@ $repository_path |_versions.txt ``` -# Generate release note from api definition changes +# Generate release note from API definition changes The script, `hermetic_build/release_note_generation/cli/generate_release_note.py` -allows you to generate release note from api definition changes in +allows you to generate release notes from API definition changes in [googleapis](https://github.com/googleapis/googleapis) GitHub repository. ## Environment @@ -348,35 +348,36 @@ allows you to generate release note from api definition changes in ### Baseline generation configuration path (`baseline-generation-config-path`) Absolute or relative path to a generation configuration. -Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository) +Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository) for more information. Note that the `googleapis_commitish` in this configuration is used to retrieve -the first commit, exclusively, to generate the release note. +the first commit, exclusively, to generate the release notes. ### Current generation configuration path (`current-generation-config-path`) Absolute or relative path to a generation configuration. -The release note will be generated from commits that related to the libraries -specified in this configuration. -Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository) +The release notes will be generated from commits that are related to the +libraries specified in this configuration. +Please refer to [Configuration to generate a repository](#configuration-to-generate-a-repository) for more information. -Note that the `googleapis_commitish` in this configuration is used to retrieve -the last commit, inclusively, to generate the release note. +Note that the `googleapis_commitish` entry in this configuration is used to +retrieve the last commit, inclusively, to generate the release notes. ### Repository path (`repository-path`), optional -The path to which the file, `pr_description.txt` containing the release note +The path to which the file, `pr_description.txt` containing the release notes will be sent. If not specified, the file will be generated to the current working directory. -## Generate a release note in local environment +## Generate a release notes file in a local environment -1. Install python (>= 3.12.0), it is recommended to create a python virtual -environment through [official guide](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments). +1. Install python (>= 3.12.0). +It is recommended to create a python virtual environment through the +[official guide](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-and-use-virtual-environments). -2. Running the following commands to install python packages +2. Run the following commands to install python packages ```shell cd /path/to/sdk-platform-java pip install --require-hashes -r hermetic_build/common/requirements.txt @@ -384,7 +385,7 @@ environment through [official guide](https://packaging.python.org/en/latest/guid pip install --require-hashes -r hermetic_build/release_note_generation/requirements.txt pip install hermetic_build/release_note_generation ``` -3. Running the following commands to generate a release note +3. Run the following commands to generate a release note ```shell cd /path/to/sdk-platform-java python hermetic_build/release_note_generation/cli/generate_release_note.py generate \ diff --git a/hermetic_build/release_note_generation/cli/generate_release_note.py b/hermetic_build/release_note_generation/cli/generate_release_note.py index 57113a8ff1..4727640a66 100644 --- a/hermetic_build/release_note_generation/cli/generate_release_note.py +++ b/hermetic_build/release_note_generation/cli/generate_release_note.py @@ -54,8 +54,8 @@ def main(ctx): default=".", show_default=True, help=""" - The path to which the file, `pr_description.txt` containing the release note - will be sent. + The path where the file, `pr_description.txt`, which contains the release + notes will be sent. If not specified, the file will be generated to the current working directory. """, From 64dc16bfa39eda54257e48df9e8a870fa575f985 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Mon, 4 Nov 2024 09:46:38 -0500 Subject: [PATCH 07/16] rephase --- hermetic_build/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/hermetic_build/README.md b/hermetic_build/README.md index f9e16b8990..ad4d8b70f1 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -335,8 +335,9 @@ $repository_path # Generate release note from API definition changes The script, `hermetic_build/release_note_generation/cli/generate_release_note.py` -allows you to generate release notes from API definition changes in -[googleapis](https://github.com/googleapis/googleapis) GitHub repository. +allows you to generate a file containing release notes from API definition +changes in [googleapis](https://github.com/googleapis/googleapis) GitHub +repository. ## Environment From 19ebfb236f3489e9be0421c39a5a4b69beb51082 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Mon, 4 Nov 2024 10:29:15 -0500 Subject: [PATCH 08/16] move build image steps to development guide --- hermetic_build/DEVELOPMENT.md | 41 +++++++++++++++++++++++++++++++ hermetic_build/README.md | 46 ++++++++++------------------------- 2 files changed, 54 insertions(+), 33 deletions(-) diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md index 722cfa05bb..a8a8e7f00b 100644 --- a/hermetic_build/DEVELOPMENT.md +++ b/hermetic_build/DEVELOPMENT.md @@ -156,6 +156,47 @@ python hermetic_build/library_generation/cli/entry_point.py generate \ --api-definitions-path="${api_definitions_path}" ``` +# Build and run the image from source + +1. Run the following command to build the image from source + + ```shell + docker build \ + -f .cloudbuild/library_generation/library_generation.Dockerfile \ + -t local:image-tag \ + . + ``` + +2. Set the version of gapic-generator-java + + ```shell + LOCAL_GENERATOR_VERSION=$(mvn \ + org.apache.maven.plugins:maven-help-plugin:evaluate \ + -Dexpression=project.version \ + -pl gapic-generator-java \ + -DforceStdout \ + -q) + ``` + +3. Run the image + + ```shell + # Assume you want to generate the library in the current working directory + # and the generation configuration is in the same directory. + docker run \ + --rm \ + --quiet \ + -u "$(id -u):$(id -g)" \ + -v "$(pwd):/workspace" \ + -v /path/to/api-definitions:/workspace/apis \ + -e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}" \ + local:image-tag \ + --generation-config-path=/workspace/generation_config_file \ + --library-names=apigee-connect,asset \ + --repository-path=/workspace \ + --api-definitions-path=/workspace/apis + ``` + ## Debug the library generation container If you are working on changing the way the containers are created, you may want to inspect the containers to check the setup. diff --git a/hermetic_build/README.md b/hermetic_build/README.md index ad4d8b70f1..ff36e1e6f6 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -193,33 +193,11 @@ libraries: - proto_path: google/cloud/asset/v1p7beta1 ``` -# Build the image from source +# Run the library generation image -1. Run the following command to build the image from source +1. Download the API definitions to a local directory, e.g., from [googleapis](https://github.com/googleapis/googleapis). - ```shell - docker build \ - -f .cloudbuild/library_generation/library_generation.Dockerfile \ - -t local:image-tag \ - . - ``` - -2. Download the API definitions to a local directory - ```shell - api_def_dir=/path/to/api_definition - ``` - -3. Set the version of gapic-generator-java - ```shell - LOCAL_GENERATOR_VERSION=$(mvn \ - org.apache.maven.plugins:maven-help-plugin:evaluate \ - -Dexpression=project.version \ - -pl gapic-generator-java \ - -DforceStdout \ - -q) - ``` - -4. Run the docker image +2. Run the docker image. ```shell # Assume you want to generate the library in the current working directory # and the generation configuration is in the same directory. @@ -228,9 +206,9 @@ libraries: --quiet \ -u "$(id -u):$(id -g)" \ -v "$(pwd):/workspace" \ - -v "${api_def_dir}:/workspace/apis" \ - -e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}" \ - local:image-tag \ + -v /path/to/api_definition:/workspace/apis \ + -e GENERATOR_VERSION=image-tag \ + gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag \ --generation-config-path=/workspace/generation_config_file \ --library-names=apigee-connect,asset \ --repository-path=/workspace \ @@ -243,11 +221,13 @@ libraries: * `-v "$(pwd):/workspace"` maps the host machine's current working directory to the /workspace folder. The image is configured to perform changes in this directory. -* `-v "${api_def_dir}:/workspace/apis"` maps the host machine's api-definition - folder to /workspace/apis folder. -* `-e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}"` set environment variable - `GENERATOR_VERSION` in the docker container to use the generator we built in - the previous step. +* `-v /path/to/api_definition:/workspace/apis` maps the host machine's API + definitions folder to `/workspace/apis` folder. +* `-e GENERATOR_VERSION=image-tag` set environment variable `GENERATOR_VERSION` + in the docker container to use the generator with the same version of the + image tag. + Note that this is intentional because we tag the image with the underlying + generator version. * `--generation-config-path=/workspace/generation_config_file` set the generation configuration to `/workspace/generation_config_file`. * `--api-definitions-path=/workspace/apis` set the API definition path to From acf40ff680ca462797f23a2a843fb9525584e107 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Mon, 4 Nov 2024 10:31:26 -0500 Subject: [PATCH 09/16] change title --- hermetic_build/DEVELOPMENT.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md index a8a8e7f00b..15aa09219d 100644 --- a/hermetic_build/DEVELOPMENT.md +++ b/hermetic_build/DEVELOPMENT.md @@ -156,7 +156,7 @@ python hermetic_build/library_generation/cli/entry_point.py generate \ --api-definitions-path="${api_definitions_path}" ``` -# Build and run the image from source +# Build the image from source 1. Run the following command to build the image from source @@ -197,7 +197,7 @@ python hermetic_build/library_generation/cli/entry_point.py generate \ --api-definitions-path=/workspace/apis ``` -## Debug the library generation container +# Debug the library generation container If you are working on changing the way the containers are created, you may want to inspect the containers to check the setup. It would be convenient in such case to have a text editor/viewer available. From eba3cb7c6299f07642c48900c60a4dfbf8ba07ae Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Mon, 4 Nov 2024 10:32:11 -0500 Subject: [PATCH 10/16] change shell type --- hermetic_build/DEVELOPMENT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md index 15aa09219d..afa2bbb67a 100644 --- a/hermetic_build/DEVELOPMENT.md +++ b/hermetic_build/DEVELOPMENT.md @@ -203,7 +203,7 @@ to inspect the containers to check the setup. It would be convenient in such case to have a text editor/viewer available. You can achieve this by modifying the Dockerfile as follows: -```docker +```dockerfile # install OS tools RUN apk update && apk add \ unzip curl rsync openjdk11 jq bash nodejs npm git less vim From dee681da2e88e0027157fc6fe55eb61408fb211b Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Mon, 4 Nov 2024 16:20:22 -0500 Subject: [PATCH 11/16] simplify docker run --- hermetic_build/README.md | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/hermetic_build/README.md b/hermetic_build/README.md index ff36e1e6f6..7ada78ac1e 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -61,8 +61,8 @@ The path to where the api definition (proto, service yaml) resides. The default value is the current working directory when running the script. -Note that you need not only the protos defined the service, but also the -transitive dependencies of those protos. +**Note that you need not only the protos defined the service, but also the +transitive dependencies of those protos.** Any missing dependencies will cause `File not found` error. For example, if your service is defined in `example_service.proto` and it @@ -206,13 +206,9 @@ libraries: --quiet \ -u "$(id -u):$(id -g)" \ -v "$(pwd):/workspace" \ - -v /path/to/api_definition:/workspace/apis \ + -v /path/to/api_definition:/workspace \ -e GENERATOR_VERSION=image-tag \ - gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag \ - --generation-config-path=/workspace/generation_config_file \ - --library-names=apigee-connect,asset \ - --repository-path=/workspace \ - --api-definitions-path=/workspace/apis + gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag ``` * `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating @@ -221,17 +217,13 @@ libraries: * `-v "$(pwd):/workspace"` maps the host machine's current working directory to the /workspace folder. The image is configured to perform changes in this directory. -* `-v /path/to/api_definition:/workspace/apis` maps the host machine's API +* `-v /path/to/api_definition:/workspace` maps the host machine's API definitions folder to `/workspace/apis` folder. * `-e GENERATOR_VERSION=image-tag` set environment variable `GENERATOR_VERSION` in the docker container to use the generator with the same version of the image tag. Note that this is intentional because we tag the image with the underlying generator version. -* `--generation-config-path=/workspace/generation_config_file` set the - generation configuration to `/workspace/generation_config_file`. -* `--api-definitions-path=/workspace/apis` set the API definition path to - `/workspace/apis`. To debug the image, please refer to [development guide](DEVELOPMENT.md#debug-the-library-generation-container) for more info. From a68ef4d56166d69ff613d2d0a5e8ef35c7eac4ba Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Mon, 4 Nov 2024 16:52:19 -0500 Subject: [PATCH 12/16] rephase --- hermetic_build/DEVELOPMENT.md | 3 +-- hermetic_build/README.md | 2 +- .../release_note_generation/cli/generate_release_note.py | 4 ++-- 3 files changed, 4 insertions(+), 5 deletions(-) diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md index afa2bbb67a..3520f6d695 100644 --- a/hermetic_build/DEVELOPMENT.md +++ b/hermetic_build/DEVELOPMENT.md @@ -37,8 +37,7 @@ python -m unittest hermetic_build/library_generation/tests/integration_tests.py # Run the unit tests -The unit tests of the hermetic build scripts are contained in several scripts, -corresponding to a specific component. +There is one unit test file per component. Every unit test script ends with `unit_tests.py`. To avoid specifying them individually, we can use the following command: diff --git a/hermetic_build/README.md b/hermetic_build/README.md index 7ada78ac1e..6d435215b0 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -304,7 +304,7 @@ $repository_path |_versions.txt ``` -# Generate release note from API definition changes +# Generate release notes from API definition changes The script, `hermetic_build/release_note_generation/cli/generate_release_note.py` allows you to generate a file containing release notes from API definition diff --git a/hermetic_build/release_note_generation/cli/generate_release_note.py b/hermetic_build/release_note_generation/cli/generate_release_note.py index 4727640a66..a54e228f62 100644 --- a/hermetic_build/release_note_generation/cli/generate_release_note.py +++ b/hermetic_build/release_note_generation/cli/generate_release_note.py @@ -54,8 +54,8 @@ def main(ctx): default=".", show_default=True, help=""" - The path where the file, `pr_description.txt`, which contains the release - notes will be sent. + The path where the file `pr_description.txt`, which contains the release + notes, will be sent. If not specified, the file will be generated to the current working directory. """, From 244f82a7de3c80988127f1c5dcc1c717d8f8311e Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Mon, 4 Nov 2024 17:02:50 -0500 Subject: [PATCH 13/16] remove generator env in docker run --- hermetic_build/README.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/hermetic_build/README.md b/hermetic_build/README.md index 6d435215b0..51322ddc4a 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -207,7 +207,6 @@ libraries: -u "$(id -u):$(id -g)" \ -v "$(pwd):/workspace" \ -v /path/to/api_definition:/workspace \ - -e GENERATOR_VERSION=image-tag \ gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag ``` @@ -219,11 +218,6 @@ libraries: The image is configured to perform changes in this directory. * `-v /path/to/api_definition:/workspace` maps the host machine's API definitions folder to `/workspace/apis` folder. -* `-e GENERATOR_VERSION=image-tag` set environment variable `GENERATOR_VERSION` - in the docker container to use the generator with the same version of the - image tag. - Note that this is intentional because we tag the image with the underlying - generator version. To debug the image, please refer to [development guide](DEVELOPMENT.md#debug-the-library-generation-container) for more info. From d31dc36a61ac8fc88e0d73f3c5def5c39e95d55f Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Tue, 5 Nov 2024 08:06:48 -0500 Subject: [PATCH 14/16] add an advanced example of docker run --- hermetic_build/README.md | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-) diff --git a/hermetic_build/README.md b/hermetic_build/README.md index 51322ddc4a..a729fd1c36 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -210,14 +210,34 @@ libraries: gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag ``` -* `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating - yourself. - This avoids folder ownership changes since it runs as root by default. -* `-v "$(pwd):/workspace"` maps the host machine's current working directory to - the /workspace folder. - The image is configured to perform changes in this directory. -* `-v /path/to/api_definition:/workspace` maps the host machine's API - definitions folder to `/workspace/apis` folder. + * `-u "$(id -u)":"$(id -g)"` makes docker run the container impersonating + yourself. + This avoids folder ownership changes since it runs as root by default. + * `-v "$(pwd):/workspace"` maps the host machine's current working directory + to the /workspace folder. + The image is configured to perform changes in this directory. + * `-v /path/to/api_definition:/workspace` maps the host machine's API + definitions folder to `/workspace/apis` folder. + +3. An advanced example: + ```shell + docker run \ + --rm \ + --quiet \ + -u "$(id -u):$(id -g)" \ + -v "$(pwd):/workspace" \ + -v /path/to/api_definition:/workspace/apis \ + gcr.io/cloud-devrel-public-resources/java-library-generation:image-tag \ + --generation-config-path=/workspace/generation_config_file \ + --library-names=apigee-connect,asset \ + --repository-path=/workspace \ + --api-definitions-path=/workspace/apis + ``` + + * `--generation-config-path=/workspace/generation_config_file` set the + generation configuration to `/workspace/generation_config_file`. + * `--api-definitions-path=/workspace/apis` set the API definition path to + `/workspace/apis`. To debug the image, please refer to [development guide](DEVELOPMENT.md#debug-the-library-generation-container) for more info. From a7b585f429f57eb8652cb3c3be507988d8767e58 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Tue, 5 Nov 2024 08:06:58 -0500 Subject: [PATCH 15/16] add a note --- hermetic_build/DEVELOPMENT.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/hermetic_build/DEVELOPMENT.md b/hermetic_build/DEVELOPMENT.md index 3520f6d695..b6e78389dc 100644 --- a/hermetic_build/DEVELOPMENT.md +++ b/hermetic_build/DEVELOPMENT.md @@ -195,6 +195,11 @@ python hermetic_build/library_generation/cli/entry_point.py generate \ --repository-path=/workspace \ --api-definitions-path=/workspace/apis ``` + Note that if you specify the generator version using environment variable, + `-e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}"` in the above example, + you should not set `gapic_generator_version` and `protoc_version` in the + generation configuration because values in the generation configuration will + take precedence. # Debug the library generation container If you are working on changing the way the containers are created, you may want From df96948dd050903fa072a6a0920cdb6b234c8327 Mon Sep 17 00:00:00 2001 From: Joe Wang Date: Tue, 5 Nov 2024 10:47:34 -0500 Subject: [PATCH 16/16] restore --- hermetic_build/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hermetic_build/README.md b/hermetic_build/README.md index a729fd1c36..583e766bdf 100644 --- a/hermetic_build/README.md +++ b/hermetic_build/README.md @@ -137,7 +137,7 @@ They are shared by all GAPICs of a library. | distribution_name | No | `{group_id}:google-{cloud_prefix}{library_name}` if not specified | | excluded_poms | No | | | excluded_dependencies | No | | -| googleapis_commitish | No | use repository level `googleapis_committish` if not specified. | +| googleapis_commitish | No | use repository level `googleapis_commitish` if not specified. | | group_id | No | `com.google.cloud` if not specified | | issue_tracker | No | | | library_name | No | `api_shortname` is not specified. This value should be unique among all libraries. |