feat: move Java Owlbot into this repository for postprocessing (#2282)

* use local synthtool and owlbot

* remove unused files

* remove more unused files

* remove cache files in owlbot

* use java 11 for it

* remove kokoro files

* use glob in owlbot entrypoint

* remove unused files

* do not do post-process IT on mac

* concise entrypoint logic

* cleanup i

* cleanup ii

* cleanup iii

* cleanup iv

* remove templates

* remove protos folder

* remove synthtool

* connect image to owlbot entrypoint

* simplify synthtool docker run command

* install synthtool locally

* install synthtool only once

* use virtualenvs to run python scripts

* install pyenv in action

* remove jar from history

* download google-java-format

* fix pyenv init

* attempt to fix pyenv installation in gh action

* fix manual pyenv installation

* install pyenv in profile

* install pyenv in bashrc as well

* use bash shell explicitly in gh action

* install pyenv in same step as IT

* do not restart shell

* set pyenv path manually

* install pyenv in its own step

* propagate environment variables to other steps

* fix global env var setup

* remove wrong env settings

* explain usage of pyenv in README

* simplify pyenv setup

* add comment to owlbot entrypoint

* rename destination_path to preprocessed_libraries_path

* infer scripts_root in postprocess_library.sh

* use temporary folder for preprocess step

* use owlbot files from workspace

* get rid of output_folder argument

* use common temp dir to clone synthtool into

* lock synthtool to a specific commitish

* fix file transfer

* fix owl-bot-staging unpacking

* remove unnecessary workspace variable

* rename workspace to postprocessing_target

* remove owlbot sha logic

* remove repository_root variable

* cleanup

* correct pyenv comment

* clean temp sources folder on each run

* safety checks for get_proto_path_from_preprocessed_sources

* fix integration test

* disable compute and asset/v1p2beta1 temporarily

they have changes in googleapis that have not been reflected yet in
google-cloud-java

* fix unit tests

* correct comment

* do not install docker for macos

* fix owlbot files check

* fix license headers

* remove unnecessary owlbot_sha

* add explanation on why are there no macos + postprocess ITs

* use `fmt:format` instead of google formatter

* clean templates

* remove more unnecessary elements

* add README entry explaining owlbot maintenance

* remove unnecessary java format version

Author: diegomarquezp

.github/workflows/verify_library_generation.yaml

@@ -12,7 +12,7 @@ jobs:
   integration_tests:
     strategy:
       matrix:
-        java: [ 8 ]
+        java: [ 11 ]
         os: [ ubuntu-22.04, macos-12 ]
         post_processing: [ 'true', 'false' ]
     runs-on: ${{ matrix.os }}
@@ -26,8 +26,23 @@ jobs:
     - uses: actions/setup-python@v4
       with:
         python-version: '3.11'
+    - name: install pyenv
+      shell: bash
+      run: |
+        set -ex
+        curl https://pyenv.run | bash
+        # setup environment
+        export PYENV_ROOT="$HOME/.pyenv"
+        export PATH="$PYENV_ROOT/bin:$PATH"
+        echo "PYENV_ROOT=${PYENV_ROOT}" >> $GITHUB_ENV
+        echo "PATH=${PATH}" >> $GITHUB_ENV
+        # init pyenv
+        eval "$(pyenv init --path)"
+        eval "$(pyenv init -)"
+        set +ex
     - name: install docker (ubuntu)
       if: matrix.os == 'ubuntu-22.04'
+      shell: bash
       run: |
         set -x
         # install docker
@@ -36,17 +51,11 @@ jobs:
 
         # launch docker
         sudo systemctl start docker
-    - name: install docker (macos)
-      if: matrix.os == 'macos-12'
-      run: |
-        brew update --preinstall
-        brew install docker docker-compose qemu 
-        brew upgrade qemu
-        colima start
-        docker run --user $(id -u):$(id -g) --rm hello-world
     - name: Run integration tests
+      # we don't run ITs with postprocessing on macos because one of its dependencies "synthtool" is designed to run on linux only
+      if: matrix.os == 'ubuntu-22.04' || matrix.post_processing == 'false'
+      shell: bash
       run: |
-        set -x
         git config --global user.email "[email protected]"
         git config --global user.name "Github Workflow"
         library_generation/test/generate_library_integration_test.sh \

library_generation/README.md

@@ -1,4 +1,4 @@
-# Generate GAPIC Client Library without post-processing
+# Generate GAPIC Client Library with and without post-processing
 
 The script, `generate_library.sh`, allows you to generate a GAPIC client library from proto files.
 
@@ -28,6 +28,12 @@ This repository will be the source of truth for pre-existing
 pom.xml files, owlbot.py and .OwlBot.yaml files. See the option belows for
 custom postprocessed generations (e.g. custom `versions.txt` file).
 
+Post-processing makes use of python scripts. The script will automatically use
+`pyenv` to use the specified version in
+`library_generation/configuration/python-version`. Pyenv is then a requirement
+in the environment.
+
+
 ## Parameters to run `generate_library.sh`
 
 You need to run the script with the following parameters.
@@ -225,3 +231,26 @@ library_generation/generate_library.sh \
 --versions_file "path/to/versions.txt" \
 --include_samples true
 ```
+
+# Owlbot Java Postprocessor
+
+We have transferred the
+[implementation](https://github.com/googleapis/synthtool/tree/59fe44fde9866a26e7ee4e4450fd79f67f8cf599/docker/owlbot/java)
+of Java Owlbot Postprocessor into `sdk-platform-java/library_generation`. The
+implementation in synthtool is still valid and used by other services, so we
+have two versions during a transition period.
+
+## Reflecting changes in synthtool/docker/owlbot/java into this repository
+The transfer was not a verbatim copy, it rather had modifications:
+ * `format-source.sh` was replaced by a call to `mvn fmt:format`
+ * `entrypoint.sh` was modified to have input arguments and slightly modified
+   the way the helper scripts are called
+ * Other helper scripts were modified to have input arguments.
+ * `fix-poms.py` modified the way the monorepo is detected
+
+All these modifications imply that whenever we want to reflect a change from the
+original owlbot in synthtool we may be better off modifying the affected source
+files one by one. The mapping is from
+[`synthtool/docker/owlbot/java`](https://github.com/googleapis/synthtool/tree/59fe44fde9866a26e7ee4e4450fd79f67f8cf599/docker/owlbot/java)
+to
+[`sdk-platform-java/library_generation/owlbot`](https://github.com/googleapis/sdk-platform-java/tree/move-java-owlbot/library_generation/owlbot)

library_generation/configuration/python-version

@@ -0,0 +1 @@
+3.11.2

library_generation/configuration/synthtool-commitish

@@ -0,0 +1 @@
+59fe44fde9866a26e7ee4e4450fd79f67f8cf599

library_generation/generate_library.sh

@@ -133,7 +133,13 @@ if [ -z "${os_architecture}" ]; then
   os_architecture=$(detect_os_architecture)
 fi
 
+temp_destination_path="${output_folder}/temp_preprocessed"
 mkdir -p "${output_folder}/${destination_path}"
+if [ -d "${temp_destination_path}" ]; then
+  # we don't want the preprocessed sources of a previous run
+  rm -rd "${temp_destination_path}"
+fi
+mkdir -p "${temp_destination_path}"
 ##################### Section 0 #####################
 # prepare tooling
 #####################################################
@@ -185,30 +191,30 @@ download_tools "${gapic_generator_version}" "${protobuf_version}" "${grpc_versio
 if [[ ! "${transport}" == "rest" ]]; then
   # do not need to generate grpc-* if the transport is `rest`.
   "${protoc_path}"/protoc "--plugin=protoc-gen-rpc-plugin=protoc-gen-grpc-java-${grpc_version}-${os_architecture}.exe" \
-  "--rpc-plugin_out=:${destination_path}/java_grpc.jar" \
+  "--rpc-plugin_out=:${temp_destination_path}/java_grpc.jar" \
   ${proto_files} # Do not quote because this variable should not be treated as one long string.
   # unzip java_grpc.jar to grpc-*/src/main/java
-  unzip_src_files "grpc"
+  unzip_src_files "grpc" "${temp_destination_path}"
   # remove empty files in grpc-*/src/main/java
-  remove_empty_files "grpc"
+  remove_empty_files "grpc" "${temp_destination_path}"
   # remove grpc version in *ServiceGrpc.java file so the content is identical with bazel build.
-  remove_grpc_version
+  remove_grpc_version "${temp_destination_path}"
 fi
 ###################### Section 2 #####################
 ## generate gapic-*/, part of proto-*/, samples/
 ######################################################
 if [[ "${proto_only}" == "false" ]]; then
   "$protoc_path"/protoc --experimental_allow_proto3_optional \
   "--plugin=protoc-gen-java_gapic=${script_dir}/gapic-generator-java-wrapper" \
-  "--java_gapic_out=metadata:${destination_path}/java_gapic_srcjar_raw.srcjar.zip" \
+  "--java_gapic_out=metadata:${temp_destination_path}/java_gapic_srcjar_raw.srcjar.zip" \
   "--java_gapic_opt=$(get_gapic_opts "${transport}" "${rest_numeric_enums}" "${gapic_yaml}" "${service_config}" "${service_yaml}")" \
   ${proto_files} ${gapic_additional_protos}
 
-  unzip -o -q "${destination_path}/java_gapic_srcjar_raw.srcjar.zip" -d "${destination_path}"
+  unzip -o -q "${temp_destination_path}/java_gapic_srcjar_raw.srcjar.zip" -d "${temp_destination_path}"
   # Sync'\''d to the output file name in Writer.java.
-  unzip -o -q "${destination_path}/temp-codegen.srcjar" -d "${destination_path}/java_gapic_srcjar"
+  unzip -o -q "${temp_destination_path}/temp-codegen.srcjar" -d "${temp_destination_path}/java_gapic_srcjar"
   # Resource name source files.
-  proto_dir=${destination_path}/java_gapic_srcjar/proto/src/main/java
+  proto_dir=${temp_destination_path}/java_gapic_srcjar/proto/src/main/java
   if [ ! -d "${proto_dir}" ]; then
     # Some APIs don't have resource name helpers, like BigQuery v2.
     # Create an empty file so we can finish building. Gating the resource name rule definition
@@ -218,14 +224,14 @@ if [[ "${proto_only}" == "false" ]]; then
     touch "${proto_dir}"/PlaceholderFile.java
   fi
   # move java_gapic_srcjar/src/main to gapic-*/src.
-  mv_src_files "gapic" "main"
+  mv_src_files "gapic" "main" "${temp_destination_path}"
   # remove empty files in gapic-*/src/main/java
-  remove_empty_files "gapic"
+  remove_empty_files "gapic" "${temp_destination_path}"
   # move java_gapic_srcjar/src/test to gapic-*/src
-  mv_src_files "gapic" "test"
+  mv_src_files "gapic" "test" "${temp_destination_path}"
   if [ "${include_samples}" == "true" ]; then
     # move java_gapic_srcjar/samples/snippets to samples/snippets
-    mv_src_files "samples" "main"
+    mv_src_files "samples" "main" "${temp_destination_path}"
   fi
 fi
 ##################### Section 3 #####################
@@ -247,16 +253,16 @@ case "${proto_path}" in
     proto_files="${proto_files//${removed_proto}/}"
     ;;
 esac
-"$protoc_path"/protoc "--java_out=${destination_path}/java_proto.jar" ${proto_files}
+"$protoc_path"/protoc "--java_out=${temp_destination_path}/java_proto.jar" ${proto_files}
 if [[ "${proto_only}" == "false" ]]; then
   # move java_gapic_srcjar/proto/src/main/java (generated resource name helper class)
   # to proto-*/src/main
-  mv_src_files "proto" "main"
+  mv_src_files "proto" "main" "${temp_destination_path}"
 fi
 # unzip java_proto.jar to proto-*/src/main/java
-unzip_src_files "proto"
+unzip_src_files "proto" "${temp_destination_path}"
 # remove empty files in proto-*/src/main/java
-remove_empty_files "proto"
+remove_empty_files "proto" "${temp_destination_path}"
 case "${proto_path}" in
   "google/cloud/aiplatform/v1beta1"*)
     prefix="google/cloud/aiplatform/v1beta1/schema"
@@ -282,14 +288,14 @@ for proto_src in ${proto_files}; do
   if [[ "${proto_src}" == "google/cloud/common/operation_metadata.proto" ]]; then
     continue
   fi
-  mkdir -p "${destination_path}/proto-${folder_name}/src/main/proto"
-  rsync -R "${proto_src}" "${destination_path}/proto-${folder_name}/src/main/proto"
+  mkdir -p "${temp_destination_path}/proto-${folder_name}/src/main/proto"
+  rsync -R "${proto_src}" "${temp_destination_path}/proto-${folder_name}/src/main/proto"
 done
 popd # output_folder
 ##################### Section 4 #####################
 # rm tar files
 #####################################################
-pushd "${output_folder}/${destination_path}"
+pushd "${temp_destination_path}"
 rm -rf java_gapic_srcjar java_gapic_srcjar_raw.srcjar.zip java_grpc.jar java_proto.jar temp-codegen.srcjar
 popd # destination path
 ##################### Section 5 #####################
@@ -298,6 +304,8 @@ popd # destination path
 if [ "${enable_postprocessing}" != "true" ];
 then
   echo "post processing is disabled"
+  cp -r ${temp_destination_path}/* "${output_folder}/${destination_path}"
+  rm -rdf "${temp_destination_path}"
   exit 0
 fi
 if [ -z "${versions_file}" ];then
@@ -311,21 +319,13 @@ fi
 
 mkdir -p "${workspace}"
 
+# if destination_path is not empty, it will be used as a starting workspace for
+# postprocessing
+if [[ $(find "${output_folder}/${destination_path}" -mindepth 1 -maxdepth 1 -type d,f | wc -l) -gt 0 ]];then
+  workspace="${output_folder}/${destination_path}"
+fi
+
 bash -x "${script_dir}/postprocess_library.sh" "${workspace}" \
-  "${script_dir}" \
-  "${destination_path}" \
-  "${proto_path}" \
-  "${versions_file}" \
-  "${output_folder}"
+  "${temp_destination_path}" \
+  "${versions_file}"
 
-# for post-procesed libraries, remove pre-processed folders
-pushd "${output_folder}/${destination_path}"
-rm -rdf "proto-${folder_name}"
-rm -rdf "grpc-${folder_name}"
-rm -rdf "gapic-${folder_name}"
-if [ "${include_samples}" == "false" ]; then
-  rm -rdf "samples"
-fi
-popd # output_folder
-# move contents of the post-processed library into destination_path
-cp -r ${workspace}/* "${output_folder}/${destination_path}"

library_generation/owlbot/bin/entrypoint.sh

@@ -0,0 +1,113 @@
+#!/bin/bash
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This is the entrypoint script for java owlbot. This is not intended to be
+# called directly but rather be called from postproces_library.sh
+# For reference, the positional arguments are
+# 1: scripts_root: location of postprocess_library.sh
+# 2: versions_file: points to a versions.txt containing versions to be applied
+# both to README and pom.xml files
+
+# The scripts assumes the CWD is the folder where postprocessing is going to be
+# applied
+
+set -ex
+scripts_root=$1
+versions_file=$2
+
+# Runs template and etc in current working directory
+function processModule() {
+  # templates as well as retrieving files from owl-bot-staging
+  echo "Generating templates and retrieving files from owl-bot-staging directory..."
+  if [ -f "owlbot.py" ]
+  then
+    # defaults to run owlbot.py
+    python3 owlbot.py
+  fi
+  echo "...done"
+
+  # write or restore pom.xml files
+  echo "Generating missing pom.xml..."
+  python3 "${scripts_root}/owlbot/src/fix-poms.py" "${versions_file}" "true"
+  echo "...done"
+
+  # write or restore clirr-ignored-differences.xml
+  echo "Generating clirr-ignored-differences.xml..."
+  ${scripts_root}/owlbot/bin/write_clirr_ignore.sh "${scripts_root}"
+  echo "...done"
+
+  # fix license headers
+  echo "Fixing missing license headers..."
+  python3 "${scripts_root}/owlbot/src/fix-license-headers.py"
+  echo "...done"
+
+  # TODO: re-enable this once we resolve thrashing
+  # restore license headers years
+  # echo "Restoring copyright years..."
+  # /owlbot/bin/restore_license_headers.sh
+  # echo "...done"
+
+  # ensure formatting on all .java files in the repository
+  echo "Reformatting source..."
+  mvn fmt:format
+  echo "...done"
+}
+
+if [ "$(ls */.OwlBot.yaml|wc -l)" -gt 1 ];then
+  # Monorepo (googleapis/google-cloud-java) has multiple OwlBot.yaml config
+  # files in the modules.
+  echo "Processing monorepo"
+  if [ -d owl-bot-staging ]; then
+    # The content of owl-bot-staging is controlled by Owlbot.yaml files in
+    # each module in the monorepo
+    echo "Extracting contents from owl-bot-staging"
+    for module in owl-bot-staging/* ; do
+      if [ ! -d "$module" ]; then
+        continue
+      fi
+      # This relocation allows us continue to use owlbot.py without modification
+      # after monorepo migration.
+      mv "owl-bot-staging/$module" "$module/owl-bot-staging"
+      pushd "$module"
+      processModule
+      popd
+    done
+    rm -r owl-bot-staging
+  else
+    echo "In monorepo but no owl-bot-staging." \
+        "Formatting changes in the last commit"
+    # Find the files that were touched by the last commit.
+    last_commit=$(git log -1 --format=%H)
+    # [A]dded, [C]reated, [M]odified, and [R]enamed
+    changed_files=$(git show --name-only --no-renames --diff-filter=ACMR \
+        "${last_commit}")
+    changed_modules=$(echo "$changed_files" |grep -E '.java$' |cut -d '/' -f 1 \
+        |sort -u)
+    for module in ${changed_modules}; do
+      if [ ! -f "$module/.OwlBot.yaml" ]; then
+        # Changes irrelevant to Owlbot-generated module (such as .github) do not
+        # need formatting
+        continue
+      fi
+      pushd "$module"
+      processModule
+      popd
+    done
+  fi
+else
+  # Split repository
+  echo "Processing a split repo"
+  processModule
+fi