SQL-2544: Merge on-prem-eap branch into master

.gitignore

@@ -14,3 +14,9 @@ local_adf
 
 # Ignore generated tests
 resources/generated_test
+
+# Ignore mongosqltranslate libraries
+src/main/resources/**/libmongosqltranslate.*
+src/main/resources/**/mongosqltranslate.*
+.library_cache/
+

README.md

@@ -114,6 +114,27 @@ You can find the generated jar in build/libs/
 ```
 ./gradlew spotlessApply
 ```
+### Downloading mongosqltranslate Library
+The driver requires the `mongosqltranslate` library for direct cluster SQL translation.  For initial gradle builds the
+library files will be automatically downloaded to the cache directory `${project.rootDir}/.library_cache/`.
+If a specific version of the library already exists in the cache, it will be used. 
+#### Specifying library version:
+By default, the `snapshot` version will be downloaded.  To use a specific version, use the libmongosqltranslateVersion 
+property:
+```bash
+./gradlew clean build -PlibmongosqltranslateVersion=1.0.0-beta-1
+```
+#### Force Re-download:
+Unless the updateLibs property is set to true, cached versions will be used to avoid repeated downloads.  
+```bash
+# Override cached versions
+./gradlew clean build -PupdateLibs=true
+```
+#### Run download task only
+```bash
+./gradlew downloadLibMongosqlTranslate
+```
+
 ## Integration Testing
 Integration testing requires a local MongoDB and Atlas Data Federation instance to be running
 #### Environment Variables

RELEASE.md

@@ -1,25 +1,43 @@
 # Versioning
+
+## Versioning
+
+Versions will follow the [semantic versioning](https://semver.org/) system.
+
+The following guidelines will be used to determine when each version component will be updated:
+- **major**: backwards-breaking changes
+- **minor**: functionality added in a backwards compatible manner
+- **patch**: backwards compatible bug fixes.
+- **pre-release**: The pre-release version, used for preview builds and when updating a preview version of `libmongosqltranslate`
+- **libv**: to specify which version of `libmongosqltranslate` gets bundled with the driver during a release
+  - Our build system will fetch *exactly* the version of `libmongosqltranslate` specified after `libv` and bundle it with the driver. If you are unsure
+  what version to use, check the [releases](https://jira.mongodb.org/projects/SQL?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page&status=released&contains=libv) page and use the latest **released** tag.
+
+### The following applies to non-eap releases
+
 Version number policy could be referred from here: https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN8855
 
 Among all the version standards Maven supports, we will use MajorVersion, IncrementalVersion and Qualifier inside this project.
 
-The current version number is specified inside `gradle.properties` file. 
-
-# Release Process
+## Release Process
 
-## Snapshot Versions
+### Snapshot Versions
 
 Every successful untagged build in evergreen will release the artifacts to the Sonatype SNAPSHOT repo in https://oss.sonatype.org/#view-repositories;snapshots~browsestorage
 
-## Release Versions
+### Release Versions
 
 Follow these instructions for creating a Release Version.
 
 ### Pre-Release Tasks
 
 #### Determine the correct version to be released
+
 Go to the [SQL releases page](https://jira.mongodb.org/projects/SQL?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page&status=unreleased), and check the content of the tickets that are included in the current release. The fix version by default is a patch version. If there is a backwards incompatible API change in the tickets that are set to be released, we should instead update the major version; if there are new features added in the tickets set to be released, we should instead update the minor version. To do so, update the version on the [SQL releases page](https://jira.mongodb.org/projects/SQL?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page&status=unreleased) under "Actions". This will update the fix version on all of the tickets as well.
 
+To determine which version of `libmongosqltranslate` to bundle, go to the [releases](https://jira.mongodb.org/projects/SQL?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page&status=released&contains=libv) page
+and choose the latest **released** version. Add this fix version to the release ticket in JIRA.
+
 #### Start Release Ticket
 Move the JIRA ticket for the release to the "In Progress" state.
 Ensure that its fixVersion matches the version being released, and update it if it changed in the previous step.
@@ -40,11 +58,22 @@ Close the release on JIRA, adding the current date (you may need to ask the SQL
 Ensure you have the `master` branch checked out, and that you have pulled the latest commit from `mongodb/mongo-jdbc-driver`.
 
 #### Create the tag and push
-Create an annotated tag and push it:
-```
-git tag -a -m X.Y.Z vX.Y.Z
+
+The tag is formatted as follows:
+
+`<JDBC major>.<JDBC minor>.<JDBC patch>[-<pre-release>]-libv<libv major>.<libv minor>.<libv patch>[-<pre-release>]`
+
+To specify which version of [`libmongosqltranslate`](https://github.com/10gen/mongosql-rs) should be built with the release, the version
+of `libmongosqltranslate` must be annotated in the git tag with `libv`. This version was obtained previously when updating the release ticket.
+
+Create an annotated tag and push it (using the correct versions):
+
+```sh
+# git tag -a -m X.Y.Z[-<JDBCprerelease>-libvXX.YY.ZZ[-<prerelease>]]
+git tag -am 3.0.0-alpha-2-libv1.0.0-alpha-3
 git push --tags
 ```
+
 This should trigger an Evergreen version that can be viewed on the [mongo-jdbc-driver waterfall](https://evergreen.mongodb.com/waterfall/mongo-jdbc-driver).
 If it does not, you may have to ask the project manager to give you the right permissions to do so.
 Make sure to run the 'release' task, if it is not run automatically.
@@ -60,12 +89,12 @@ priority.
 #### Wait for evergreen
 Wait for the evergreen version to finish, and ensure that the release task completes successfully.
 
-#### Verify release artifacts
-Check that the version just released is available in the [Sonatype Nexus Repo Manager](https://oss.sonatype.org/#nexus-search;quick~mongodb-jdbc).  
+#### Verify release artifacts (ignore for EAP)
+Check that the version just released is available in the [Sonatype Nexus Repo Manager](https://oss.sonatype.org/#nexus-search;quick~mongodb-jdbc).
 The release artifacts should appear on [Maven Central](https://search.maven.org/search?q=g:org.mongodb%20AND%20a:mongodb-jdbc) after a while.
 
-#### Notify the Web team about the new release
-Create a ticket through the [service desk](https://jira.mongodb.org/plugins/servlet/desk/portal/61/create/926) and request the link for the `Download from Maven` button on the Download Center page `https://www.mongodb.com/try/download/jdbc-driver` to be updated.  
+#### Notify the Web team about the new release (ignore for EAP)
+Create a ticket through the [service desk](https://jira.mongodb.org/plugins/servlet/desk/portal/61/create/926) and request the link for the `Download from Maven` button on the Download Center page `https://www.mongodb.com/try/download/jdbc-driver` to be updated.
 You can find the new link in the [json feed](https://translators-connectors-releases.s3.amazonaws.com/mongo-jdbc-driver/mongo-jdbc-downloads.json) under `versions[0].download_link`. Include this link in your ticket.
 
 #### Close Release Ticket

build.gradle

@@ -12,6 +12,8 @@ import com.github.jk1.license.render.TextReportRenderer
 
 if (project.hasProperty('isTagTriggered')) {
     version = getAbbreviatedGitVersion()
+} else if (project.hasProperty('isEapBuild')){
+    version = getLatestEapTag()
 } else {
     version = getGitVersion()
 }
@@ -20,6 +22,7 @@ ext {
     println("Driver version = " + version)
     releaseVersion = getReleaseVersion()
     println("Artifacts version = " + releaseVersion)
+
     javaDataLoader = "com.mongodb.jdbc.integration.testharness.DataLoader"
     javaTestGenerator = "com.mongodb.jdbc.integration.testharness.TestGenerator"
     aspectjVersion = '1.9.7'
@@ -168,8 +171,9 @@ task integrationTest(type: Test) {
     description = 'Runs integration tests.'
     group = 'verification'
 
+    dependsOn tasks.named('jar')
     testClassesDirs = sourceSets.integrationTest.output.classesDirs
-    classpath = sourceSets.integrationTest.runtimeClasspath
+    classpath = sourceSets.integrationTest.runtimeClasspath + files(tasks.jar.archiveFile.get()) - sourceSets.main.output
     shouldRunAfter test
 }
 
@@ -270,6 +274,16 @@ def getGitVersion() {
     out.toString().substring(1).trim()
 }
 
+def getLatestEapTag() {
+    def stdout = new ByteArrayOutputStream()
+    exec {
+        commandLine 'git', 'tag', '--list', '*libv*', '--sort=-creatordate'
+        standardOutput = stdout
+    }
+    def tags = stdout.toString().trim().readLines()
+    return tags ? tags[0].toString().substring(1).trim() : null
+}
+
 def getAbbreviatedGitVersion() {
     def out = new ByteArrayOutputStream()
     exec {
@@ -279,6 +293,110 @@ def getAbbreviatedGitVersion() {
     out.toString().substring(1).trim()
 }
 
+// Determines the version of libmongosqltranslate to use based on the following priority:
+// Command line property 'libVersion' ie -PlibmongosqltranslateVersion=1.2.3 for manual testing
+// If build is triggered by a tag, check that LIBMONGOSQLTRANSLATE_VER environment variable is set and use that value
+// Otherwise default to "snapshot" version
+def getLibMongosqlTranslateVersion() {
+    if (project.hasProperty('libmongosqltranslateVersion')) {
+        logger.lifecycle("Using manually specified libVersion: ${project.property('libmongosqltranslateVersion')}")
+        return project.property('libmongosqltranslateVersion')
+    }
+    if (project.hasProperty('isTagTriggered')) {
+        if (System.getenv('LIBMONGOSQLTRANSLATE_VER')) {
+            logger.lifecycle("Using version from environment: ${System.getenv('LIBMONGOSQLTRANSLATE_VER')}")
+            return System.getenv('LIBMONGOSQLTRANSLATE_VER')
+        } else {
+            throw new GradleException("Build is tag-triggered but LIBMONGOSQLTRANSLATE_VER " +
+                    "environment variable is not set. This is required for tag-triggered builds.")
+        }
+    }
+    logger.lifecycle("Using snapshot version")
+    return "snapshot"
+}
+
+def libraryCache = new File("${project.rootDir}/.library_cache")
+
+task downloadLibMongosqlTranslate {
+    def libraryPlatforms = [
+            [platform: 'linux', arch: 'arm', libPrefix: 'lib', ext: 'so'],
+            [platform: 'linux', arch: 'x86_64', libPrefix: 'lib', ext: 'so'],
+            [platform: 'macos', arch: 'arm', libPrefix: 'lib', ext: 'dylib'],
+            [platform: 'macos', arch: 'x86_64', libPrefix: 'lib', ext: 'dylib'],
+            [platform: 'win', arch: 'x86_64', libPrefix: '', ext: 'dll']
+    ]
+    description = 'Downloads mongosqltranslate libraries for all platforms'
+    group = 'Build Setup'
+
+    // Read the force-update flag from the command line `-PupdateLibs=true`
+    def updateLibs = project.hasProperty('updateLibs') ? project.property('updateLibs').toBoolean() : false
+    doLast {
+        def libVersion = getLibMongosqlTranslateVersion()
+        logger.lifecycle("Using libmongosqltranslate version: ${libVersion}")
+
+        libraryCache.mkdirs()
+
+        libraryPlatforms.each { platform ->
+            def libraryFileName = "${platform.libPrefix}mongosqltranslate.${platform.ext}"
+            def s3FileName =
+                    "${platform.libPrefix}mongosqltranslate-v${libVersion}-${platform.platform}-${platform.arch}.${platform.ext}"
+            def s3Url = "https://translators-connectors-releases.s3.amazonaws.com/mongosqltranslate/${s3FileName}"
+
+            def cacheFile = new File(libraryCache, s3FileName)
+            def resourceDir = new File("${project.rootDir}/src/main/resources/${platform.arch}/${platform.platform}")
+            resourceDir.mkdirs()
+
+            def destinationFile = new File(resourceDir, libraryFileName)
+
+            // Skip the download if the force-update flag is not set and the library already exists in library cache
+            if (!updateLibs && cacheFile.exists() && cacheFile.length() > 0) {
+                logger.lifecycle("Using cached version of ${s3FileName} for ${platform.platform}-${platform.arch}")
+                destinationFile.bytes = cacheFile.bytes
+                return
+            }
+
+            try {
+                logger.lifecycle("Downloading ${s3Url}...")
+
+                def connection = new URL(s3Url).openConnection()
+                connection.connectTimeout = 30000
+                connection.readTimeout = 30000
+
+                cacheFile.withOutputStream { outputStream ->
+                    connection.getInputStream().withCloseable { inputStream ->
+                        outputStream << inputStream
+                    }
+                }
+
+                // Verify we downloaded actual content
+                if (cacheFile.length() == 0) {
+                    throw new IOException("Downloaded file is empty")
+                }
+
+                destinationFile.bytes = cacheFile.bytes
+                logger.lifecycle("Successfully downloaded ${s3FileName} for ${platform.platform}-${platform.arch}")
+
+            } catch (Exception e) {
+                logger.warn("Could not download ${s3FileName}: ${e.message}")
+
+                if (cacheFile.exists() && cacheFile.length() > 0) {
+                    logger.lifecycle("Using cached version from ${cacheFile.path}")
+                    destinationFile.bytes = cacheFile.bytes
+                } else {
+                    logger.error("ERROR: Could not download ${s3FileName} and no valid cached version available.")
+                    logger.error("S3 URL attempted: ${s3Url}")
+                    throw new GradleException("Failed to download " + s3FileName +
+                            " and no valid cached version exists. Build cannot continue.")
+                }
+            }
+        }
+    }
+}
+
+tasks.named('compileJava').configure {
+    dependsOn tasks.named('downloadLibMongosqlTranslate')
+}
+
 tasks.register('runMongoSQLTranslateLibTest', Test) {
     description = 'Runs MongoSQLTranslateLibTest'
     group = 'verification'