Test cloud based repository plugins using real services backends #29349
Assignees
Labels
:Distributed Coordination/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
Meta
>test
Issues or PRs that are addressing/adding tests
Comments
tlrx
added
>test
|
Pinging @elastic/es-distributed |
|
Having the ability to test against the real services sounds great (given sufficient setup information passed into the test). One ask: it would be nice to have the qa/third-party tests work "normally" so we know eg the rest test files work, and "swap out" the fixture for the real service when running the test when credentials/etc are passed in. Your idea to use the rest tests from the plugin itself sound good, but I worry the setup of the test in gradle would be left untested until we actually pass in the credentials. What if the fixture based tests were all under qa/third-party? |
|
Thanks @rjernst, that's the feedback I wanted. I updated the description of the issue and moving REST tests and fixtures under qa/third-party sounds like a good idea. |
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this issue
Apr 27, 2018
This commit moves the repository-s3 fixture test added in elastic#29296 in a new `qa/third-party/amazon-s3` project. This new project allows the REST integration tests to be executed using the real S3 service when all the required environment variables are provided. When no env var is provided, then the tests are executed using the fixture added in elastic#29296. The REST tests located in the plugin project now only verify that the plugin is correctly loaded. The REST tests have been adapted to allow a bucket name and a base path to be specified as env vars. This way it is possible to run the tests with different base paths (could be anything, like a CI job name or a branch name) without multiplicating buckets. Related to elastic#29349
tlrx
added a commit
that referenced
this issue
Apr 27, 2018
This commit moves the repository-s3 fixture test added in #29296 in a new `repository-s3/qa/amazon-s3` project. This new project allows the REST integration tests to be executed using the real S3 service when all the required environment variables are provided. When no env var is provided, then the tests are executed using the fixture added in #29296. The REST tests located at the `repository-s3`plugin project now only verify that the plugin is correctly loaded. The REST tests have been adapted to allow a bucket name and a base path to be specified as env vars. This way it is possible to run the tests with different base paths (could be anything, like a CI job name or a branch name) without multiplicating buckets. Related to #29349
tlrx
added a commit
that referenced
this issue
Apr 27, 2018
This commit moves the repository-s3 fixture test added in #29296 in a new `repository-s3/qa/amazon-s3` project. This new project allows the REST integration tests to be executed using the real S3 service when all the required environment variables are provided. When no env var is provided, then the tests are executed using the fixture added in #29296. The REST tests located at the `repository-s3`plugin project now only verify that the plugin is correctly loaded. The REST tests have been adapted to allow a bucket name and a base path to be specified as env vars. This way it is possible to run the tests with different base paths (could be anything, like a CI job name or a branch name) without multiplicating buckets. Related to #29349
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this issue
Apr 30, 2018
Similarly to what has been done in for the repository-s3 plugin, this pull request moves the fixture test into a dedicated repository-azure/qa/microsoft-azure-storage project. It also exposes some environment variables which allows to execute the integration tests against the real Azure Storage service. When the environment variables are not defined, the integration tests are executed using the fixture added in elastic#29347. Related to elastic#29349
tlrx
added a commit
that referenced
this issue
Apr 30, 2018
Similarly to what has been done in for the repository-s3 plugin, this commit moves the fixture test into a dedicated repository-gcs/qa/google-cloud-storage project. It also exposes some environment variables which allows to execute the integration tests against the real Google Cloud Storage service. When the environment variables are not defined, the integration tests are executed using the fixture added in #28788. Related to #29349.
tlrx
added a commit
that referenced
this issue
Apr 30, 2018
Similarly to what has been done in for the repository-s3 plugin, this commit moves the fixture test into a dedicated repository-gcs/qa/google-cloud-storage project. It also exposes some environment variables which allows to execute the integration tests against the real Google Cloud Storage service. When the environment variables are not defined, the integration tests are executed using the fixture added in #28788. Related to #29349.
tlrx
added a commit
that referenced
this issue
May 2, 2018
Similarly to what has been done in for the repository-s3 plugin, this pull request moves the fixture test into a dedicated repository-azure/qa/microsoft-azure-storage project. It also exposes some environment variables which allows to execute the integration tests against the real Azure Storage service. When the environment variables are not defined, the integration tests are executed using the fixture added in #29347. Closes #29349
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Disclaimer: this issue concerns the repository plugins that use cloud-based services like Amazon S3, Azure Storage and Google Cloud Storage in order to store snapshot files.
These plugins used to be tested automatically against the real service they implement but it is not the case anymore. While some developers complained that the integration tests failed too often because of service outage/disconnections, we still think that we must have these automatized tests if we want to catch more bugs and add more features.
We recently added integration tests that use fixtures which emulate service backends (#28788, #29296, #29347). After discussing with @rjernst we could move these tests in QA sub projects under each repository plugin project. We could change the build.gradle files so that the required credentials for each service are pulled out from environment variables. If the env vars are supplied then the REST integration tests are executed against the real service backend. Otherwise the fixtures will be used to execute the tests.
repository-s3/qa/amazon-s3project and allow testing against external service (Move repository-s3 fixture tests to QA third party project #29372)repository-gcs/qa/google-cloud-storageproject and allow testing against external service (Move repository-gcs fixture test to QA project and allow testing against external service #30208)qa/third-party/microsoft-azure-storageand use env vars to retrieve the container and account names (Move repository-azure fixture test to QA project and allow testing using external service #30253)Note: The
fsandhdfsplugins are already correctly tested I think. Theurlcould have integration tests but it can be treated as a separate issue. Same for discovery plugins that could benefit from fixture based integration tests and also real integration tests, but I'd like to address those in a separate issue.Related to #20862
The text was updated successfully, but these errors were encountered: