Implement terraform based system tests (#227)

* Terraform executor as Docker image

* Implement docker-compose for tf executor

* Enable _dev/deploy for data streams

* Fix

* Gohack: github.com/mtojek/package-spec/code/go

* Install service deployer as static resource

* Cleanup

* Define sample test config for AWS SNS

* WIP

* Fix

* Build terraform environment

* Mount TF_DIR

* Set envs in policies

* Fix: lint

* Fix: build dir (empty value)

* Fix: yml file

* Fix: service name

* Fix: brackets

* Fix

* Fix: reference issue

* Fix: period

* Set time

* Do not print request body to prevent leaks

* Fix: format

* Sample for EC2 metrics

* Fix EC2 system test

* Disable SNS test

* Pass test run ID

* Next fixes

* Fix: formatter

* Try: enable monitoring

* Update dependency on package-spec

* Address PR comments

* Fix failing test

* Fix: go.sum

* Don't set ports

* Update package-spec

* Collect logs for integrations

* Use cloud environment

* Fix: go mod tidy

* Update package-spec dependency

* Support session token

* Terraform service deployer

* nit-picks

* Don't connect to the stack network

* writeTerraformDeployerResources

* Link: env-file

* Link GH issue

* Fix comment

* Next improvements

* nit: separator

* Docs

* Update docs/howto/system_testing.md

Co-authored-by: Shaunak Kashyap <[email protected]>

* Update docs/howto/system_testing.md

Co-authored-by: Shaunak Kashyap <[email protected]>

* Introductory sentences

* Docker compose, terraform

* More generic

* Dont set hostname

* Check if there is a single definition of service deployer

* Fix

* Another fix

Co-authored-by: Shaunak Kashyap <[email protected]>

Author: mtojek

.ci/Jenkinsfile

@@ -8,6 +8,7 @@ pipeline {
     BASE_DIR="src/github.com/elastic/elastic-package"
     JOB_GIT_CREDENTIALS = "f6c7695a-671e-4f4f-a331-acdce44ff9ba"
     PIPELINE_LOG_LEVEL='INFO'
+    AWS_ACCOUNT_SECRET = 'secret/observability-team/ci/elastic-observability-aws-account-auth'
   }
   options {
     timeout(time: 1, unit: 'HOURS')
@@ -40,16 +41,19 @@ pipeline {
       steps {
         cleanup()
         withMageEnv(){
-          dir("${BASE_DIR}"){
-            sh(label: 'Check',script: 'make check')
+          withCloudTestEnv() {
+            dir("${BASE_DIR}"){
+              sh(label: 'Check',script: 'make check')
+            }
           }
         }
       }
       post {
         always {
           dir("${BASE_DIR}") {
             archiveArtifacts(allowEmptyArchive: true, artifacts: 'build/test-results/*.xml')
-            archiveArtifacts(allowEmptyArchive: true, artifacts: 'build/elastic-stack-dump/logs/*.log')
+            archiveArtifacts(allowEmptyArchive: true, artifacts: 'build/elastic-stack-dump/stack/logs/*.log')
+            archiveArtifacts(allowEmptyArchive: true, artifacts: 'build/elastic-stack-dump/check/logs/*.log')
             junit(allowEmptyResults: false,
                 keepLongStdio: true,
                 testResults: "build/test-results/*.xml")
@@ -71,3 +75,22 @@ def cleanup(){
   }
   unstash 'source'
 }
+
+def withCloudTestEnv(Closure body) {
+  def maskedVars = []
+  // AWS
+  def aws = getVaultSecret(secret: "${AWS_ACCOUNT_SECRET}").data
+  if (!aws.containsKey('access_key')) {
+    error("${AWS_ACCOUNT_SECRET} doesn't contain 'access_key'")
+  }
+  if (!aws.containsKey('secret_key')) {
+    error("${AWS_ACCOUNT_SECRET} doesn't contain 'secret_key'")
+  }
+  maskedVars.addAll([
+    [var: "AWS_ACCESS_KEY_ID", password: aws.access_key],
+    [var: "AWS_SECRET_ACCESS_KEY", password: aws.secret_key],
+  ])
+  withEnvMask(vars: maskedVars) {
+    body()
+  }
+}

docs/howto/system_testing.md

@@ -12,19 +12,18 @@ Conceptually, running a system test involves the following steps:
 1. Depending on the Elastic Package whose data stream is being tested, deploy an instance of the package's integration service.
 1. Create a test policy that configures a single data stream for a single package.
 1. Assign the test policy to the enrolled Agent.
-1. Wait a reasonable amount of time for the Agent to collect data from the 
+1. Wait a reasonable amount of time for the Agent to collect data from the
    integration service and index it into the correct Elasticsearch data stream.
 1. Query the first 500 documents based on `@timestamp` for validation.
 1. Validate mappings are defined for the fields contained in the indexed documents.
 1. Validate that the JSON data types contained `_source` are compatible with
-   mappings declared for the field. 
+   mappings declared for the field.
 1. Delete test artifacts and tear down the instance of the package's integration service.
 1. Once all desired data streams have been system tested, tear down the Elastic Stack.
 
 ## Limitations
 
 At the moment system tests have limitations. The salient ones are:
-* They can only test packages whose integration services can be deployed via Docker Compose. Eventually they will be able to test packages that can be deployed via other means, e.g. a Terraform configuration.
 * There isn't a way to do assert that the indexed data matches data from a file (e.g. golden file testing).
 
 ## Defining a system test
@@ -39,21 +38,38 @@ Packages have a specific folder structure (only relevant parts shown).
   manifest.yml
 ```
 
-To define a system test we must define configuration at two levels: the package level and each data stream's level.
+To define a system test we must define configuration on at least one level: a package or a data stream's one.
 
-### Package-level configuration
-
-First, we must define the configuration for deploying a package's integration service. As mentioned in the [_Limitations_](#Limitations) section above, only packages whose integration services can be deployed via Docker Compose are supported at the moment.
+First, we must define the configuration for deploying a package's integration service. We can define it on either the package level:
 
 ```
 <package root>/
   _dev/
     deploy/
-      docker/
-        docker-compose.yml
+      <service deployer>/
+        <service deployer files>
+```
+
+or the data stream's level:
+
 ```
+<package root>/
+  data_stream/
+    <data stream>/
+      _dev/
+        deploy/
+          <service deployer>/
+            <service deployer files>
+```
+
+`<service deployer>` - a name of the supported service deployer: `docker` (Docker Compose service deployer) or `tf` (Terraform service deployer).
+
+### Docker Compose service deployer
 
-The `docker-compose.yml` file defines the integration service(s) for the package. If your package has a logs data stream, the log files from your package's integration service must be written to a volume. For example, the `apache` package has the following definition in it's integration service's `docker-compose.yml` file.
+When using the Docker Compose service deployer, the `<service deployer files>` must include a `docker-compose.yml` file.
+The `docker-compose.yml` file defines the integration service(s) for the package. If your package has a logs data stream,
+the log files from your package's integration service must be written to a volume. For example, the `apache` package has
+the following definition in it's integration service's `docker-compose.yml` file.
 
 ```
 version: '2.3'
@@ -66,7 +82,43 @@ services:
 
 Here, `SERVICE_LOGS_DIR` is a special keyword. It is something that we will need later.
 
-### Data stream-level configuration
+### Terraform service deployer
+
+When using the Terraform service deployer, the `<service deployer files>` must include at least one `*.tf` file.
+The `*.tf` files define the infrastructure using the Terraform syntax. The terraform based service can be handy to boot up
+resources using selected cloud provider and use them for testing (e.g. observe and collect metrics).
+
+Sample `main.tf` definition:
+
+```
+variable "TEST_RUN_ID" {
+  default = "detached"
+}
+
+provider "aws" {}
+
+resource "aws_instance" "i" {
+  ami           = data.aws_ami.latest-amzn.id
+  monitoring = true
+  instance_type = "t1.micro"
+  tags = {
+    Name = "elastic-package-test-${var.TEST_RUN_ID}"
+  }
+}
+
+data "aws_ami" "latest-amzn" {
+  most_recent = true
+  owners = [ "amazon" ] # AWS
+  filter {
+    name   = "name"
+    values = ["amzn2-ami-hvm-*"]
+  }
+}
+```
+
+Notice the use of the `TEST_RUN_ID` variable. It contains a unique ID, which can help differentiate resources created in potential concurrent test runs.
+
+### Test case definition
 
 Next, we must define configuration for each data stream that we want to system test.
 
@@ -97,10 +149,8 @@ The `data_stream.vars` field corresponds to data stream-level variables for the
 
 Notice the use of the `{{SERVICE_LOGS_DIR}}` placeholder. This corresponds to the `${SERVICE_LOGS_DIR}` variable we saw in the `docker-compose.yml` file earlier. In the above example, the net effect is as if the `/usr/local/apache2/logs/access.log*` files located inside the Apache integration service container become available at the same path from Elastic Agent's perspective.
 
-When a data stream's manifest declares multiple streams with different inputs
-you can use the `input` option to select the stream to test. The first stream
-whose input type matches the `input` value will be tested. By default, the first
-stream declared in the manifest will be tested. 
+When a data stream's manifest declares multiple streams with different inputs you can use the `input` option to select the stream to test. The first stream
+whose input type matches the `input` value will be tested. By default, the first stream declared in the manifest will be tested.
 
 #### Placeholders
 
@@ -152,4 +202,4 @@ Finally, when you are done running all system tests, bring down the Elastic Stac
 
 ```
 elastic-package stack down
-```
+```

go.mod

@@ -10,7 +10,7 @@ require (
 	github.com/elastic/go-elasticsearch/v7 v7.9.0
 	github.com/elastic/go-licenser v0.3.1
 	github.com/elastic/go-ucfg v0.8.3
-	github.com/elastic/package-spec/code/go v0.0.0-20210126144901-46090e1310d3
+	github.com/elastic/package-spec/code/go v0.0.0-20210127201409-dd08da649371
 	github.com/go-git/go-billy/v5 v5.0.0
 	github.com/go-git/go-git/v5 v5.1.0
 	github.com/go-openapi/strfmt v0.19.6 // indirect