Add a smoke test for observability SRE container (#17298)

* Add a smoke test for observability SRE container

Add a CI cell to ensure the observability contater is building successfully. In
order to show success run a quick smoke test to point out any glaring issues.

This adds some general, low risk plugins for doing quick testing. This will help
developers in debugging as we work on this image.

* Show what is happening when rake fails

* Debug deeper in the stack

Show the stdout/stderr when shelling out fails.

* Debug layers of build tooling

Open3 is not capturing stdout for some reason. Capture it and print to see what is wrong in CI.

* Actually run ls command in docker container 🤦

* Update safe_system based on code review suggestion

* Dynamically generate version for container invocation

Co-authored-by: Ry Biesemeyer <[email protected]>

* Refactor smoke test setup to script

Avoid interpolation backflips with buildkite by extracting to a script.

* Split out message surfacing improvment to separate PR.

Moved to: #17310

* Extract version qualifier into standalone script

* Wait for version-qualifier.sh script to land upstream

Use  #17311 once it lands and gets
backported to 8.x. For now just hard code version.

---------

Co-authored-by: Ry Biesemeyer <[email protected]>

Author: donoghuc

.buildkite/pull_request_pipeline.yml

@@ -123,6 +123,26 @@ steps:
       manual:
         allowed: true
 
+  - label: "Observability SRE container smoke test"
+    key: "observability-sre-container-smoke-test"
+    agents:
+      provider: gcp
+      imageProject: elastic-images-prod
+      image: family/platform-ingest-logstash-ubuntu-2204
+      machineType: "n2-standard-4"
+      diskSizeGb: 64
+    retry:
+      automatic:
+        - limit: 3
+    command: |
+      set -euo pipefail
+      source .buildkite/scripts/common/vm-agent.sh
+      # TODO: Use https://github.com/elastic/logstash/pull/17311 to compute QUALIFIED_VERSION once merged
+      QUALIFIED_VERSION="8.19.0-SNAPSHOT"
+      ./gradlew --stacktrace artifactDockerObservabilitySRE
+      docker run docker.elastic.co/logstash/logstash-observability-sre:$${QUALIFIED_VERSION} \
+        logstash -e 'input { generator { count => 3 } } output { stdout { codec => rubydebug } }'
+
   - label: ":lab_coat: Integration Tests - FIPS mode / part 1"
     key: "integration-tests-fips-part-1"
     agents:

x-pack/distributions/internal/observabilitySRE/plugin-allow-list.txt

@@ -1,6 +1,7 @@
 logstash-codec-json
 logstash-codec-multiline
 logstash-codec-plain
+logstash-codec-rubydebug
 logstash-filter-age
 logstash-filter-date
 logstash-filter-drop
@@ -9,7 +10,9 @@ logstash-filter-grok
 logstash-filter-json
 logstash-filter-mutate
 logstash-input-beats
+logstash-input-generator
 logstash-input-pipeline
 logstash-output-elasticsearch
 logstash-output-pipeline
+logstash-output-stdout
 logstash-patterns-core