Prometheus Remote Write Exporter (5/6) (#216)

Azfaar Qureshi · web-flow · commit 6514f3717783 · 2020-12-09T10:46:31.000-08:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -12,18 +12,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `opentelemetry-sdk-extension-aws` Add method to return fields injected by propagator
   ([#226](https://github.com/open-telemetry/opentelemetry-python/pull/226))
 - `opentelemetry-exporter-prometheus-remote-write` Prometheus Remote Write Exporter Setup
-  ((#180)[https://github.com/open-telemetry/opentelemetry-python-contrib/pull/180])
-- `opentelemetry-exporter-prometheus-remote-write` Add Exporter constructor validation methods
-  ((#206)[https://github.com/open-telemetry/opentelemetry-python-contrib/pull/206])
-- `opentelemetry-exporter-prometheus-remote-write` Add conversion to TimeSeries methods
-  ((#207)[https://github.com/open-telemetry/opentelemetry-python-contrib/pull/207])
-- `opentelemetry-exporter-prometheus-remote-write` Add request methods
-  ((#212)[https://github.com/open-telemetry/opentelemetry-python-contrib/pull/212])
+  ([#180](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/180))
+- `opentelemetry-exporter-prometheus-remote-write` Add Exporter constructor validation methods in Prometheus Remote Write Exporter
+  ([#206](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/206))
+- `opentelemetry-exporter-prometheus-remote-write` Add conversion to TimeSeries methods in Prometheus Remote Write Exporter
+  ([#207](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/207))
+- `opentelemetry-exporter-prometheus-remote-write` Add request methods to Prometheus Remote Write Exporter
+  ([#212](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/212))
 - `opentelemetry-instrumentation-fastapi` Added support for excluding some routes with env var `OTEL_PYTHON_FASTAPI_EXCLUDED_URLS`
   ([#237](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/237))
 - `opentelemetry-instrumentation-starlette` Added support for excluding some routes with env var `OTEL_PYTHON_STARLETTE_EXCLUDED_URLS`
   ([#237](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/237))
-
+- Add Prometheus Remote Write Exporter integration tests in opentelemetry-docker-tests
+  ([#216](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/216))
 
 ### Changed
 - `opentelemetry-instrumentation-asgi`, `opentelemetry-instrumentation-wsgi` Return `None` for `CarrierGetter` if key not found
@@ -92,7 +93,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   ([#1239](https://github.com/open-telemetry/opentelemetry-python/pull/1239))
 - `opentelemetry-instrumentation-django` Bugfix use request.path replace request.get_full_path(). It will get correct span name
   ([#1309](https://github.com/open-telemetry/opentelemetry-python/pull/1309#))
-- `opentelemetry-instrumentation-django` Record span status and http.status_code attribute on exception 
+- `opentelemetry-instrumentation-django` Record span status and http.status_code attribute on exception
   ([#1257](https://github.com/open-telemetry/opentelemetry-python/pull/1257))
 - `opentelemetry-instrumentation-grpc` Rewrite gRPC server interceptor
   ([#1171](https://github.com/open-telemetry/opentelemetry-python/pull/1171))
@@ -120,7 +121,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `opentelemetry-instrumentation-pymysql` Bumped version from 0.9.3 to 0.10.1
   ([#1228](https://github.com/open-telemetry/opentelemetry-python/pull/1228))
 - `opentelemetry-instrumentation-django` Changed span name extraction from request to comply semantic convention
-  ([#992](https://github.com/open-telemetry/opentelemetry-python/pull/992)) 
+  ([#992](https://github.com/open-telemetry/opentelemetry-python/pull/992))
 
 ## [0.13b0](https://github.com/open-telemetry/opentelemetry-python/releases/tag/v0.13b0) - 2020-09-17
 
@@ -264,7 +265,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [0.8b0](https://github.com/open-telemetry/opentelemetry-python/releases/tag/v0.8.0) - 2020-05-27
 
 ### Added
-- `opentelemetry-ext-datadog` Add exporter to Datadog 
+- `opentelemetry-ext-datadog` Add exporter to Datadog
   ([#572](https://github.com/open-telemetry/opentelemetry-python/pull/572))
 - `opentelemetry-ext-sqlite3` Initial release
 - `opentelemetry-ext-psycopg2` Implement instrumentor interface, enabling auto-instrumentation
diff --git a/tests/opentelemetry-docker-tests/tests/docker-compose.yml b/tests/opentelemetry-docker-tests/tests/docker-compose.yml
@@ -39,3 +39,11 @@ services:
         - "16686:16686"
         - "14268:14268"
         - "9411:9411"
+  cortex:
+    image: quay.io/cortexproject/cortex:v1.5.0
+    command:
+      - -config.file=./config/cortex-config.yml
+    volumes:
+      - ./prometheus-remote-write-cortex/cortex-config.yml:/config/cortex-config.yml:ro
+    ports:
+      - 9009:9009
diff --git a/tests/opentelemetry-docker-tests/tests/prometheus-remote-write-cortex/cortex-config.yml b/tests/opentelemetry-docker-tests/tests/prometheus-remote-write-cortex/cortex-config.yml
@@ -0,0 +1,100 @@
+# This Cortex Config is copied from the Cortex Project documentation
+# Source: https://github.com/cortexproject/cortex/blob/master/docs/configuration/single-process-config.yaml
+
+# Configuration for running Cortex in single-process mode.
+# This configuration should not be used in production.
+# It is only for getting started and development.
+
+# Disable the requirement that every request to Cortex has a
+# X-Scope-OrgID header. `fake` will be substituted in instead.
+auth_enabled: false
+
+server:
+  http_listen_port: 9009
+
+  # Configure the server to allow messages up to 100MB.
+  grpc_server_max_recv_msg_size: 104857600
+  grpc_server_max_send_msg_size: 104857600
+  grpc_server_max_concurrent_streams: 1000
+
+distributor:
+  shard_by_all_labels: true
+  pool:
+    health_check_ingesters: true
+
+ingester_client:
+  grpc_client_config:
+    # Configure the client to allow messages up to 100MB.
+    max_recv_msg_size: 104857600
+    max_send_msg_size: 104857600
+    use_gzip_compression: true
+
+ingester:
+  # We want our ingesters to flush chunks at the same time to optimise
+  # deduplication opportunities.
+  spread_flushes: true
+  chunk_age_jitter: 0
+
+  walconfig:
+    wal_enabled: true
+    recover_from_wal: true
+    wal_dir: /tmp/cortex/wal
+
+  lifecycler:
+    # The address to advertise for this ingester.  Will be autodiscovered by
+    # looking up address on eth0 or en0; can be specified if this fails.
+    # address: 127.0.0.1
+
+    # We want to start immediately and flush on shutdown.
+    join_after: 0
+    min_ready_duration: 0s
+    final_sleep: 0s
+    num_tokens: 512
+    tokens_file_path: /tmp/cortex/wal/tokens
+
+    # Use an in memory ring store, so we don't need to launch a Consul.
+    ring:
+      kvstore:
+        store: inmemory
+      replication_factor: 1
+
+# Use local storage - BoltDB for the index, and the filesystem
+# for the chunks.
+schema:
+  configs:
+    - from: 2019-07-29
+      store: boltdb
+      object_store: filesystem
+      schema: v10
+      index:
+        prefix: index_
+        period: 1w
+
+storage:
+  boltdb:
+    directory: /tmp/cortex/index
+
+  filesystem:
+    directory: /tmp/cortex/chunks
+
+  delete_store:
+    store: boltdb
+
+purger:
+  object_store_type: filesystem
+
+frontend_worker:
+  # Configure the frontend worker in the querier to match worker count
+  # to max_concurrent on the queriers.
+  match_max_concurrent: true
+
+# Configure the ruler to scan the /tmp/cortex/rules directory for prometheus
+# rules: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules
+ruler:
+  enable_api: true
+  enable_sharding: false
+  storage:
+    type: local
+    local:
+      directory: /tmp/cortex/rules
+
diff --git a/tests/opentelemetry-docker-tests/tests/prometheus-remote-write-cortex/prometheus_remote_write_cortex_test.py b/tests/opentelemetry-docker-tests/tests/prometheus-remote-write-cortex/prometheus_remote_write_cortex_test.py
@@ -0,0 +1,116 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from opentelemetry import metrics
+from opentelemetry.exporter.prometheus_remote_write import (
+    PrometheusRemoteWriteMetricsExporter,
+)
+from opentelemetry.test.test_base import TestBase
+
+
+def observer_callback(observer):
+    array = [1.0, 15.0, 25.0, 26.0]
+    for (index, usage) in enumerate(array):
+        labels = {"test_label": str(index)}
+        observer.observe(usage, labels)
+
+
+class TestPrometheusRemoteWriteExporterCortex(TestBase):
+    def setUp(self):
+        super().setUp
+        self.exporter = PrometheusRemoteWriteMetricsExporter(
+            endpoint="http://localhost:9009/api/prom/push",
+            headers={"X-Scope-Org-ID": "5"},
+        )
+        self.labels = {"environment": "testing"}
+        self.meter = self.meter_provider.get_meter(__name__)
+        metrics.get_meter_provider().start_pipeline(
+            self.meter, self.exporter, 1,
+        )
+
+    def test_export_counter(self):
+        try:
+            requests_counter = self.meter.create_counter(
+                name="counter",
+                description="test_export_counter",
+                unit="1",
+                value_type=int,
+            )
+            requests_counter.add(25, self.labels)
+        except Exception as e:
+            self.fail(
+                "Export counter failed with unexpected error {}".format(e)
+            )
+
+    def test_export_valuerecorder(self):
+        try:
+            requests_size = self.meter.create_valuerecorder(
+                name="valuerecorder",
+                description="test_export_valuerecorder",
+                unit="1",
+                value_type=int,
+            )
+            requests_size.record(25, self.labels)
+        except Exception as e:
+            self.fail(
+                "Export valuerecorder failed with unexpected error {}".format(
+                    e
+                )
+            )
+
+    def test_export_updowncounter(self):
+        try:
+            requests_size = self.meter.create_updowncounter(
+                name="updowncounter",
+                description="test_export_updowncounter",
+                unit="1",
+                value_type=int,
+            )
+            requests_size.add(-25, self.labels)
+        except Exception as e:
+            self.fail(
+                "Export updowncounter failed with unexpected error {}".format(
+                    e
+                )
+            )
+
+    def test_export_sumobserver(self):
+        try:
+            self.meter.register_sumobserver(
+                callback=observer_callback,
+                name="sumobserver",
+                description="test_export_sumobserver",
+                unit="1",
+                value_type=float,
+            )
+        except Exception as e:
+            self.fail(
+                "Export sumobserver failed with unexpected error {}".format(e)
+            )
+
+    def test_export_updownsumobserver(self):
+        try:
+            self.meter.register_updownsumobserver(
+                callback=observer_callback,
+                name="updownsumobserver",
+                description="test_export_updownsumobserver",
+                unit="1",
+                value_type=float,
+            )
+        except Exception as e:
+            self.fail(
+                "Export updownsumobserver failed with unexpected error {}".format(
+                    e
+                )
+            )
diff --git a/tox.ini b/tox.ini
@@ -198,7 +198,7 @@ commands_pre =
   test: pip install {toxinidir}/opentelemetry-python-core/opentelemetry-api {toxinidir}/opentelemetry-python-core/opentelemetry-sdk {toxinidir}/opentelemetry-python-core/tests/util
 
   test: pip install {toxinidir}/opentelemetry-python-core/opentelemetry-instrumentation
-  
+
   celery: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-celery[test]
 
   grpc: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-grpc[test]
@@ -341,7 +341,8 @@ deps =
   sqlalchemy ~= 1.3.16
   redis ~= 3.3.11
   celery ~= 4.0, != 4.4.4
-
+  protobuf>=3.13.0
+  requests==2.25.0
 changedir =
   tests/opentelemetry-docker-tests/tests
 
@@ -361,7 +362,10 @@ commands_pre =
               -e {toxinidir}/instrumentation/opentelemetry-instrumentation-aiopg \
               -e {toxinidir}/instrumentation/opentelemetry-instrumentation-redis \
               -e {toxinidir}/instrumentation/opentelemetry-instrumentation-system-metrics \
-              -e {toxinidir}/opentelemetry-python-core/exporter/opentelemetry-exporter-opencensus
+              -e {toxinidir}/opentelemetry-python-core/exporter/opentelemetry-exporter-opencensus \
+              -e {toxinidir}/exporter/opentelemetry-exporter-prometheus-remote-write
+  sudo apt-get install libsnappy-dev
+  pip install python-snappy
   docker-compose up -d
   python check_availability.py
 commands =
