Skip to content

Commit 5358eb6

Browse files
author
Azfaar Qureshi
committed
adding README
adding sample app adding examples readme fixing lint errors linting examples updating readme tls_config example excluding examples adding examples to exclude in all linters adding isort.cfg skip changing isort to path ignoring yml only adding it to excluded directories in pylintrc only adding exclude to directory removing readme.rst and adding explicit file names to ignore adding the rest of the files adding readme.rst back adding to ignore glob instead reverting back to ignore list converting README.md to README.rst
1 parent 6514f37 commit 5358eb6

File tree

10 files changed

+583
-17
lines changed

10 files changed

+583
-17
lines changed

‎.flake8

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ ignore =
44
F401 # unused import, defer to pylint
55
W503 # allow line breaks before binary ops
66
W504 # allow line breaks after binary ops
7-
E203 # allow whitespace before ':' (https://github.com/psf/black#slices)
7+
E203 # allow whitespace before ':' (https://github.com/psf/black#slices)
88
exclude =
99
.bzr
1010
.git

‎.pylintrc

+1-1
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ extension-pkg-whitelist=
77

88
# Add list of files or directories to be excluded. They should be base names, not
99
# paths.
10-
ignore=CVS,gen
10+
ignore=CVS,gen,Dockerfile,docker-compose.yml,README.md,requirements.txt,cortex-config.yml
1111

1212
# Add files or directories matching the regex patterns to be excluded. The
1313
# regex matches against base names, not paths.

‎CHANGELOG.md

+2
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2525
([#237](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/237))
2626
- Add Prometheus Remote Write Exporter integration tests in opentelemetry-docker-tests
2727
([#216](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/216))
28+
- Add README and example app for Prometheus Remote Write Exporter
29+
([#227](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/227]))
2830

2931
### Changed
3032
- `opentelemetry-instrumentation-asgi`, `opentelemetry-instrumentation-wsgi` Return `None` for `CarrierGetter` if key not found

‎exporter/opentelemetry-exporter-prometheus-remote-write/README.rst

+236-15
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,248 @@
1-
OpenTelemetry Prometheus Remote Write Exporter
2-
==============================================
1+
OpenTelemetry Python SDK Prometheus Remote Write Exporter
2+
=========================================================
33

4-
This library allows exporting metric data to `Prometheus Remote Write Integrated Backends
5-
<https://prometheus.io/docs/operating/integrations/>`_. Latest `types.proto
6-
<https://github.com/prometheus/prometheus/blob/master/prompb/types.proto>` and `remote.proto
7-
<https://github.com/prometheus/prometheus/blob/master/prompb/remote.proto>` Protocol Buffers
8-
used to create WriteRequest objects were taken from Prometheus repository. Development is
9-
currently in progress.
4+
This package contains an exporter to send `OTLP`_ metrics from the
5+
Python SDK directly to a Prometheus Remote Write integrated backend
6+
(such as Cortex or Thanos) without having to run an instance of the
7+
Prometheus server. The image below shows the two Prometheus exporters in
8+
the OpenTelemetry Python SDK.
9+
10+
Pipeline 1 illustrates the setup required for a Prometheus "pull"
11+
exporter.
12+
13+
Pipeline 2 illustrates the setup required for the Prometheus Remote
14+
Write exporter.
15+
16+
|Prometheus SDK pipelines|
17+
18+
The Prometheus Remote Write Exporter is a "push" based exporter and only
19+
works with the OpenTelemetry `push controller`_. The controller
20+
periodically collects data and passes it to the exporter. This exporter
21+
then converts the data into `timeseries`_ and sends it to the Remote
22+
Write integrated backend through HTTP POST requests. The metrics
23+
collection datapath is shown below:
24+
25+
|controller_datapath_final|
26+
27+
See the ``example`` folder for a demo usage of this exporter
28+
29+
Table of Contents
30+
=================
31+
32+
- `Summary`_
33+
- `Table of Contents`_
34+
35+
- `Installation`_
36+
- `Quickstart`_
37+
- `Configuring the Exporter`_
38+
- `Securing the Exporter`_
39+
40+
- `Authentication`_
41+
- `TLS`_
42+
43+
- `Supported Aggregators`_
44+
- `Error Handling`_
45+
- `Retry Logic`_
46+
- `Contributing`_
47+
48+
- `Design Doc`_
1049

1150
Installation
1251
------------
52+
Prerequisite
53+
~~~~~~~~~~~~
54+
1. Install the snappy c-library
55+
**DEB**: `sudo apt-get install libsnappy-dev`
56+
**RPM**: `sudo yum install libsnappy-devel`
57+
**OSX/Brew**: `brew install snappy`
58+
2. Install python-snappy
59+
`pip install python-snappy`
60+
61+
Exporter
62+
~~~~~~~~
63+
64+
- To install from the latest PyPi release, run
65+
``pip install opentelemetry-exporter-prometheus-remote-write``
66+
- To install from the local repository, run
67+
``pip install -e exporter/opentelemetry-exporter-prometheus-remote-write/``
68+
in the project root
69+
70+
Quickstart
71+
----------
72+
73+
.. code:: python
74+
75+
from opentelemetry import metrics
76+
from opentelemetry.sdk.metrics import MeterProvider
77+
from opentelemetry.exporter.prometheus_remote_write import (
78+
PrometheusRemoteWriteMetricsExporter
79+
)
80+
81+
# Sets the global MeterProvider instance
82+
metrics.set_meter_provider(MeterProvider())
83+
84+
# The Meter is responsible for creating and recording metrics. Each meter has a unique name, which we set as the module's name here.
85+
meter = metrics.get_meter(__name__)
86+
87+
exporter = PrometheusRemoteWriteMetricsExporter(endpoint="endpoint_here") # add other params as needed
88+
89+
metrics.get_meter_provider().start_pipeline(meter, exporter, 5)
90+
91+
Configuring the Exporter
92+
------------------------
93+
94+
The exporter can be configured through parameters passed to the
95+
constructor. Here are all the options:
96+
97+
- ``endpoint``: url where data will be sent **(Required)**
98+
- ``basic_auth``: username and password for authentication
99+
**(Optional)**
100+
- ``headers``: additional headers for remote write request as
101+
determined by the remote write backend's API **(Optional)**
102+
- ``timeout``: timeout for requests to the remote write endpoint in
103+
seconds **(Optional)**
104+
- ``proxies``: dict mapping request proxy protocols to proxy urls
105+
**(Optional)**
106+
- ``tls_config``: configuration for remote write TLS settings
107+
**(Optional)**
108+
109+
Example with all the configuration options:
110+
111+
.. code:: python
112+
113+
exporter = PrometheusRemoteWriteMetricsExporter(
114+
endpoint="http://localhost:9009/api/prom/push",
115+
timeout=30,
116+
basic_auth={
117+
"username": "user",
118+
"password": "pass123",
119+
},
120+
headers={
121+
"X-Scope-Org-ID": "5",
122+
"Authorization": "Bearer mytoken123",
123+
},
124+
proxies={
125+
"http": "http://10.10.1.10:3000",
126+
"https": "http://10.10.1.10:1080",
127+
},
128+
tls_config={
129+
"cert_file": "path/to/file",
130+
"key_file": "path/to/file",
131+
"ca_file": "path_to_file",
132+
"insecure_skip_verify": true, # for developing purposes
133+
}
134+
)
135+
136+
Securing the Exporter
137+
---------------------
138+
139+
Authentication
140+
~~~~~~~~~~~~~~
141+
142+
The exporter provides two forms of authentication which are shown below.
143+
Users can add their own custom authentication by setting the appropriate
144+
values in the ``headers`` dictionary
145+
146+
1. Basic Authentication Basic authentication sets a HTTP Authorization
147+
header containing a base64 encoded username/password pair. See `RFC
148+
7617`_ for more information. This
149+
150+
.. code:: python
151+
152+
exporter = PrometheusRemoteWriteMetricsExporter(
153+
basic_auth={"username": "base64user", "password": "base64pass"}
154+
)
155+
156+
2. Bearer Token Authentication This custom configuration can be achieved
157+
by passing in a custom ``header`` to the constructor. See `RFC 6750`_
158+
for more information.
159+
160+
.. code:: python
161+
162+
header = {
163+
"Authorization": "Bearer mytoken123"
164+
}
165+
166+
TLS
167+
~~~
168+
169+
Users can add TLS to the exporter's HTTP Client by providing certificate
170+
and key files in the ``tls_config`` parameter.
171+
172+
Supported Aggregators
173+
---------------------
174+
175+
- Sum
176+
- MinMaxSumCount
177+
- Histogram
178+
- LastValue
179+
- ValueObserver
180+
181+
Error Handling
182+
--------------
183+
184+
In general, errors are raised by the calling function. The exception is
185+
for failed requests where any error status code is logged as a warning
186+
instead.
187+
188+
This is because the exporter does not implement any retry logic as it
189+
sends cumulative metrics data. This means that data will be preserved
190+
even if some exports fail.
191+
192+
For example, consider a situation where a user increments a Counter
193+
instrument 5 times and an export happens between each increment. If the
194+
exports happen like so:
13195

14196
::
15197

16-
pip install opentelemetry-exporter-prometheus-remote-write
198+
SUCCESS FAIL FAIL SUCCESS SUCCESS
199+
1 2 3 4 5
17200

201+
Then the recieved data will be:
18202

19-
.. _Prometheus: https://prometheus.io/
20-
.. _OpenTelemetry: https://github.com/open-telemetry/opentelemetry-python/
203+
::
21204

205+
1 4 5
22206

23-
References
24-
----------
207+
The end result is the same since the aggregations are cumulative
208+
Contributing
209+
------------
210+
211+
This exporter's datapath is as follows:
212+
213+
|Exporter datapath| *Entites with ``*`` after their name are not actual
214+
classes but rather logical groupings of functions within the exporter.*
215+
216+
If you would like to learn more about the exporter's structure and
217+
design decisions please view the design document below
218+
219+
Design Doc
220+
~~~~~~~~~~
221+
222+
`Design Document`_
223+
224+
This document is stored elsewhere as it contains large images which will
225+
significantly increase the size of this repo.
25226

26-
* `Prometheus <https://prometheus.io/>`_
27-
* `OpenTelemetry Project <https://opentelemetry.io/>`_
227+
.. _Design Document: https://github.com/open-o11y/docs/tree/master/python-prometheus-remote-write
228+
.. |Exporter datapath| image:: https://user-images.githubusercontent.com/20804975/100285717-604c7280-2f3f-11eb-9b73-bdf70afce9dd.png
229+
.. _OTLP: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/protocol/otlp.md
230+
.. _push controller: https://github.com/open-telemetry/opentelemetry-python/blob/master/opentelemetry-sdk/src/opentelemetry/sdk/metrics/export/controller.py
231+
.. _`timeseries`: https://prometheus.io/docs/concepts/data_model/
232+
.. _Summary: #opentelemetry-python-sdk-prometheus-remote-write-exporter
233+
.. _Table of Contents: #table-of-contents
234+
.. _Installation: #installation
235+
.. _Quickstart: #quickstart
236+
.. _Configuring the Exporter: #configuring-the-exporter
237+
.. _Securing the Exporter: #securing-the-exporter
238+
.. _Authentication: #authentication
239+
.. _TLS: #tls
240+
.. _Supported Aggregators: #supported-aggregators
241+
.. _Error Handling: #error-handling
242+
.. _Retry Logic: #retry-logic
243+
.. _Contributing: #contributing
244+
.. _Design Doc: #design-doc
245+
.. |Prometheus SDK pipelines| image:: https://user-images.githubusercontent.com/20804975/100285430-e320fd80-2f3e-11eb-8217-a562c559153c.png
246+
.. |controller_datapath_final| image:: https://user-images.githubusercontent.com/20804975/100486582-79d1f380-30d2-11eb-8d17-d3e58e5c34e9.png
247+
.. _RFC 7617: https://tools.ietf.org/html/rfc7617
248+
.. _RFC 6750: https://tools.ietf.org/html/rfc6750

‎exporter/opentelemetry-exporter-prometheus-remote-write/examples/Dockerfile

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
FROM python:3.7
2+
WORKDIR /code
3+
4+
COPY . .
5+
RUN apt-get update -y && apt-get install libsnappy-dev -y
6+
RUN pip install -e .
7+
RUN pip install -r ./examples/requirements.txt
8+
CMD ["python", "./examples/sampleapp.py"]

‎exporter/opentelemetry-exporter-prometheus-remote-write/examples/README.md

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Prometheus Remote Write Exporter Example
2+
This example uses [Docker Compose](https://docs.docker.com/compose/) to set up:
3+
4+
1. A Python program that creates 5 instruments with 5 unique
5+
aggregators and a randomized load generator
6+
2. An instance of [Cortex](https://cortexmetrics.io/) to recieve the metrics
7+
data
8+
3. An instance of [Grafana](https://grafana.com/) to visualizse the exported
9+
data
10+
11+
## Requirements
12+
* Have Docker Compose [installed](https://docs.docker.com/compose/install/)
13+
14+
*Users do not need to install Python as the app will be run in the Docker Container*
15+
16+
## Instructions
17+
1. Run `docker-compose up -d` in the the `examples/` directory
18+
19+
The `-d` flag causes all services to run in detached mode and frees up your
20+
terminal session. This also causes no logs to show up. Users can attach themselves to the service's logs manually using `docker logs ${CONTAINER_ID} --follow`
21+
22+
2. Log into the Grafana instance at [http://localhost:3000](http://localhost:3000)
23+
* login credentials are `username: admin` and `password: admin`
24+
* There may be an additional screen on setting a new password. This can be skipped and is optional
25+
26+
3. Navigate to the `Data Sources` page
27+
* Look for a gear icon on the left sidebar and select `Data Sources`
28+
29+
4. Add a new Prometheus Data Source
30+
* Use `http://cortex:9009/api/prom` as the URL
31+
* (OPTIONAl) set the scrape interval to `2s` to make updates appear quickly
32+
* click `Save & Test`
33+
34+
5. Go to `Metrics Explore` to query metrics
35+
* Look for a compass icon on the left sidebar
36+
* click `Metrics` for a dropdown list of all the available metrics
37+
* (OPTIONAL) Adjust time range by clicking the `Last 6 hours` button on the upper right side of the graph
38+
* (OPTIONAL) Set up auto-refresh by selecting an option under the dropdown next to the refresh button on the upper right side of the graph
39+
* Click the refresh button and data should show up on hte graph
40+
41+
6. Shutdown the services when finished
42+
* Run `docker-compose down` in the examples directory

0 commit comments

Comments
 (0)