Support Prometheus via plugin

[alexey-pelykh]

Sending stats and access logs to https://prometheus.io/ would be a great addition. For inspiration, there's https://github.com/gmm42/tinyproxy_exporter/blob/master/tinyproxy_exporter

[abhinavsingh]

Thank you, I'll look into it :)

[magmax]

Hi there!

I'd like to try to fix this, if you don't mind.

which is the best approach? Perhaps to give support on core to store metrics, so plugins can send their own metrics there and be served from the prometheus plugin?

[abhinavsingh]

@magmax Thank you, that would be great and much appreciated.

Indeed giving support directly in the core would be the best way to implement it. I think proxy.py in-built eventing mechanism might be a good fit here. We currently use eventing for CDT integration. Here is a little example of using eventing mechanism in standalone fashion and a little abstract about why we built eventing feature within proxy.py

1. Eventing can be enabled by passing --enable-events flag
2. Just as with CDT integration, each plugin/core/others can publish metrics into a single queue
3. A subscriber can receive these published metrics and send to upstream prometheus or other endpoints

[magmax]

Nice, @abhinavsingh !

I still have some problems with this XD

First of all, Prometheus works on "pull" method instead of "push". That means that each resource won't send data to the server; it is the Prometheus server which will ask for data whenever it wants. That means the prometheus plugin will require to add an endpoint to the webserver.

So... This is the plan:

1. I will start by enabling events and allow core and plugins to send metrics.
2. Then, I can create the plugin to consume those metrics and render them in screen.
3. Finally, I can show them in an endpoint on the web server.

I've thought in avoiding to depend on prometheus-client library on core, what means I will require to duplicate part of its functionality: creating some types of metrics to allow communication between emitter, core and prometheus-plugin.

[abhinavsingh]

@magmax Whoops, correct. Subscriber can simply accumulate and store metrics, while an endpoint indeed will be necessary from where prometheus can pull the data. While onto it, you can define an abstract class (to write/read metrics and associated basic types) and write a single implementation of your choice :)

• in-memory
• on disk (files/sqlite/time-series db)
• in cache (redis, memcache, ..)

I've thought in avoiding to depend on prometheus-client library on core

Indeed. We can start with adding a couple of metric types. Flow wise, event emitters will emit these types, subscribers will accumulate these types and store.

You can tie all this functionality under 2 flags:

1. --enable-metrics (which internally can also --enable-events) -- This flag essentially starts metric storage collection
2. --prometheus-path -- This flag simply adds an endpoint at provided /path/ which can be polled by prometheus

And 1), 2) can both be provided independently. E.g even if --prometheus-path is provided without --enable-metrics, endpoint will keep returning empty responses. The 2nd part (--prometheus-path) itself can be its own plugin. And it's perfectly ok to use prometheus-client within this plugin file. This file will read out of proxy.py core metrics and convert them into prometheus objects.

PS: While I do understand the concept, I have not used prometheus as much as I would have liked, so I'll have to bank upon yours and community expertise for final integration here :)

[magmax]

I almost have the implementation. Just fighting against processes and threads. Anyway, I have the solution in the top of my head, and just need time to implement it. Probably It will be out today or tomorrow.

[abhinavsingh]

Few things that crossed my mind and may help:

1. To keep it simply, you can put subscriber within the main thread instead sleep_loop as starter.
2. Interesting would be how is every process/thread be able to publish events. I think having a flags based facility might be a viable option, because flags are available throughout the plugins/core code base.
3. In the main thread, we can populate necessary flags attribute as reusable publish method.

[abhinavsingh]

@magmax Thank you for the PR and apologies for being away for so long. I need prometheus integration in one of the projects based on proxy.py. I'll take forward your PR and try to ship it in the next release. @alexey-pelykh Thanks for the reference implementation :D

[abhinavsingh]

@alexey-pelykh @magmax PR #1447 ships prometheus support. Please give it a try when you can. Thank you for reference links and PRs.