Investigate options to support async (Twisted)

[beniwohli]

We currently don't support any asynchronous Python frameworks like Twisted, Tornado & Co. Supporting these frameworks is inherently difficult due to the fact that we made a lot of design decisions on the assumption that a thread only handles a single request at a time. This is not true for async frameworks.

There are some ways we could achieve at least some support for Twisted by storing data of the current transaction on the request handler instead of a thread local. Similar workarounds may or may not exist for other async frameworks.

In the future, we may be able to use execution contexts as proposed by PEP 550, but this will take some time until it is available.

[beniwohli]

For Tornado, we could potentially piggyback on StackContext using something like https://gist.github.com/virtuald/50bf7cacdc8cfb05e323f350539f0efa

I haven't found something similar for Twisted. There is some discussion around context tracking here, but the general advice there seems to be against trying to use twisted.python.context.ContextTracker to track asynchronous context.

[beniwohli]

There's some more development in this space:

• PEP 567 - Context Variables has been accepted/merged and will be available in Python 3.7
• There is a backport of PEP 567 to Python 3.6 (albeit without an asyncio event loop)
• As pointed out by @watson, there is a 3rd party library called AioContext that might be helpful.

All of these are built around asyncio though, so I'm not sure of how much benefit either one would be in conjunction with Twisted and Tornado (both of which use their own event loop implementation)

[beniwohli]

See #252 for a very WIPy implementation of asyncio support

[chris-erickson]

The asgi apps are starting to gain traction, and are a compelling alternative to Go, Node, etc for better concurrency in heavy IO blocking apps. APM support is a sticking point for me in building apps this way (Starlette in particular) and I’d love to see support in this area, or even recommendations on how to get partial support of some kind (ie. request timing) working. Thanks!

[bhodorog]

Another asynchronous framework quite used by python projects is gevent. Needless to say currently that's not supported by the apm-agent-python. The first significant effect is that transactions are not being supported.

While I appreciate the initial design decision which were mentioned by the OP in the first post would you guys consider supporting something like gevent ? I'm thinking you might be able to use werkzeug.local to achieve the same effect as contextvar does for native threads and asyncio (with python >=3.7).

[basepi]

We have support for tornado, starlette/fastapi and various async libraries as well. If there's continued interest in gevent, twisted, or others, we request that separate issues be made.