Implemented the v2 API wire protocol (#281)

This implements the v2 API. The largest change is that the queue moved
from `TransactionStore` to `Transport`, and that instead of queuing
Python objects, it queues (by default compressed) lines of JSON.

Another large conceptual change is that events (errors, transactions and
spans) are immediately queued when they are finished.

Architecturally, `elasticapm.base.Client` does only have one single
method to interact with the transport layer, `queue()`, which in turn
calls the transports `queue` method. `send`, `send_remote`, `encode`
etc. are all removed.

Author: beniwohli

CHANGELOG.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## v4.0.0
+
+**BREAKING** Version 4 of the agent implements a new wire protocol for communicating with
+the APM Server. This format is only supported in APM Server 6.5+.
+
+Further breaking changes:
+
+ * the undocumented `AsyncioHTTPTransport` has been removed.
+ * the `flush_interval` and `max_queue_size` settings have been removed.
+ * new settings introduced: `api_request_time` and `api_request_size`.
+ * Some settings now require a unit for duration or size. See documentation on
+   configuration for more information.
+
 ## v3.0.1
 
 [Check the diff](https://github.com/elastic/apm-agent-python/compare/v3.0.0...v3.0.1)

docs/configuration.asciidoc

@@ -230,10 +230,11 @@ Transactions that match any of the of the configured patterns will be ignored an
 [options="header"]
 |============
 | Environment                  | Django/Flask         | Default
-| `ELASTIC_APM_SERVER_TIMEOUT` | `SERVER_TIMEOUT`     | `5`
+| `ELASTIC_APM_SERVER_TIMEOUT` | `SERVER_TIMEOUT`     | `"5s"`
 |============
 
-A timeout in seconds.
+A timeout for requests to the APM Server.
+The setting has to be provided in *<<config-format-duration, duration format>>*.
 If a request to the APM server takes longer than the configured timeout,
 the request is cancelled and the event (exception or transaction) is discarded.
 Set to `None` to disable timeouts.
@@ -369,21 +370,6 @@ It contains the name of the field, and the name of the uploaded file, if provide
 WARNING: request bodies often contain sensitive values like passwords, credit card numbers etc.
 If your service handles data like this, we advise to only enable this feature with care.
 
-[float]
-[[config-flush-interval]]
-==== `flush_interval`
-
-|============
-| Environment                  | Django/Flask     | Default
-| `ELASTIC_APM_FLUSH_INTERVAL` | `FLUSH_INTERVAL` | `10`
-|============
-
-Interval with which transactions should be sent to the APM server, in seconds.
-A lower value will increase the load on your APM server,
-while a higher value can increase the memory pressure on your app.
-A higher value also impacts the time until transactions are indexed and searchable in Elasticsearch.
-
-
 [float]
 [[config-transaction-max-spans]]
 ==== `transaction_max_spans`
@@ -402,8 +388,8 @@ Setting an upper limit will prevent overloading the agent and the APM server wit
 ==== `span_frames_min_duration`
 
 |============
-| Environment                               | Django/Flask            | Default
-| `ELASTIC_APM_SPAN_FRAMES_MIN_DURATION` | `SPAN_FRAMES_MIN_DURATION` | `5`
+| Environment                            | Django/Flask               | Default
+| `ELASTIC_APM_SPAN_FRAMES_MIN_DURATION` | `SPAN_FRAMES_MIN_DURATION` | `"5ms"`
 |============
 
 In its default settings, the APM agent will collect a stack trace with every recorded span.
@@ -416,21 +402,42 @@ with durations equal or longer than the given value in milliseconds, e.g. 5 mill
 
 To disable stack trace collection for spans completely, set the value to `0`.
 
+Except for the special values `-1` and `0`,
+this setting has to be provided in *<<config-format-duration, duration format>>*.
+
 [float]
-[[config-max-queue-size]]
-==== `max_queue_size`
+[[config-api-request-size]]
+==== `api_request_size`
 
 |============
-| Environment                  | Django/Flask     | Default
-| `ELASTIC_APM_MAX_QUEUE_SIZE` | `MAX_QUEUE_SIZE` | `500`
+| Environment                    | Django/Flask       | Default
+| `ELASTIC_APM_API_REQUEST_SIZE` | `API_REQUEST_TIME` | `"724kb"`
+|============
+
+Maximum queue length of the request buffer before sending the request to the APM server.
+A lower value will increase the load on your APM server,
+while a higher value can increase the memory pressure of your app.
+A higher value also impacts the time until data is indexed and searchable in Elasticsearch.
+
+This setting is useful to limit memory consumption if you experience a sudden spike of traffic.
+It has to be provided in *<<config-format-size, size format>>*.
+
+[float]
+[[config-api-request-time]]
+==== `api_request_size`
+
+|============
+| Environment                    | Django/Flask       | Default
+| `ELASTIC_APM_API_REQUEST_SIZE` | `API_REQUEST_SIZE` | `"724kb"`
 |============
 
-Maximum queue length of transactions before sending transactions to the APM server.
+Maximum queue time of the request buffer before sending the request to the APM server.
 A lower value will increase the load on your APM server,
 while a higher value can increase the memory pressure of your app.
-A higher value also impacts the time until transactions are indexed and searchable in Elasticsearch.
+A higher value also impacts the time until data is indexed and searchable in Elasticsearch.
 
 This setting is useful to limit memory consumption if you experience a sudden spike of traffic.
+It has to be provided in *<<config-format-duration, duration format>>*.
 
 [float]
 [[config-processors]]
@@ -551,3 +558,45 @@ By default, the agent verifies the SSL certificate if you use an HTTPS connectio
 Verification can be disabled by changing this setting to `False`.
 
 NOTE: SSL certificate verification is only available in Python 2.7.9+ and Python 3.4.3+.
+
+
+[float]
+[[config-formats]]
+=== Configuration formats
+
+Some options require a unit, either duration or size.
+These need to be provided in a specific format.
+
+[float]
+[[config-format-duration]]
+==== Duration format
+
+The _duration_ format is used for options like timeouts.
+The unit is provided as suffix directly after the number, without and separation by whitespace.
+
+*Example*: `5ms`
+
+*Supported units* 
+
+ * `ms` (milliseconds)
+ * `s` (seconds)
+ * `m` (minutes)
+ 
+[float]
+[[config-format-size]]
+==== Size format
+
+The _size_ format is used for options like maximum buffer sizes.
+The unit is provided as suffix directly after the number, without and separation by whitespace.
+
+
+*Example*: `10kb`
+
+*Supported units*:
+
+ * `b` (bytes)
+ * `kb` (kilobytes)
+ * `mb` (megabytes)
+ * `gb` (gigabytes)
+
+NOTE: we use the power-of-two sizing convention, e.g. `1 kilobyte == 1024 bytes`

docs/sanitizing-data.asciidoc

@@ -12,12 +12,19 @@ This is an example of a processor that removes the exception stacktrace from an
 
 [source,python]
 ----
+from elasticapm.conf.constants import ERROR
+from elasticapm.processors import for_events
+
+@for_events(ERROR)
 def my_processor(client, event):
     if 'exception' in event and 'stacktrace' in event['exception']:
         event['exception'].pop('stacktrace')
     return event
 ----
 
+You can use the `@for_events` decorator to limit for which event type the processor should be called.
+Possible choices are `ERROR`, `TRANSACTION`, and `SPAN`, all of which are defined in `elasticapm.conf.constants`.
+
 To use this processor, update your `ELASTIC_APM` settings like this:
 
 [source,python]

docs/tuning.asciidoc

@@ -47,11 +47,11 @@ holding on to the transaction data in a queue uses memory.
 If you notice that using the Python agent results in a large increase of memory use,
 you can use these settings:
 
- * <<config-flush-interval,`flush_interval`>> to reduce the time between queue flushes
- * <<config-max-queue-size,`max_queue_size`>> to reduce the maximum size of the queue
+ * <<config-api-request-time,`api_request_time`>> to reduce the time between queue flushes
+ * <<config-api-request-size,`api_request_size`>> to reduce the maximum size of the queue
 
-The first setting, `flush_interval`, is helpful if you have a sustained high number of transactions.
-The second setting, `max_queue_size`, can help if you experience peaks of transactions
+The first setting, `api_request_time`, is helpful if you have a sustained high number of transactions.
+The second setting, `api_request_size`, can help if you experience peaks of transactions
 (a large amount of transactions in a short period of time).
 
 Keep in mind that reducing the value of either setting will cause the agent to send more HTTP requests to the APM Server,
@@ -78,6 +78,6 @@ Another option to reduce overhead of collecting contextual data for spans is to
 While this contextual data (specifically, the stack trace) can be very useful to pinpoint where exectly the span is caused in your code,
 it is less interesting for very short spans.
 You can define a minimal threshold for span duration in milliseconds,
-using the <<config-span-frames-min-duration,`span_frames_min_duration_ms`>> setting.
+using the <<config-span-frames-min-duration,`span_frames_min_duration`>> setting.
 If a span takes less than this duration, no stack frames will be collected for this span.
 Other contextual information, like the SQL query, will still be available.