Add predicate to MetricReader and MetricProducer (open-telemetry#3566)

Author: asafm

CHANGELOG.md

@@ -42,6 +42,8 @@ release.
 
 ### Metrics
 
+- Define experimental MetricFilter as a mechanism to filter collected metrics by the MetricReader
+  ([#3566](https://github.com/open-telemetry/opentelemetry-specification/pull/3566))
 - Add optional configuration for Prometheus exporters to promote resource attributes to metric attributes
   ([#3761](https://github.com/open-telemetry/opentelemetry-specification/pull/3761))
 - Clarifications and flexibility in Exemplar speicification.

spec-compliance-matrix.md

@@ -212,6 +212,9 @@ formats is required. Implementing more than one format is optional.
 | The metrics SDK provides an `AlignedHistogramBucketExemplarReservoir` that is used by default for `ExplicitBucketHistogram` aggregation.                               |          |    | +    |     | -      |      |        |     |      |     | -    |       |
 | The metrics SDK provides an `ExemplarFilter` interface or extension point.                                                                                             | X        |    | -    |     | -      |      |        | +   |      |     | -    |       |
 | An `ExemplarFilter` has access to the measurement value, attributes, `Context` and timestamp.                                                                          | X        |    | -    |     | -      |      |        | +   |      |     | -    |       |
+| A metric Producer accepts an optional metric Filter                                                                                                                    |          |    |      |     |        |      |        |     |      |     |      |       |
+| The metric Reader implementation supports registering metric Filter and passing them  its registered metric Producers                                                  |          |    |      |     |        |      |        |     |      |     |      |       |
+| The metric SDK's metric Producer implementations uses the metric Filter                                                                                                |          |    |      |     |        |      |        |     |      |     |      |       |
 
 ## Logs
 

specification/metrics/sdk.md

@@ -74,7 +74,11 @@ linkTitle: SDK
   * [Pull Metric Exporter](#pull-metric-exporter)
 - [MetricProducer](#metricproducer)
   * [Interface Definition](#interface-definition-1)
-    + [Produce() batch](#produce-batch)
+    + [Produce batch](#produce-batch)
+- [MetricFilter](#metricfilter)
+  * [Interface Definition](#interface-definition-2)
+    + [TestMetric](#testmetric)
+    + [TestAttributes](#testattributes)
 - [Defaults and configuration](#defaults-and-configuration)
 - [Numerical limits handling](#numerical-limits-handling)
 - [Compatibility requirements](#compatibility-requirements)
@@ -1113,8 +1117,12 @@ SHOULD provide at least the following:
 * The default output `aggregation` (optional), a function of instrument kind.  If not configured, the [default aggregation](#default-aggregation) SHOULD be used.
 * The default output `temporality` (optional), a function of instrument kind.  If not configured, the Cumulative temporality SHOULD be used.
 * **Status**: [Experimental](../document-status.md) - The default aggregation cardinality limit to use, a function of instrument kind.  If not configured, a default value of 2000 SHOULD be used.
+* **Status**: [Experimental](../document-status.md) - The [MetricFilter](#metricfilter) to apply to metrics and attributes during `MetricReader#Collect`.
 * Zero of more [MetricProducer](#metricproducer)s (optional) to collect metrics from in addition to metrics from the SDK.
 
+**Status**: [Experimental](../document-status.md) - A `MetricReader` SHOULD provide the [MetricFilter](#metricfilter) to the SDK or registered [MetricProducer](#metricproducer)(s)
+when calling the `Produce` operation.
+
 The [MetricReader.Collect](#collect) method allows general-purpose
 `MetricExporter` instances to explicitly initiate collection, commonly
 used with pull-based metrics collection.  A common implementation of
@@ -1502,10 +1510,10 @@ modeled to interact with other components in the SDK:
 
 ## MetricProducer
 
-**Status**: [Stable](../document-status.md)
+**Status**: [Stable](../document-status.md) except where otherwise specified
 
 `MetricProducer` defines the interface which bridges to third-party metric
-sources MUST implement so they can be plugged into an OpenTelemetry
+sources MUST implement, so they can be plugged into an OpenTelemetry
 [MetricReader](#metricreader) as a source of aggregated metric data. The SDK's
 in-memory state MAY implement the `MetricProducer` interface for convenience.
 
@@ -1535,10 +1543,14 @@ bridge pre-processed data.
 
 A `MetricProducer` MUST support the following functions:
 
-#### Produce() batch
+#### Produce batch
 
 `Produce` provides metrics from the MetricProducer to the caller. `Produce`
-MUST return a batch of [Metric points](./data-model.md#metric-points).
+MUST return a batch of [Metric points](./data-model.md#metric-points), filtered by the optional
+`metricFilter` parameter. Implementation SHOULD use the filter as early as
+possible to gain as much performance gain possible (memory allocation,
+internal metric fetching, etc).
+
 If the batch of [Metric points](./data-model.md#metric-points) includes
 resource information, `Produce` SHOULD require a resource as a parameter.
 `Produce` does not have any other required parameters, however, [OpenTelemetry
@@ -1555,6 +1567,80 @@ If a batch of [Metric points](./data-model.md#metric-points) can include
 `Produce` SHOULD include a single InstrumentationScope which identifies the
 `MetricProducer`.
 
+**Parameters:**
+
+**Status**: [Experimental](../document-status.md) `metricFilter`: An optional [MetricFilter](#metricfilter).
+
+## MetricFilter
+
+**Status**: [Experimental](../document-status.md)
+
+`MetricFilter` defines the interface which enables the [MetricReader](#metricreader)'s
+registered [MetricProducers](#metricproducer) or the SDK's [MetricProducer](#metricproducer) to filter aggregated data points
+([Metric points](./data-model.md#metric-points)) inside its `Produce` operation.
+The filtering is done at the [MetricProducer](#metricproducer) for performance reasons.
+
+The `MetricFilter` allows filtering an entire metric stream - dropping or allowing all its attribute sets -
+by its `TestMetric` operation, which accepts the metric stream information
+(scope, name, kind and unit)  and returns an enumeration: `Accept`, `Drop`
+or `Allow_Partial`. If the latter returned, the `TestAttributes` operation
+is to be called per attribute set of that metric stream, returning an enumeration
+determining if the data point for that (metric stream, attributes) pair is to be
+allowed in the result of the [MetricProducer](#metricproducer) `Produce` operation.
+
+### Interface Definition
+
+A `MetricFilter` MUST support the following functions:
+
+#### TestMetric
+
+This operation is called once for every metric stream, in each [MetricProducer](#metricproducer) `Produce`
+operation.
+
+**Parameters:**
+
+- `instrumentationScope`: the metric stream instrumentation scope
+- `name`: the name of the metric stream
+- `kind`: the metric stream [kind](./data-model.md#point-kinds)
+- `unit`: the metric stream unit
+
+Returns: `MetricFilterResult`
+
+`MetricFilterResult` is one of:
+
+* `Accept` - All attributes of the given metric stream are allowed (not to be filtered).
+   This provides a "short-circuit" as there is no need to call `TestAttributes` operation
+   for each attribute set.
+* `Drop` - All attributes of the given metric stream are NOT allowed (filtered out - dropped).
+  This provides a "short-circuit" as there is no need to call `TestAttributes` operation
+  for each attribute set, and no need to collect those data points be it synchronous or asynchronous:
+  e.g. the callback for this given instrument does not need to be invoked.
+* `Accept_Partial` - Some attributes are allowed and some aren't, hence `TestAttributes`
+  operation must be called for each attribute set of that instrument.
+
+#### TestAttributes
+
+An operation which determines for a given metric stream and attribute set if it should be allowed
+or filtered out.
+
+This operation should only be called if `TestMetric` operation returned `Accept_Partial` for
+the given metric stream arguments (`instrumentationScope`, `name`, `kind`, `unit`).
+
+**Parameters:**
+
+- `instrumentationScope`: the metric stream instrumentation scope
+- `name`: the name of the metric stream
+- `kind`: the metric stream kind
+- `unit`: the metric stream unit
+- `attributes`: the attributes
+
+Returns: `AttributesFilterResult`
+
+`AttributesFilterResult` is one of:
+
+* `Accept` - This given `attributes` are allowed (not to be filtered).
+* `Drop` - This given `attributes` are NOT allowed (filtered out - dropped).
+
 ## Defaults and configuration
 
 The SDK MUST provide configuration according to the [SDK environment