updated monitoring composition for local development (#21180)

Author: pH14

doc/developer/assets/grafana-tempo-trace-id-lookup.png

@@ -144,16 +144,14 @@ _automatically_ collected and exported to analysis tools with _no additional wor
 ### Setup
 
 - For local setup, use the instructions here:
-<https://github.com/MaterializeInc/materialize/tree/main/misc/opentelemetry> to setup a local OpenTelemetry collector
-and ui to view traces.
-- TODO(guswynn): cloud honeycomb setup when its available
-- TODO(guswynn): link to demo video
+<https://github.com/MaterializeInc/materialize/tree/main/misc/monitoring> to setup a local OpenTelemetry collector
+  (Tempo) and UI (Grafana) to view traces.
 
 ### Span visualization
 
-OpenTelemetry UI's like the [Jaeger] one in the local setup _are extraordinarly powerful tools for debugging_. They allow you to visualize
+OpenTelemetry UI's like the Grafana/Tempo one in the local setup _are extraordinarly powerful tools for debugging_. They allow you to visualize
 and inspect the exact control flow graph of your service, including the latency of each operation, and contextual recorded fields. See
-[Best Pratices](#best-practices) for more information.
+[Best Practices](#best-practices) for more information.
 
 ### Distributed tracing
 [OpenTelemetry] allows us to associate `tracing` spans _from distributed services_ with each other, allowing us to not only visualize
@@ -224,6 +222,52 @@ and `mz_ore::tracing::OpenTelemetryContext::attach_as_parent()` on the receiving
 - `TRACE`:
   - exceedingly verbose information intended only for local debugging/tracing
 
+## Accessing Tracing Data Locally
+
+To setup trace collection locally, start the `../../misc/monitoring` composition. It will spin up Tempo to
+automatically start storing traces, and Grafana to visualize them.
+
+Then start `./bin/environmentd --monitoring`.
+
+### Setting the Trace Filter
+
+By default, `./bin/environmentd` will only emit `INFO`-level spans. The filter is controlled through the
+`opentelemetry_filter` system variable and can be toggled dynamically:
+
+```
+> psql -U mz_system -h localhost -p 6877
+
+mz_system=> ALTER SYSTEM SET opentelemetry_filter="debug";
+```
+
+Or on startup:
+
+```
+./bin/environmentd --reset --monitoring -- --system-parameter-default='opentelemetry_filter=debug'
+```
+
+More details on [the filter syntax] can be found in Notion.
+
+### Trace ID Notices
+
+It's often valuable to see the traces associated with specific queries. This can be done
+by setting the session variable:
+
+```
+SET emit_trace_id_notice = true;
+```
+
+Then each subsequent query will emit a NOTICE containing the trace ID for its execution:
+
+```
+materialize=> SELECT 1;
+NOTICE:  trace id: 65e87c160063307a9d1221f78ae55cf8
+```
+
+This trace ID can then be plugged into Grafana's TraceQL lookup for an exact match:
+
+![grafana tempo trace id lookup](./assets/grafana-tempo-trace-id-lookup.png)
+
 
 
 
@@ -240,6 +284,6 @@ and `mz_ore::tracing::OpenTelemetryContext::attach_as_parent()` on the receiving
 [many crates]: https://docs.rs/tracing/latest/tracing/#related-crates
 [OpenTelemetry]: https://opentelemetry.io/
 [here]: https://docs.rs/tracing/latest/tracing/struct.Span.html#in-asynchronous-code
-[Jaeger]: https://www.jaegertracing.io/
 [`tracing::Span`]: https://docs.rs/tracing/latest/tracing/struct.Span.html
 [the docs]: https://dev.materialize.com/api/rust/mz_ore/tracing/struct.OpenTelemetryContext.html
+[the filter syntax]: https://www.notion.so/materialize/Filtering-Logs-and-Traces-6e8fcce8f39e4b45b94ea2923cce05dc?pvs=4

doc/developer/tracing.md

@@ -1,8 +1,13 @@
 # Local monitoring composition
 
-An [mzcompose] composition for running Prometheus and Grafana locally. Metrics
-from all processes run by the `bin/environmentd` script will be automatically
-discovered and scraped.
+An [mzcompose] composition for running a minimal monitoring stack of Prometheus
+(metrics), Tempo (distributed tracing), and Grafana locally. Metrics from all
+processes run by the `bin/environmentd` script will be automatically discovered
+and collected, and traces will be collected when run with `--monitoring`:
+
+```
+./bin/environmentd --monitoring
+```
 
 ## Usage
 
@@ -11,17 +16,23 @@ cd misc/monitoring
 ./mzcompose run default
 ```
 
-To access Prometheus, run `./mzcompose web prometheus` or navigate directly to
-<http://localhost:9090> in your browser.
-
 To access Grafana, run `./mzcompose web grafana` or navigate directly to
-<http://localhost:3000> in your browser.
+<http://localhost:3000> in your browser. Grafana will have datasources
+for both Prometheus and Tempo by default.
+
+If needed, to access Prometheus directly, run `./mzcompose web prometheus`
+or navigate directly to <http://localhost:9090> in your browser.
+
+Tempo does not have its own UI and can be only accessed via Grafana.
+See [the tracing docs] for more info on how to filter and access tracing
+data.
 
 ## Modifications
 
-You can adjust the Prometheus configuration by editing the
-[prometheus.yml](./prometheus.yml) file in this directory. It is bind mounted
-into the container, so edits will be picked up the next time you restart:
+You can adjust the Prometheus and Tempo configurations by editing the
+[prometheus.yml](./prometheus.yml) and [tempo.yml](./tempo.yml) files in
+this directory. They are bind mounted into the containers, so edits will
+be picked up the next time you restart:
 
 ```
 ./mzcompose down -v
@@ -32,4 +43,5 @@ You can adjust the Grafana configuration through its UI. Beware that the
 Grafana state is ephemeral, and will be lost the next time you run
 `./mzcompose down`.
 
+[the tracing docs]: ../../doc/developer/tracing.md#accessing-tracing-data-locally
 [mzcompose]: ../../doc/developer/mzcompose.md

misc/monitoring/README.md

@@ -12,3 +12,5 @@ datasources:
   - name: Prometheus
     type: prometheus
     url: http://prometheus:9090
+    jsonData:
+      timeInterval: 5s

misc/monitoring/grafana/datasources/prometheus.yml

@@ -1,5 +1,3 @@
-#!/usr/bin/env bash
-
 # Copyright Materialize, Inc. and contributors. All rights reserved.
 #
 # Use of this software is governed by the Business Source License
@@ -8,7 +6,9 @@
 # As of the Change Date specified in that file, in accordance with
 # the Business Source License, use of this software will be governed
 # by the Apache License, Version 2.0.
-#
-# mzcompose — runs Docker Compose with Materialize customizations.
 
-exec "$(dirname "$0")"/../../bin/pyactivate -m materialize.cli.mzcompose "$@"
+apiVersion: 1
+datasources:
+  - name: Tempo
+    type: tempo
+    url: http://tempo:3200

misc/opentelemetry/mzcompose renamed to misc/monitoring/grafana/datasources/tempo.yml

@@ -15,20 +15,37 @@
     Service(
         "prometheus",
         {
-            "image": "prom/prometheus:v2.41.0",
+            "image": "prom/prometheus:v2.46.0",
             "ports": ["9090:9090"],
             "volumes": [
                 "./prometheus.yml:/etc/prometheus/prometheus.yml",
                 "../../mzdata/prometheus:/mnt/services",
             ],
+            "command": [
+                "--config.file=/etc/prometheus/prometheus.yml",
+                "--web.enable-remote-write-receiver",
+            ],
             "extra_hosts": ["host.docker.internal:host-gateway"],
             "allow_host_ports": True,
         },
     ),
+    Service(
+        "tempo",
+        {
+            "image": "grafana/tempo:2.2.0",
+            "ports": ["4317:4317", "3200:3200"],
+            "volumes": [
+                "./tempo.yml:/etc/tempo.yml",
+                "../../mzdata/tempo:/tmp/tempo",
+            ],
+            "command": ["-config.file=/etc/tempo.yml"],
+            "allow_host_ports": True,
+        },
+    ),
     Service(
         "grafana",
         {
-            "image": "grafana/grafana:9.3.2",
+            "image": "grafana/grafana:10.0.3",
             "ports": ["3000:3000"],
             "environment": [
                 "GF_AUTH_ANONYMOUS_ENABLED=true",
@@ -44,11 +61,12 @@
 
 
 def workflow_default(c: Composition) -> None:
-    # Create the `mzdata/prometheus` directory that will be bind mounted into
-    # the container before invoking Docker Compose, since otherwise the Docker
-    # daemon will create the directory as root, and `environmentd` won't be
-    # able to write to it.
+    # Create the `mzdata/prometheus|tempo` directories that will be bind mounted into
+    # the containers before invoking Docker Compose, since otherwise the Docker daemon
+    # will create the directory as root, and `environmentd` won't be able to write to them.
     (MZ_ROOT / "mzdata" / "prometheus").mkdir(parents=True, exist_ok=True)
+    (MZ_ROOT / "mzdata" / "tempo").mkdir(parents=True, exist_ok=True)
     c.up()
     print(f"Prometheus running at http://localhost:{c.default_port('prometheus')}")
+    print(f"Tempo running at http://localhost:{c.default_port('tempo')}")
     print(f"Grafana running at http://localhost:{c.default_port('grafana')}")

misc/monitoring/mzcompose.py

@@ -8,7 +8,7 @@
 # by the Apache License, Version 2.0.
 
 global:
-  scrape_interval: 15s
+  scrape_interval: 5s
 scrape_configs:
   - job_name: environmentd
     static_configs:
@@ -20,7 +20,7 @@ scrape_configs:
     file_sd_configs:
     - files:
       - /mnt/services/*.json
-      refresh_interval: 30s
+      refresh_interval: 5s
     relabel_configs:
       # Rewrite references to 127.0.0.1 or 0.0.0.0 to host.docker.internal,
       # since the services are running on the host.
@@ -44,3 +44,9 @@ scrape_configs:
         separator: "-"
         target_label: pod
         replacement: $1-0
+  - job_name: prometheus
+    static_configs:
+      - targets: [ localhost:9090 ]
+  - job_name: tempo
+    static_configs:
+      - targets: [ tempo:3200 ]

misc/monitoring/prometheus.yml

@@ -0,0 +1,46 @@
+# Copyright Materialize, Inc. and contributors. All rights reserved.
+#
+# Use of this software is governed by the Business Source License
+# included in the LICENSE file at the root of this repository.
+#
+# As of the Change Date specified in that file, in accordance with
+# the Business Source License, use of this software will be governed
+# by the Apache License, Version 2.0.
+
+server:
+  http_listen_port: 3200
+
+distributor:
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+
+ingester:
+  max_block_duration: 5m
+
+compactor:
+  compaction:
+    block_retention: 15m
+
+metrics_generator:
+  registry:
+    external_labels:
+      source: tempo
+      cluster: docker-compose
+  storage:
+    path: /tmp/tempo/generator/wal
+    remote_write:
+      - url: http://prometheus:9090/api/v1/write
+        send_exemplars: true
+
+storage:
+  trace:
+    backend: local
+    wal:
+      path: /tmp/tempo/wal
+    local:
+      path: /tmp/tempo/blocks
+
+overrides:
+  metrics_generator_processors: [service-graphs, span-metrics]