Telemetry: Actions and Groups aren't properly marked as errored #185
Comments
rubberduck203
added a commit
to rubberduck203/scope
that referenced
this issue
Feb 11, 2025
1. Actions and groups are now marked as errored when they fail 2. Set the otel.name so it includes the group or action name This lets us easily query/aggregate which groups or actions are failing most often. This fixes oscope-dev#185 and improves on the solution for oscope-dev#154 implemented in oscope-dev#157. We accomplish the dynamic span naming by leveraging one of the "special fields" detailed here https://docs.rs/tracing-opentelemetry/latest/tracing_opentelemetry/ The naming convention is loosely based on the OTEL http semantic convention of "{method} {url}". Several other of the semantic convention standards are similar as well. https://opentelemetry.io/docs/specs/semconv/http/http-spans/ In order to access the OTEL Span's `set_status()` method, I needed to upgrade all of our opentelemetry dependencies to the latest version. When I did this, traces were no longer being properly flushed before exiting the application, so I updated the implementation based off of this example from the opentelemetry-otlp crate. https://github.com/tokio-rs/tracing-opentelemetry/blob/v0.1.x/examples/opentelemetry-otlp.rs
rubberduck203
added a commit
to rubberduck203/scope
that referenced
this issue
Feb 11, 2025
1. Actions and groups are now marked as errored when they fail 2. Set the otel.name so it includes the group or action name This lets us easily query/aggregate which groups or actions are failing most often. This fixes oscope-dev#185 and improves on the solution for oscope-dev#154 implemented in oscope-dev#157. We accomplish the dynamic span naming by leveraging one of the "special fields" detailed here https://docs.rs/tracing-opentelemetry/latest/tracing_opentelemetry/ The naming convention is loosely based on the OTEL http semantic convention of "{method} {url}". Several other of the semantic convention standards are similar as well. https://opentelemetry.io/docs/specs/semconv/http/http-spans/ In order to access the OTEL Span's `set_status()` method, I needed to upgrade all of our opentelemetry dependencies to the latest version. When I did this, traces were no longer being properly flushed before exiting the application, so I updated the implementation based off of this example from the opentelemetry-otlp crate. https://github.com/tokio-rs/tracing-opentelemetry/blob/v0.1.x/examples/opentelemetry-otlp.rs
rubberduck203
added a commit
that referenced
this issue
Feb 11, 2025
1. Actions and groups are now marked as errored when they fail 2. Set the `otel.name` so it includes the group or action name This lets us easily query/aggregate which groups or actions are failing most often. This fixes #185 and improves on the solution for #154 implemented in #157. We accomplish the dynamic span naming by leveraging one of the "special fields" detailed here https://docs.rs/tracing-opentelemetry/latest/tracing_opentelemetry/ The naming convention is loosely based on the OTEL http semantic convention of `{method} {url}`. Several other of the semantic convention standards are similar as well. https://opentelemetry.io/docs/specs/semconv/http/http-spans/ In order to access the OTEL Span's `set_status()` method, I needed to upgrade all of our `opentelemetry` dependencies to the latest version. When I did this, traces were no longer being properly flushed before exiting the application, so I updated the implementation based off of this example from the `opentelemetry-otlp` crate. https://github.com/tokio-rs/tracing-opentelemetry/blob/v0.1.x/examples/opentelemetry-otlp.rs Screenshot of what the traces look like now 
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Screenshot is from DataDog.
I need to verify this is also an issue with other tools that support OTEL, like Jaeger.
In this screenshot, we can see there were error logs in the
capture_outputand I verified that the action did fail.The action and its parent group should be marked as "errored".
Without these being appropriately flagged, it is hard to create queries and dashboards for things like "Top 10 failed groups".
MVCE
Start jaeger: https://www.jaegertracing.io/docs/1.6/getting-started/#all-in-one-docker-image
cd examples SCOPE_OTEL_ENDPOINT=http://localhost:4318 scope doctor run --only failThe text was updated successfully, but these errors were encountered: