Enable symbolication of native stack frames in ANR events #4061

lealanko-rt · 2025-01-17T21:20:19Z

📜 Description

Add instruction address information and debug image information to AnrV2 events. This enables symbolication of native stack frames in stripped libraries.

Before

After

💡 Motivation and Context

Earlier, it was nearly impossible to debug ANRs caused by native code in stripped libraries, since no address information was passed in the events. The introduction of io.sentry.anr.attach-thread-dumps (#3791) at least made the raw data available for manual investigation, but that was toilsome.

Symbolication of a stack frame requires an instruction address and a way of identifying a debug image. Both of these are available in the Android stack dumps that AnrV2 reads, the latter in the form of a BuildId. By sending this data and annotating the stack frames correctly, the Sentry server is able to symbolicate them.

💚 How did you test it?

Unit tests were modified to test all added features. Moreover, a button for triggering an ANR via native code was added to the sample Android application. The results can be seen in the screenshots above.

📝 Checklist

I added tests to verify the changes.
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
I updated the docs if needed.
I updated the wizard if needed.
Review from the native team if needed.
No breaking change or entry added to the changelog.
No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

Functionally this is pretty complete, I think. The rendering for managed frames in a hybrid stack (as seen in the second screenshot) is a bit ugly, and omits the module (i.e. package+class), so that could perhaps be improved a little.

A natural continuation of this work would be tackling #3295, which is fundamentally about the same thing, but needs to deserialize a protobuf tombstone instead of a textual dump, so there is not much code to be reused.

markushi · 2025-01-22T08:18:57Z

@lealanko-rt thanks for opening this up, that's amazing! We'll have a closer look at this asap.

sentry-android-core/src/main/java/io/sentry/android/core/AnrV2Integration.java

lealanko-rt · 2025-01-22T10:23:38Z

@lealanko-rt thanks for opening this up, that's amazing! We'll have a closer look at this asap.

Hi, thanks for working on this.

What is the workflow for PRs here? My own preference is to absorb all fixes into appropriate commits to produce a clean history, but I'm not sure if that's feasible with multiple people contributing to the branch...

markushi · 2025-03-13T09:46:31Z

@lealanko-rt thanks for opening this up, that's amazing! We'll have a closer look at this asap.

Hi, thanks for working on this.

What is the workflow for PRs here? My own preference is to absorb all fixes into appropriate commits to produce a clean history, but I'm not sure if that's feasible with multiple people contributing to the branch...

Sorry for the late reply. For sentry-java we always squash PRs into a single commit before merging, as it allows us to revert changes more easily (in case of emergency). I know it has a few downsides and isn't that clean, but it's a trade-off 😅

Add a button to trigger ANR by holding a lock too long in native code. This can be used to test native stack frames in ANR events.

Handle offsets and deleted files, recognize "???" as a marker for unknown functions. Use named capturing groups for better readability and editability.

Use the "native" attribute of stack frames to indicate JNI invocation frames, like SentryStackTraceFactory does.

The images are parsed from the build ids and filenames in the thread dump's stack frames.

The instruction addresses of native stack frames in thread dumps are relative to the image file from which the code is loaded, and there are no absolute mapping addresses of images available. So explicitly inform the Sentry server about the correct images by using a relative "addr_mode" attribute. Also add the attribute to the SentryStackFrame class since it was not yet supported by it. The field documentation is converted from event.schema.json in the sentry server repo.

...-android-core/src/main/java/io/sentry/android/core/internal/threaddump/ThreadDumpParser.java

…nal/threaddump/ThreadDumpParser.java

...-android-core/src/main/java/io/sentry/android/core/internal/threaddump/ThreadDumpParser.java

…nal/threaddump/ThreadDumpParser.java

romtsn · 2025-03-14T15:42:39Z

...-android-core/src/main/java/io/sentry/android/core/internal/threaddump/ThreadDumpParser.java

+        final String debugId = buildId == null ? null : buildIdToDebugId(buildId);
+        if (debugId != null) {
+          if (!debugImages.containsKey(debugId)) {
+            final DebugImage debugImage = new DebugImage();


@markushi I'm reading our dev docs which say that image_addr and image_size are required, but apparently we're not setting them here and symbolicator is still able to symbolicate the frames. Shall we amend the docs? Or is there a possibility of wrong symbolication given we don't have all the sufficient information here?

Good point, I think that's fine because
1.) the native dumps doesn't seem to even provide this information (which makes sense, as the state is not in memory anymore)
2.) we set the address mode to relative for every frame here as well:
frame.setAddrMode("rel:" + debugId); (link)

Maybe @loewenheim could chime in here as an expert? If all frame addresses are relative, is it fine to provide no image_addr/ image_size in debug meta?

Image size is always optional (we just conservatively assume it extends to the beginning of the next image if it's not set), and the base address is only required for the absolute address mode. As you say, it's fine to leave it out with the relative address mode.

I'm a bit confused about this, though:

we set the address mode to relative for every frame here as well:
frame.setAddrMode("rel:" + debugId); (link)

The argument of rel in Symbolicator is not a debug ID, it's the index of the module in the debug_meta/images list in the event.
https://github.com/getsentry/symbolicator/blob/2e229b8470fabc5051677ef07d1e77217f921bd2/crates/symbolicator-native/src/interface.rs#L548-L556

Is this fixed up at some later point where the debug ID is replaced by an index? Otherwise this won't work.

Actually, I tell a lie, this is taken care of in sentry: https://github.com/getsentry/sentry/blob/f242899987f9645bd629b80ef2c86361e53819d4/src/sentry/lang/native/processing.py#L371-L389

alright @loewenheim thanks for clearing this out! Do you think we should update the dev docs to specify those are not required in the case of relative mode?

Yes, I would say so.

romtsn

@lealanko-rt LGTM, thanks for this PR. While it looks like not much has been changed, it has massive impact and we appreciate your contribution! This is dope.

romtsn · 2025-03-17T16:41:13Z

@lealanko-rt I've merged the PR, it'll be available with the next version (likely 8.5.0). Thanks for your contribution again!

lealanko-rt requested review from adinauer, romtsn, stefanosiano and markushi as code owners January 17, 2025 21:20

lealanko-rt force-pushed the feat/anr-native-symbolication branch 7 times, most recently from eebfcdf to b1f58bf Compare January 20, 2025 08:33

lealanko-rt commented Jan 22, 2025

View reviewed changes

sentry-android-core/src/main/java/io/sentry/android/core/AnrV2Integration.java Show resolved Hide resolved

markushi requested a review from lcian as a code owner March 13, 2025 09:42

Lauri Alanko and others added 9 commits March 13, 2025 14:55

Add native ANR button to sentry-samples-android

14c4f0c

Add a button to trigger ANR by holding a lock too long in native code. This can be used to test native stack frames in ANR events.

Improve native stack frame parsing

3448e7d

Handle offsets and deleted files, recognize "???" as a marker for unknown functions. Use named capturing groups for better readability and editability.

Add PC value and platform to native stack frames

9240d3f

Mark JNI method frames as "native"

47f44f0

Use the "native" attribute of stack frames to indicate JNI invocation frames, like SentryStackTraceFactory does.

Add debug images to ANR events

49e37b3

The images are parsed from the build ids and filenames in the thread dump's stack frames.

Add ChangeLog entry for ANR native symbolication

79216e2

Fix code formatting, make API 21 level compatible, minor improvements

4ae20df

Add ADDR_MODE entries to sentry.api

cb6f40e

lealanko-rt force-pushed the feat/anr-native-symbolication branch from e1ada90 to cb6f40e Compare March 13, 2025 18:10

romtsn reviewed Mar 14, 2025

View reviewed changes

...-android-core/src/main/java/io/sentry/android/core/internal/threaddump/ThreadDumpParser.java Outdated Show resolved Hide resolved

Update sentry-android-core/src/main/java/io/sentry/android/core/inter…

b553c09

…nal/threaddump/ThreadDumpParser.java

romtsn reviewed Mar 14, 2025

View reviewed changes

...-android-core/src/main/java/io/sentry/android/core/internal/threaddump/ThreadDumpParser.java Outdated Show resolved Hide resolved

Update sentry-android-core/src/main/java/io/sentry/android/core/inter…

fcce7b2

…nal/threaddump/ThreadDumpParser.java

romtsn reviewed Mar 14, 2025

View reviewed changes

Merge branch 'main' into feat/anr-native-symbolication

cb6fb86

romtsn approved these changes Mar 14, 2025

View reviewed changes

Merge branch 'main' into feat/anr-native-symbolication

11c5574

romtsn merged commit 6be3488 into getsentry:main Mar 17, 2025
34 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable symbolication of native stack frames in ANR events #4061

Enable symbolication of native stack frames in ANR events #4061

lealanko-rt commented Jan 17, 2025 •

edited

Loading

markushi commented Jan 22, 2025

lealanko-rt commented Jan 22, 2025

markushi commented Mar 13, 2025

romtsn Mar 14, 2025

markushi Mar 17, 2025

loewenheim Mar 17, 2025 •

edited

Loading

loewenheim Mar 17, 2025

romtsn Mar 17, 2025

loewenheim Mar 17, 2025

romtsn left a comment

romtsn commented Mar 17, 2025

Enable symbolication of native stack frames in ANR events #4061

Enable symbolication of native stack frames in ANR events #4061

Conversation

lealanko-rt commented Jan 17, 2025 • edited Loading

📜 Description

Before

After

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

markushi commented Jan 22, 2025

lealanko-rt commented Jan 22, 2025

markushi commented Mar 13, 2025

romtsn Mar 14, 2025

Choose a reason for hiding this comment

markushi Mar 17, 2025

Choose a reason for hiding this comment

loewenheim Mar 17, 2025 • edited Loading

Choose a reason for hiding this comment

loewenheim Mar 17, 2025

Choose a reason for hiding this comment

romtsn Mar 17, 2025

Choose a reason for hiding this comment

loewenheim Mar 17, 2025

Choose a reason for hiding this comment

romtsn left a comment

Choose a reason for hiding this comment

romtsn commented Mar 17, 2025

lealanko-rt commented Jan 17, 2025 •

edited

Loading

loewenheim Mar 17, 2025 •

edited

Loading