Add span propagation for Pekko scheduled tasks #8765

nikita-tkachenko-datadog · 2025-05-05T10:30:46Z

What Does This Do

Implements propagation of span context for Pekko's asynchronous scheduled tasks.
The new instrumentation plugs into LightArrayRevolverScheduler#schedule method: when a Runnable is scheduled, the context of currently active span is captured and attached to that Runnable.
Later the existing runnable instrumentation activates this context when the task is executed (potentially in a different thread).

Activating a continuation of a span is one-time only, meaning that a context that was captured can only be activated once. For this reason this specific method is instrumented, because it works by re-scheduling periodic tasks after they finish executing (so a new continuation is captured every time a periodic task is triggered).

Motivation

CI Visibility's per-test code coverage relies on spans propagation to capture coverage data from different threads.
Without this instrumentation coverage data for test cases that use Pekko's scheduled tasks is incomplete.

Additional Notes

For now the new instrumentation is disabled by default and has to be enabled explicitly with a config property.
This is done to reduce possible negative impact.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1957

pr-commenter · 2025-05-05T11:20:13Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/pekko-scheduler-instrumentation
git_commit_date	1746516330	1746520049
git_commit_sha	`0e18e0e`	`2ce5ed4`
release_version	1.49.0-SNAPSHOT~0e18e0e887	1.49.0-SNAPSHOT~2ce5ed4064

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746523418	1746523418
ci_job_id	925006416	925006416
ci_pipeline_id	64211741	64211741
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-drymq1by-project-304-concurrent-0-18is7160 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-drymq1by-project-304-concurrent-0-18is7160 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 65 metrics, 6 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.007 s) : 0, 1007163
Total [baseline] (10.516 s) : 0, 10516407
Agent [candidate] (1.006 s) : 0, 1005886
Total [candidate] (10.519 s) : 0, 10518843
section appsec
Agent [baseline] (1.151 s) : 0, 1151272
Total [baseline] (10.71 s) : 0, 10709847
Agent [candidate] (1.155 s) : 0, 1155321
Total [candidate] (10.685 s) : 0, 10684596
section iast
Agent [baseline] (1.137 s) : 0, 1137343
Total [baseline] (10.799 s) : 0, 10798853
Agent [candidate] (1.137 s) : 0, 1136719
Total [candidate] (10.834 s) : 0, 10834103
section profiling
Agent [baseline] (1.261 s) : 0, 1260746
Total [baseline] (10.856 s) : 0, 10855576
Agent [candidate] (1.254 s) : 0, 1254370
Total [candidate] (10.752 s) : 0, 10751706

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.007 s	-
Agent	appsec	1.151 s	144.108 ms (14.3%)
Agent	iast	1.137 s	130.179 ms (12.9%)
Agent	profiling	1.261 s	253.583 ms (25.2%)
Total	tracing	10.516 s	-
Total	appsec	10.71 s	193.44 ms (1.8%)
Total	iast	10.799 s	282.445 ms (2.7%)
Total	profiling	10.856 s	339.169 ms (3.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.006 s	-
Agent	appsec	1.155 s	149.435 ms (14.9%)
Agent	iast	1.137 s	130.833 ms (13.0%)
Agent	profiling	1.254 s	248.484 ms (24.7%)
Total	tracing	10.519 s	-
Total	appsec	10.685 s	165.753 ms (1.6%)
Total	iast	10.834 s	315.26 ms (3.0%)
Total	profiling	10.752 s	232.863 ms (2.2%)

gantt
    title petclinic - break down per module: candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (670.513 ms) : 0, 670513
BytebuddyAgent [candidate] (670.837 ms) : 0, 670837
GlobalTracer [baseline] (240.232 ms) : 0, 240232
GlobalTracer [candidate] (239.564 ms) : 0, 239564
AppSec [baseline] (54.77 ms) : 0, 54770
AppSec [candidate] (54.515 ms) : 0, 54515
Debugger [baseline] (6.146 ms) : 0, 6146
Debugger [candidate] (6.149 ms) : 0, 6149
Remote Config [baseline] (694.791 µs) : 0, 695
Remote Config [candidate] (704.954 µs) : 0, 705
Telemetry [baseline] (11.413 ms) : 0, 11413
Telemetry [candidate] (10.657 ms) : 0, 10657
section appsec
BytebuddyAgent [baseline] (690.224 ms) : 0, 690224
BytebuddyAgent [candidate] (693.26 ms) : 0, 693260
GlobalTracer [baseline] (236.687 ms) : 0, 236687
GlobalTracer [candidate] (237.845 ms) : 0, 237845
AppSec [baseline] (175.19 ms) : 0, 175190
AppSec [candidate] (175.358 ms) : 0, 175358
Debugger [baseline] (5.889 ms) : 0, 5889
Debugger [candidate] (5.882 ms) : 0, 5882
Remote Config [baseline] (643.278 µs) : 0, 643
Remote Config [candidate] (630.785 µs) : 0, 631
Telemetry [baseline] (8.151 ms) : 0, 8151
Telemetry [candidate] (7.78 ms) : 0, 7780
IAST [baseline] (21.854 ms) : 0, 21854
IAST [candidate] (22.017 ms) : 0, 22017
section iast
BytebuddyAgent [baseline] (790.015 ms) : 0, 790015
BytebuddyAgent [candidate] (788.986 ms) : 0, 788986
GlobalTracer [baseline] (230.359 ms) : 0, 230359
GlobalTracer [candidate] (230.484 ms) : 0, 230484
AppSec [baseline] (56.202 ms) : 0, 56202
AppSec [candidate] (56.559 ms) : 0, 56559
Debugger [baseline] (5.92 ms) : 0, 5920
Debugger [candidate] (5.971 ms) : 0, 5971
Remote Config [baseline] (589.248 µs) : 0, 589
Remote Config [candidate] (596.381 µs) : 0, 596
Telemetry [baseline] (7.968 ms) : 0, 7968
Telemetry [candidate] (7.971 ms) : 0, 7971
IAST [baseline] (22.824 ms) : 0, 22824
IAST [candidate] (22.726 ms) : 0, 22726
section profiling
BytebuddyAgent [baseline] (662.136 ms) : 0, 662136
BytebuddyAgent [candidate] (661.115 ms) : 0, 661115
GlobalTracer [baseline] (380.227 ms) : 0, 380227
GlobalTracer [candidate] (378.28 ms) : 0, 378280
AppSec [baseline] (55.296 ms) : 0, 55296
AppSec [candidate] (53.488 ms) : 0, 53488
Debugger [baseline] (6.184 ms) : 0, 6184
Debugger [candidate] (6.11 ms) : 0, 6110
Remote Config [baseline] (669.768 µs) : 0, 670
Remote Config [candidate] (650.246 µs) : 0, 650
Telemetry [baseline] (8.29 ms) : 0, 8290
Telemetry [candidate] (8.115 ms) : 0, 8115
ProfilingAgent [baseline] (97.649 ms) : 0, 97649
ProfilingAgent [candidate] (96.452 ms) : 0, 96452
Profiling [baseline] (97.674 ms) : 0, 97674
Profiling [candidate] (96.475 ms) : 0, 96475

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.016 s) : 0, 1016350
Total [baseline] (8.665 s) : 0, 8665379
Agent [candidate] (1.003 s) : 0, 1003472
Total [candidate] (8.604 s) : 0, 8603955
section iast
Agent [baseline] (1.145 s) : 0, 1144953
Total [baseline] (9.224 s) : 0, 9223889
Agent [candidate] (1.136 s) : 0, 1136445
Total [candidate] (9.209 s) : 0, 9208890
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.136 s) : 0, 1135993
Total [baseline] (9.168 s) : 0, 9167617
Agent [candidate] (1.139 s) : 0, 1139334
Total [candidate] (9.159 s) : 0, 9159491
section iast_TELEMETRY_OFF
Agent [baseline] (1.131 s) : 0, 1130890
Total [baseline] (9.182 s) : 0, 9181618
Agent [candidate] (1.146 s) : 0, 1145755
Total [candidate] (9.223 s) : 0, 9223204

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.016 s	-
Agent	iast	1.145 s	128.603 ms (12.7%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.136 s	119.643 ms (11.8%)
Agent	iast_TELEMETRY_OFF	1.131 s	114.539 ms (11.3%)
Total	tracing	8.665 s	-
Total	iast	9.224 s	558.51 ms (6.4%)
Total	iast_HARDCODED_SECRET_DISABLED	9.168 s	502.238 ms (5.8%)
Total	iast_TELEMETRY_OFF	9.182 s	516.24 ms (6.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.003 s	-
Agent	iast	1.136 s	132.973 ms (13.3%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.139 s	135.861 ms (13.5%)
Agent	iast_TELEMETRY_OFF	1.146 s	142.283 ms (14.2%)
Total	tracing	8.604 s	-
Total	iast	9.209 s	604.936 ms (7.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.159 s	555.536 ms (6.5%)
Total	iast_TELEMETRY_OFF	9.223 s	619.25 ms (7.2%)

gantt
    title insecure-bank - break down per module: candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.954 ms) : 0, 674954
BytebuddyAgent [candidate] (668.362 ms) : 0, 668362
GlobalTracer [baseline] (241.086 ms) : 0, 241086
GlobalTracer [candidate] (238.712 ms) : 0, 238712
AppSec [baseline] (54.85 ms) : 0, 54850
AppSec [candidate] (54.287 ms) : 0, 54287
Debugger [baseline] (8.39 ms) : 0, 8390
Debugger [candidate] (8.254 ms) : 0, 8254
Remote Config [baseline] (708.713 µs) : 0, 709
Remote Config [candidate] (684.968 µs) : 0, 685
Telemetry [baseline] (12.71 ms) : 0, 12710
Telemetry [candidate] (9.655 ms) : 0, 9655
section iast
BytebuddyAgent [baseline] (795.414 ms) : 0, 795414
BytebuddyAgent [candidate] (788.98 ms) : 0, 788980
GlobalTracer [baseline] (231.577 ms) : 0, 231577
GlobalTracer [candidate] (230.401 ms) : 0, 230401
AppSec [baseline] (56.718 ms) : 0, 56718
AppSec [candidate] (56.454 ms) : 0, 56454
Debugger [baseline] (6.027 ms) : 0, 6027
Debugger [candidate] (5.921 ms) : 0, 5921
Remote Config [baseline] (608.962 µs) : 0, 609
Remote Config [candidate] (610.919 µs) : 0, 611
Telemetry [baseline] (8.079 ms) : 0, 8079
Telemetry [candidate] (7.884 ms) : 0, 7884
IAST [baseline] (22.928 ms) : 0, 22928
IAST [candidate] (22.752 ms) : 0, 22752
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (789.092 ms) : 0, 789092
BytebuddyAgent [candidate] (792.014 ms) : 0, 792014
GlobalTracer [baseline] (229.824 ms) : 0, 229824
GlobalTracer [candidate] (230.345 ms) : 0, 230345
AppSec [baseline] (56.416 ms) : 0, 56416
AppSec [candidate] (56.361 ms) : 0, 56361
Debugger [baseline] (5.935 ms) : 0, 5935
Debugger [candidate] (5.872 ms) : 0, 5872
Remote Config [baseline] (582.916 µs) : 0, 583
Remote Config [candidate] (589.792 µs) : 0, 590
Telemetry [baseline] (7.887 ms) : 0, 7887
Telemetry [candidate] (7.882 ms) : 0, 7882
IAST [baseline] (22.837 ms) : 0, 22837
IAST [candidate] (22.824 ms) : 0, 22824
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (785.117 ms) : 0, 785117
BytebuddyAgent [candidate] (793.821 ms) : 0, 793821
GlobalTracer [baseline] (229.488 ms) : 0, 229488
GlobalTracer [candidate] (233.748 ms) : 0, 233748
AppSec [baseline] (56.408 ms) : 0, 56408
AppSec [candidate] (57.386 ms) : 0, 57386
Debugger [baseline] (5.923 ms) : 0, 5923
Debugger [candidate] (6.016 ms) : 0, 6016
Remote Config [baseline] (584.579 µs) : 0, 585
Remote Config [candidate] (615.97 µs) : 0, 616
Telemetry [baseline] (7.749 ms) : 0, 7749
Telemetry [candidate] (7.783 ms) : 0, 7783
IAST [baseline] (22.288 ms) : 0, 22288
IAST [candidate] (22.849 ms) : 0, 22849

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-05-06T08:54:08	2025-05-06T09:01:52
git_branch	master	nikita-tkachenko/pekko-scheduler-instrumentation
git_commit_date	1746516330	1746520049
git_commit_sha	`0e18e0e`	`2ce5ed4`
release_version	1.49.0-SNAPSHOT~0e18e0e887	1.49.0-SNAPSHOT~2ce5ed4064
start_time	2025-05-06T08:53:54	2025-05-06T09:01:38

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746522510	1746522510
ci_job_id	925006417	925006417
ci_pipeline_id	64211741	64211741
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-16jikxse-project-304-concurrent-0-jw1942ih 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-16jikxse-project-304-concurrent-0-jw1942ih 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.365 ms) : 1346, 1384
.   : milestone, 1365,
appsec (1.739 ms) : 1715, 1762
.   : milestone, 1739,
appsec_no_iast (1.754 ms) : 1731, 1777
.   : milestone, 1754,
code_origins (1.672 ms) : 1645, 1699
.   : milestone, 1672,
iast (1.521 ms) : 1497, 1546
.   : milestone, 1521,
profiling (1.528 ms) : 1504, 1551
.   : milestone, 1528,
tracing (1.484 ms) : 1459, 1510
.   : milestone, 1484,
section candidate
no_agent (1.365 ms) : 1345, 1384
.   : milestone, 1365,
appsec (1.751 ms) : 1728, 1775
.   : milestone, 1751,
appsec_no_iast (1.735 ms) : 1712, 1758
.   : milestone, 1735,
code_origins (1.688 ms) : 1662, 1715
.   : milestone, 1688,
iast (1.52 ms) : 1495, 1544
.   : milestone, 1520,
profiling (1.516 ms) : 1493, 1539
.   : milestone, 1516,
tracing (1.497 ms) : 1471, 1522
.   : milestone, 1497,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.365 ms [1.346 ms, 1.384 ms]	-
appsec	1.739 ms [1.715 ms, 1.762 ms]	373.928 µs (27.4%)
appsec_no_iast	1.754 ms [1.731 ms, 1.777 ms]	389.305 µs (28.5%)
code_origins	1.672 ms [1.645 ms, 1.699 ms]	306.883 µs (22.5%)
iast	1.521 ms [1.497 ms, 1.546 ms]	156.345 µs (11.5%)
profiling	1.528 ms [1.504 ms, 1.551 ms]	163.035 µs (11.9%)
tracing	1.484 ms [1.459 ms, 1.51 ms]	119.626 µs (8.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.365 ms [1.345 ms, 1.384 ms]	-
appsec	1.751 ms [1.728 ms, 1.775 ms]	386.43 µs (28.3%)
appsec_no_iast	1.735 ms [1.712 ms, 1.758 ms]	370.566 µs (27.2%)
code_origins	1.688 ms [1.662 ms, 1.715 ms]	323.518 µs (23.7%)
iast	1.52 ms [1.495 ms, 1.544 ms]	154.82 µs (11.3%)
profiling	1.516 ms [1.493 ms, 1.539 ms]	151.314 µs (11.1%)
tracing	1.497 ms [1.471 ms, 1.522 ms]	131.945 µs (9.7%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887
    dateFormat X
    axisFormat %s
section baseline
no_agent (386.792 µs) : 367, 407
.   : milestone, 387,
iast (515.092 µs) : 492, 538
.   : milestone, 515,
iast_FULL (733.733 µs) : 710, 757
.   : milestone, 734,
iast_GLOBAL (575.787 µs) : 552, 600
.   : milestone, 576,
iast_HARDCODED_SECRET_DISABLED (531.206 µs) : 508, 554
.   : milestone, 531,
iast_INACTIVE (467.789 µs) : 445, 490
.   : milestone, 468,
iast_TELEMETRY_OFF (515.135 µs) : 492, 538
.   : milestone, 515,
tracing (470.192 µs) : 448, 492
.   : milestone, 470,
section candidate
no_agent (385.915 µs) : 366, 405
.   : milestone, 386,
iast (520.042 µs) : 497, 543
.   : milestone, 520,
iast_FULL (745.264 µs) : 722, 769
.   : milestone, 745,
iast_GLOBAL (570.204 µs) : 547, 593
.   : milestone, 570,
iast_HARDCODED_SECRET_DISABLED (526.215 µs) : 503, 549
.   : milestone, 526,
iast_INACTIVE (470.462 µs) : 449, 492
.   : milestone, 470,
iast_TELEMETRY_OFF (518.388 µs) : 495, 541
.   : milestone, 518,
tracing (460.684 µs) : 440, 482
.   : milestone, 461,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	386.792 µs [366.81 µs, 406.774 µs]	-
iast	515.092 µs [492.222 µs, 537.962 µs]	128.3 µs (33.2%)
iast_FULL	733.733 µs [710.21 µs, 757.256 µs]	346.941 µs (89.7%)
iast_GLOBAL	575.787 µs [552.037 µs, 599.536 µs]	188.995 µs (48.9%)
iast_HARDCODED_SECRET_DISABLED	531.206 µs [508.484 µs, 553.929 µs]	144.414 µs (37.3%)
iast_INACTIVE	467.789 µs [445.43 µs, 490.149 µs]	80.997 µs (20.9%)
iast_TELEMETRY_OFF	515.135 µs [491.964 µs, 538.305 µs]	128.343 µs (33.2%)
tracing	470.192 µs [448.041 µs, 492.343 µs]	83.4 µs (21.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	385.915 µs [366.453 µs, 405.377 µs]	-
iast	520.042 µs [497.175 µs, 542.91 µs]	134.127 µs (34.8%)
iast_FULL	745.264 µs [722.025 µs, 768.504 µs]	359.349 µs (93.1%)
iast_GLOBAL	570.204 µs [547.038 µs, 593.371 µs]	184.289 µs (47.8%)
iast_HARDCODED_SECRET_DISABLED	526.215 µs [503.122 µs, 549.309 µs]	140.3 µs (36.4%)
iast_INACTIVE	470.462 µs [448.949 µs, 491.974 µs]	84.546 µs (21.9%)
iast_TELEMETRY_OFF	518.388 µs [495.302 µs, 541.475 µs]	132.473 µs (34.3%)
tracing	460.684 µs [439.648 µs, 481.719 µs]	74.768 µs (19.4%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/pekko-scheduler-instrumentation
git_commit_date	1746516330	1746520049
git_commit_sha	`0e18e0e`	`2ce5ed4`
release_version	1.49.0-SNAPSHOT~0e18e0e887	1.49.0-SNAPSHOT~2ce5ed4064

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1746523019	1746523019
ci_job_id	925006418	925006418
ci_pipeline_id	64211741	64211741
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-df2tp9k5-project-304-concurrent-0-mntpkoai 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-df2tp9k5-project-304-concurrent-0-mntpkoai 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.469 ms) : 1458, 1481
.   : milestone, 1469,
appsec (2.405 ms) : 2355, 2455
.   : milestone, 2405,
iast (2.194 ms) : 2132, 2256
.   : milestone, 2194,
iast_GLOBAL (2.225 ms) : 2163, 2287
.   : milestone, 2225,
profiling (2.03 ms) : 1980, 2079
.   : milestone, 2030,
tracing (2.003 ms) : 1955, 2051
.   : milestone, 2003,
section candidate
no_agent (1.471 ms) : 1460, 1483
.   : milestone, 1471,
appsec (2.406 ms) : 2356, 2455
.   : milestone, 2406,
iast (2.184 ms) : 2122, 2246
.   : milestone, 2184,
iast_GLOBAL (2.219 ms) : 2157, 2282
.   : milestone, 2219,
profiling (2.482 ms) : 2237, 2726
.   : milestone, 2482,
tracing (2.002 ms) : 1954, 2050
.   : milestone, 2002,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.469 ms [1.458 ms, 1.481 ms]	-
appsec	2.405 ms [2.355 ms, 2.455 ms]	935.728 µs (63.7%)
iast	2.194 ms [2.132 ms, 2.256 ms]	724.545 µs (49.3%)
iast_GLOBAL	2.225 ms [2.163 ms, 2.287 ms]	755.762 µs (51.4%)
profiling	2.03 ms [1.98 ms, 2.079 ms]	560.511 µs (38.2%)
tracing	2.003 ms [1.955 ms, 2.051 ms]	533.599 µs (36.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.471 ms [1.46 ms, 1.483 ms]	-
appsec	2.406 ms [2.356 ms, 2.455 ms]	934.297 µs (63.5%)
iast	2.184 ms [2.122 ms, 2.246 ms]	713.009 µs (48.5%)
iast_GLOBAL	2.219 ms [2.157 ms, 2.282 ms]	748.17 µs (50.9%)
profiling	2.482 ms [2.237 ms, 2.726 ms]	1.01 ms (68.7%)
tracing	2.002 ms [1.954 ms, 2.05 ms]	530.745 µs (36.1%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.28 s) : 15280000, 15280000
.   : milestone, 15280000,
appsec (15.241 s) : 15241000, 15241000
.   : milestone, 15241000,
iast (18.803 s) : 18803000, 18803000
.   : milestone, 18803000,
iast_GLOBAL (18.034 s) : 18034000, 18034000
.   : milestone, 18034000,
profiling (15.262 s) : 15262000, 15262000
.   : milestone, 15262000,
tracing (14.937 s) : 14937000, 14937000
.   : milestone, 14937000,
section candidate
no_agent (15.214 s) : 15214000, 15214000
.   : milestone, 15214000,
appsec (14.99 s) : 14990000, 14990000
.   : milestone, 14990000,
iast (18.93 s) : 18930000, 18930000
.   : milestone, 18930000,
iast_GLOBAL (18.246 s) : 18246000, 18246000
.   : milestone, 18246000,
profiling (15.145 s) : 15145000, 15145000
.   : milestone, 15145000,
tracing (15.171 s) : 15171000, 15171000
.   : milestone, 15171000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.28 s [15.28 s, 15.28 s]	-
appsec	15.241 s [15.241 s, 15.241 s]	-39.0 ms (-0.3%)
iast	18.803 s [18.803 s, 18.803 s]	3.523 s (23.1%)
iast_GLOBAL	18.034 s [18.034 s, 18.034 s]	2.754 s (18.0%)
profiling	15.262 s [15.262 s, 15.262 s]	-18.0 ms (-0.1%)
tracing	14.937 s [14.937 s, 14.937 s]	-343.0 ms (-2.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.214 s [15.214 s, 15.214 s]	-
appsec	14.99 s [14.99 s, 14.99 s]	-224.0 ms (-1.5%)
iast	18.93 s [18.93 s, 18.93 s]	3.716 s (24.4%)
iast_GLOBAL	18.246 s [18.246 s, 18.246 s]	3.032 s (19.9%)
profiling	15.145 s [15.145 s, 15.145 s]	-69.0 ms (-0.5%)
tracing	15.171 s [15.171 s, 15.171 s]	-43.0 ms (-0.3%)

amarziali

LGTM. Is it possible to add a specific test for it?

nikita-tkachenko-datadog · 2025-05-05T14:11:38Z

LGTM. Is it possible to add a specific test for it?

Done

…tation

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-logging](https://github.com/googleapis/java-logging) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `3.22.2` -> `3.22.3` | | [com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.28.0` -> `2.28.1` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.48.2` -> `1.49.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | --- ### Release Notes <details> <summary>googleapis/java-logging (com.google.cloud:google-cloud-logging)</summary> ### [`v3.22.3`](https://github.com/googleapis/java-logging/blob/HEAD/CHANGELOG.md#3223-2025-05-06) ##### Bug Fixes - **deps:** Update the Java code generator (gapic-generator-java) to 2.56.3 ([844f4fa](googleapis/java-logging@844f4fa)) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.46.3 ([#1801](googleapis/java-logging#1801)) ([d7aa7bc](googleapis/java-logging@d7aa7bc)) - Update dependency com.google.cloud:sdk-platform-java-config to v3.47.0 ([#1803](googleapis/java-logging#1803)) ([5967ffe](googleapis/java-logging@5967ffe)) - Update googleapis/sdk-platform-java action to v2.57.0 ([#1804](googleapis/java-logging#1804)) ([e9a27ec](googleapis/java-logging@e9a27ec)) </details> <details> <summary>googleapis/java-datastore (com.google.cloud:google-cloud-datastore)</summary> ### [`v2.28.1`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2281-2025-05-06) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.47.0 ([#1841](googleapis/java-datastore#1841)) ([ac393e6](googleapis/java-datastore@ac393e6)) - Update googleapis/sdk-platform-java action to v2.57.0 ([#1842](googleapis/java-datastore#1842)) ([0745906](googleapis/java-datastore@0745906)) </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.49.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.49.0): 1.49.0 ### Components #### Configuration at Runtime - ✨ Add process tags as list to remote config payload ([#8705](DataDog/dd-trace-java#8705) - [@amarziali](https://github.com/amarziali)) #### Continuous Integration Visibility - 🐛 Add span propagation for Pekko scheduled tasks ([#8765](DataDog/dd-trace-java#8765) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Update test.retry_reason to use full name of the feature ([#8689](DataDog/dd-trace-java#8689) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - 🧹 Remove unused TestEventsHandler methods ([#8674](DataDog/dd-trace-java#8674) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) #### Dynamic Instrumentation - 🐛 Fix exclude identifiers normalization ([#8742](DataDog/dd-trace-java#8742) - [@jpbempel](https://github.com/jpbempel)) - ✨ Make source file tracking asynchronous ([#8684](DataDog/dd-trace-java#8684) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add scope filtering for symbol extraction ([#8676](DataDog/dd-trace-java#8676) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add support for [@key](https://github.com/key) and [@value](https://github.com/value) for Map filtering ([#8669](DataDog/dd-trace-java#8669) - [@jpbempel](https://github.com/jpbempel)) #### Library Injection - ✨ Add system property to force injection of the tracing library even though multiple javaagents have been detected ([#8697](DataDog/dd-trace-java#8697) - [@cecile75](https://github.com/cecile75)) #### Metrics - ✨ Allow dogstatsd port to be configurable with DD_DOGSTATSD_PORT ([#8693](DataDog/dd-trace-java#8693) - [@randomanderson](https://github.com/randomanderson)) #### Profiling - ✨ Bump ddprof-java to 1.25.1 ([#8750](DataDog/dd-trace-java#8750) - [@jbachorik](https://github.com/jbachorik)) - 🐛 Remove cleanup-on-shutdown for temporary files ([#8746](DataDog/dd-trace-java#8746) - [@jbachorik](https://github.com/jbachorik)) - ✨⚡ Replace a regex-based SMAP parser with a hand-crafted one ([#8730](DataDog/dd-trace-java#8730) - [@jbachorik](https://github.com/jbachorik)) - ✨ Improve error reporting on profiler startup ([#8714](DataDog/dd-trace-java#8714) - [@jbachorik](https://github.com/jbachorik)) - ✨ Exclude ProxyLeakTask exception from exception profiling ([#8666](DataDog/dd-trace-java#8666) - [@jbachorik](https://github.com/jbachorik)) - ✨ Use jvmstat for JDKs 9+ programmatically ([#8641](DataDog/dd-trace-java#8641) - [@MattAlp](https://github.com/MattAlp)) #### Telemetry - ✨ Allow dogstatsd port to be configurable with DD_DOGSTATSD_PORT ([#8693](DataDog/dd-trace-java#8693) - [@randomanderson](https://github.com/randomanderson)) - 🐛 Fix appsec.waf.requests telemetry metric ([#8644](DataDog/dd-trace-java#8644) - [@jandro996](https://github.com/jandro996)) #### Tracer core - ✨ Exclude jackson afterburner dynamic classes from instrumentation ([#8747](DataDog/dd-trace-java#8747) - [@amarziali](https://github.com/amarziali)) - ✨ Introduce Java 8 bytecode bridge for instrumentation API ([#8736](DataDog/dd-trace-java#8736) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ⚡🧹 Use byte-buddy classes optimized for Java8+ ([#8735](DataDog/dd-trace-java#8735) - [@mcculls](https://github.com/mcculls)) - 🐛 Do not set the hibernate or datanucleus span service name when disabled ([#8727](DataDog/dd-trace-java#8727) - [@ygree](https://github.com/ygree)) - ✨ Update bytebuddy and ASM to support JDK 24 ([#8720](DataDog/dd-trace-java#8720) - [@sarahchen6](https://github.com/sarahchen6)) - 🐛 Turn off JDK socket support by default ([#8715](DataDog/dd-trace-java#8715) - [@mcculls](https://github.com/mcculls)) - 🐛 Log warning when trace buffer overflow occurs ([#8712](DataDog/dd-trace-java#8712) - [@ygree](https://github.com/ygree)) - ✨🧪 Introducing an internal integration name ([#8708](DataDog/dd-trace-java#8708) - [@amarziali](https://github.com/amarziali)) - ✨ Add process tags to client stats payload ([#8704](DataDog/dd-trace-java#8704) - [@amarziali](https://github.com/amarziali)) - ✨ Collect process tags for tracing ([#8698](DataDog/dd-trace-java#8698) - [@amarziali](https://github.com/amarziali)) - ✨ Stable Config file: target system properties in process_arguments and support template variables in YamlParser ([#8690](DataDog/dd-trace-java#8690) - [@mtoffl01](https://github.com/mtoffl01)) - ✨⚡ Use prefix trie for proxy ignores ([#8678](DataDog/dd-trace-java#8678) - [@amarziali](https://github.com/amarziali)) - ✨ Allow agent to be automatically injected when running aside Log4J patch agent ([#8648](DataDog/dd-trace-java#8648) - [@paullegranddc](https://github.com/paullegranddc)) - ✨ Use jvmstat for JDKs 9+ programmatically ([#8641](DataDog/dd-trace-java#8641) - [@MattAlp](https://github.com/MattAlp)) #### Tracer internal logging - 🐛 Delete print line ([#8686](DataDog/dd-trace-java#8686) - [@sarahchen6](https://github.com/sarahchen6)) ### Instrumentations #### Akka instrumentation - 🐛 Handle reentrant scope cleanup in Akka/Pekko actor instrumentations ([#8722](DataDog/dd-trace-java#8722) - [@mcculls](https://github.com/mcculls)) #### Apache Spark instrumentation - ✨ Use OpenLineage root parent information to generate trace id ([#8726](DataDog/dd-trace-java#8726) - [@mobuchowski](https://github.com/mobuchowski)) - ✨ Spark job cancellation no longer marks application as failed ([#8701](DataDog/dd-trace-java#8701) - [@paul-laffon-dd](https://github.com/paul-laffon-dd)) #### JDBC instrumentation - 💡 Add support for sybase tds jdbc driver ([#8764](DataDog/dd-trace-java#8764) - [@amarziali](https://github.com/amarziali)) #### Kotlin instrumentation - 🐛 Take defensive copy of parent scope stack when closing nested coroutines ([#8749](DataDog/dd-trace-java#8749) - [@mcculls](https://github.com/mcculls)) #### Reactor instrumentation - ✨⚡ Do not inspect reactor context when not needed ([#8745](DataDog/dd-trace-java#8745) - [@amarziali](https://github.com/amarziali)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 795f347ae34d056efc1194c2f606cee7bca1beea

Add span propagation for Pekko scheduled tasks

ce5a4b2

nikita-tkachenko-datadog added comp: ci visibility Continuous Integration Visibility type: bug labels May 5, 2025

nikita-tkachenko-datadog marked this pull request as ready for review May 5, 2025 11:02

nikita-tkachenko-datadog requested a review from a team as a code owner May 5, 2025 11:02

amarziali approved these changes May 5, 2025

View reviewed changes

Add instrumentation test and a config knob for Pekko scheduler

3f85a74

nikita-tkachenko-datadog requested a review from a team as a code owner May 5, 2025 14:09

nikita-tkachenko-datadog requested a review from ygree May 5, 2025 14:09

nikita-tkachenko-datadog enabled auto-merge (squash) May 5, 2025 14:09

amarziali approved these changes May 5, 2025

View reviewed changes

Merge branch 'master' into nikita-tkachenko/pekko-scheduler-instrumen…

2ce5ed4

…tation

nikita-tkachenko-datadog merged commit 07d1350 into master May 6, 2025
454 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/pekko-scheduler-instrumentation branch May 6, 2025 10:19

github-actions bot added this to the 1.49.0 milestone May 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add span propagation for Pekko scheduled tasks #8765

Add span propagation for Pekko scheduled tasks #8765

nikita-tkachenko-datadog commented May 5, 2025 •

edited by jira bot

Loading

pr-commenter bot commented May 5, 2025 •

edited

Loading

amarziali left a comment

nikita-tkachenko-datadog commented May 5, 2025

Add span propagation for Pekko scheduled tasks #8765

Add span propagation for Pekko scheduled tasks #8765

Conversation

nikita-tkachenko-datadog commented May 5, 2025 • edited by jira bot Loading

What Does This Do

Motivation

Additional Notes

Contributor Checklist

pr-commenter bot commented May 5, 2025 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

amarziali left a comment

Choose a reason for hiding this comment

nikita-tkachenko-datadog commented May 5, 2025

nikita-tkachenko-datadog commented May 5, 2025 •

edited by jira bot

Loading

pr-commenter bot commented May 5, 2025 •

edited

Loading