Improve benchmark stability by amarziali · Pull Request #11008 · DataDog/dd-trace-java

amarziali · 2026-03-31T07:58:59Z

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

pr-commenter · 2026-03-31T08:53:21Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/benchmarks
git_commit_date	1774925260	1774943903
git_commit_sha	`b7fb6fc`	`99cfd10`
release_version	1.61.0-SNAPSHOT~b7fb6fc123	1.61.0-SNAPSHOT~99cfd101b5

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1774947095	1774947095
ci_job_id	1553241977	1553241977
ci_pipeline_id	105198286	105198286
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-pbp0f9yl 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-pbp0f9yl 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 metrics, 9 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~99cfd101b5, baseline=1.61.0-SNAPSHOT~b7fb6fc123

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1057449
Total [baseline] (8.91 s) : 0, 8910160
Agent [candidate] (1.058 s) : 0, 1058472
Total [candidate] (8.872 s) : 0, 8871772
section iast
Agent [baseline] (1.236 s) : 0, 1235915
Total [baseline] (9.603 s) : 0, 9603405
Agent [candidate] (1.237 s) : 0, 1237438
Total [candidate] (9.595 s) : 0, 9595254

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.057 s	-
Agent	iast	1.236 s	178.466 ms (16.9%)
Total	tracing	8.91 s	-
Total	iast	9.603 s	693.245 ms (7.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	iast	1.237 s	178.966 ms (16.9%)
Total	tracing	8.872 s	-
Total	iast	9.595 s	723.482 ms (8.2%)

gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~99cfd101b5, baseline=1.61.0-SNAPSHOT~b7fb6fc123

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.207 ms) : 0, 1207
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (629.321 ms) : 0, 629321
BytebuddyAgent [candidate] (629.551 ms) : 0, 629551
AgentMeter [baseline] (29.433 ms) : 0, 29433
AgentMeter [candidate] (29.446 ms) : 0, 29446
GlobalTracer [baseline] (256.936 ms) : 0, 256936
GlobalTracer [candidate] (257.772 ms) : 0, 257772
AppSec [baseline] (31.803 ms) : 0, 31803
AppSec [candidate] (31.825 ms) : 0, 31825
Debugger [baseline] (59.664 ms) : 0, 59664
Debugger [candidate] (59.704 ms) : 0, 59704
Remote Config [baseline] (592.19 µs) : 0, 592
Remote Config [candidate] (586.397 µs) : 0, 586
Telemetry [baseline] (8.087 ms) : 0, 8087
Telemetry [candidate] (8.016 ms) : 0, 8016
Flare Poller [baseline] (4.246 ms) : 0, 4246
Flare Poller [candidate] (4.251 ms) : 0, 4251
section iast
crashtracking [baseline] (1.217 ms) : 0, 1217
crashtracking [candidate] (1.225 ms) : 0, 1225
BytebuddyAgent [baseline] (803.281 ms) : 0, 803281
BytebuddyAgent [candidate] (804.113 ms) : 0, 804113
AgentMeter [baseline] (11.55 ms) : 0, 11550
AgentMeter [candidate] (11.462 ms) : 0, 11462
GlobalTracer [baseline] (248.225 ms) : 0, 248225
GlobalTracer [candidate] (249.011 ms) : 0, 249011
IAST [baseline] (25.465 ms) : 0, 25465
IAST [candidate] (25.58 ms) : 0, 25580
AppSec [baseline] (26.661 ms) : 0, 26661
AppSec [candidate] (26.715 ms) : 0, 26715
Debugger [baseline] (68.614 ms) : 0, 68614
Debugger [candidate] (68.188 ms) : 0, 68188
Remote Config [baseline] (537.094 µs) : 0, 537
Remote Config [candidate] (531.077 µs) : 0, 531
Telemetry [baseline] (10.252 ms) : 0, 10252
Telemetry [candidate] (10.471 ms) : 0, 10471
Flare Poller [baseline] (3.7 ms) : 0, 3700
Flare Poller [candidate] (3.79 ms) : 0, 3790

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~99cfd101b5, baseline=1.61.0-SNAPSHOT~b7fb6fc123

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1063614
Total [baseline] (11.068 s) : 0, 11067704
Agent [candidate] (1.061 s) : 0, 1060577
Total [candidate] (11.061 s) : 0, 11061205
section appsec
Agent [baseline] (1.258 s) : 0, 1258050
Total [baseline] (11.244 s) : 0, 11243638
Agent [candidate] (1.259 s) : 0, 1259232
Total [candidate] (11.196 s) : 0, 11196451
section iast
Agent [baseline] (1.234 s) : 0, 1234007
Total [baseline] (11.373 s) : 0, 11373456
Agent [candidate] (1.236 s) : 0, 1235794
Total [candidate] (11.37 s) : 0, 11369845
section profiling
Agent [baseline] (1.188 s) : 0, 1188046
Total [baseline] (11.056 s) : 0, 11056188
Agent [candidate] (1.186 s) : 0, 1186348
Total [candidate] (11.121 s) : 0, 11120723

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.064 s	-
Agent	appsec	1.258 s	194.436 ms (18.3%)
Agent	iast	1.234 s	170.393 ms (16.0%)
Agent	profiling	1.188 s	124.432 ms (11.7%)
Total	tracing	11.068 s	-
Total	appsec	11.244 s	175.934 ms (1.6%)
Total	iast	11.373 s	305.752 ms (2.8%)
Total	profiling	11.056 s	-11.516 ms (-0.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.061 s	-
Agent	appsec	1.259 s	198.655 ms (18.7%)
Agent	iast	1.236 s	175.218 ms (16.5%)
Agent	profiling	1.186 s	125.771 ms (11.9%)
Total	tracing	11.061 s	-
Total	appsec	11.196 s	135.246 ms (1.2%)
Total	iast	11.37 s	308.64 ms (2.8%)
Total	profiling	11.121 s	59.519 ms (0.5%)

gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~99cfd101b5, baseline=1.61.0-SNAPSHOT~b7fb6fc123

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.224 ms) : 0, 1224
crashtracking [candidate] (1.196 ms) : 0, 1196
BytebuddyAgent [baseline] (633.244 ms) : 0, 633244
BytebuddyAgent [candidate] (629.849 ms) : 0, 629849
AgentMeter [baseline] (29.512 ms) : 0, 29512
AgentMeter [candidate] (29.488 ms) : 0, 29488
GlobalTracer [baseline] (257.821 ms) : 0, 257821
GlobalTracer [candidate] (258.267 ms) : 0, 258267
AppSec [baseline] (31.897 ms) : 0, 31897
AppSec [candidate] (31.938 ms) : 0, 31938
Debugger [baseline] (60.745 ms) : 0, 60745
Debugger [candidate] (60.798 ms) : 0, 60798
Remote Config [baseline] (594.072 µs) : 0, 594
Remote Config [candidate] (591.82 µs) : 0, 592
Telemetry [baseline] (8.07 ms) : 0, 8070
Telemetry [candidate] (8.437 ms) : 0, 8437
Flare Poller [baseline] (4.214 ms) : 0, 4214
Flare Poller [candidate] (3.869 ms) : 0, 3869
section appsec
crashtracking [baseline] (1.22 ms) : 0, 1220
crashtracking [candidate] (1.209 ms) : 0, 1209
BytebuddyAgent [baseline] (665.587 ms) : 0, 665587
BytebuddyAgent [candidate] (664.653 ms) : 0, 664653
AgentMeter [baseline] (12.24 ms) : 0, 12240
AgentMeter [candidate] (12.25 ms) : 0, 12250
GlobalTracer [baseline] (259.849 ms) : 0, 259849
GlobalTracer [candidate] (261.372 ms) : 0, 261372
IAST [baseline] (24.292 ms) : 0, 24292
IAST [candidate] (24.595 ms) : 0, 24595
AppSec [baseline] (178.638 ms) : 0, 178638
AppSec [candidate] (179.195 ms) : 0, 179195
Debugger [baseline] (66.987 ms) : 0, 66987
Debugger [candidate] (66.876 ms) : 0, 66876
Remote Config [baseline] (659.081 µs) : 0, 659
Remote Config [candidate] (637.493 µs) : 0, 637
Telemetry [baseline] (8.416 ms) : 0, 8416
Telemetry [candidate] (8.354 ms) : 0, 8354
Flare Poller [baseline] (3.6 ms) : 0, 3600
Flare Poller [candidate] (3.593 ms) : 0, 3593
section iast
crashtracking [baseline] (1.211 ms) : 0, 1211
crashtracking [candidate] (1.21 ms) : 0, 1210
BytebuddyAgent [baseline] (801.003 ms) : 0, 801003
BytebuddyAgent [candidate] (802.436 ms) : 0, 802436
AgentMeter [baseline] (11.43 ms) : 0, 11430
AgentMeter [candidate] (11.442 ms) : 0, 11442
GlobalTracer [baseline] (248.008 ms) : 0, 248008
GlobalTracer [candidate] (248.413 ms) : 0, 248413
IAST [baseline] (25.435 ms) : 0, 25435
IAST [candidate] (25.403 ms) : 0, 25403
AppSec [baseline] (26.611 ms) : 0, 26611
AppSec [candidate] (26.658 ms) : 0, 26658
Debugger [baseline] (70.423 ms) : 0, 70423
Debugger [candidate] (69.498 ms) : 0, 69498
Remote Config [baseline] (536.385 µs) : 0, 536
Remote Config [candidate] (539.778 µs) : 0, 540
Telemetry [baseline] (9.547 ms) : 0, 9547
Telemetry [candidate] (10.225 ms) : 0, 10225
Flare Poller [baseline] (3.533 ms) : 0, 3533
Flare Poller [candidate] (3.764 ms) : 0, 3764
section profiling
crashtracking [baseline] (1.178 ms) : 0, 1178
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (685.519 ms) : 0, 685519
BytebuddyAgent [candidate] (684.799 ms) : 0, 684799
AgentMeter [baseline] (9.022 ms) : 0, 9022
AgentMeter [candidate] (8.988 ms) : 0, 8988
GlobalTracer [baseline] (215.713 ms) : 0, 215713
GlobalTracer [candidate] (215.336 ms) : 0, 215336
AppSec [baseline] (32.397 ms) : 0, 32397
AppSec [candidate] (32.54 ms) : 0, 32540
Debugger [baseline] (66.474 ms) : 0, 66474
Debugger [candidate] (65.488 ms) : 0, 65488
Remote Config [baseline] (584.116 µs) : 0, 584
Remote Config [candidate] (576.265 µs) : 0, 576
Telemetry [baseline] (7.816 ms) : 0, 7816
Telemetry [candidate] (8.166 ms) : 0, 8166
Flare Poller [baseline] (3.546 ms) : 0, 3546
Flare Poller [candidate] (3.871 ms) : 0, 3871
ProfilingAgent [baseline] (94.436 ms) : 0, 94436
ProfilingAgent [candidate] (94.313 ms) : 0, 94313
Profiling [baseline] (94.993 ms) : 0, 94993
Profiling [candidate] (94.865 ms) : 0, 94865

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/benchmarks
git_commit_date	1774925260	1774943903
git_commit_sha	`b7fb6fc`	`99cfd10`
release_version	1.61.0-SNAPSHOT~b7fb6fc123	1.61.0-SNAPSHOT~99cfd101b5

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1774946399	1774946399
ci_job_id	1553241978	1553241978
ci_pipeline_id	105198286	105198286
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-v9hw2t5a 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-v9hw2t5a 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 4 performance improvements and 0 performance regressions! Performance is the same for 17 metrics, 15 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:petclinic:profiling:high_load	better [-1310.140µs; -591.493µs] or [-6.715%; -3.032%]	unsure [-1576.828µs; -384.597µs] or [-5.111%; -1.247%]	unstable [-4.263op/s; +26.747op/s] or [-1.747%; +10.959%]	18.559ms	29.870ms	255.306op/s	19.509ms	30.850ms	244.065op/s
scenario:load:petclinic:no_agent:high_load	better [-1280.319µs; -462.609µs] or [-7.383%; -2.668%]	unsure [-1890.964µs; -12.089µs] or [-6.596%; -0.042%]	unstable [-6.405op/s; +28.825op/s] or [-2.366%; +10.648%]	16.469ms	27.718ms	281.919op/s	17.340ms	28.669ms	270.710op/s
scenario:load:petclinic:appsec:high_load	better [-1.400ms; -0.758ms] or [-7.222%; -3.912%]	better [-1.758ms; -0.820ms] or [-5.666%; -2.642%]	unstable [-0.329op/s; +30.329op/s] or [-0.134%; +12.404%]	18.303ms	29.742ms	259.516op/s	19.382ms	31.031ms	244.516op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~99cfd101b5, baseline=1.61.0-SNAPSHOT~b7fb6fc123
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.234 ms) : 1226, 1242
.   : milestone, 1234,
iast (3.305 ms) : 3276, 3335
.   : milestone, 3305,
iast_FULL (6.022 ms) : 5978, 6065
.   : milestone, 6022,
iast_GLOBAL (3.639 ms) : 3598, 3680
.   : milestone, 3639,
profiling (2.068 ms) : 2055, 2081
.   : milestone, 2068,
tracing (1.842 ms) : 1831, 1852
.   : milestone, 1842,
section candidate
no_agent (1.241 ms) : 1232, 1249
.   : milestone, 1241,
iast (3.293 ms) : 3263, 3324
.   : milestone, 3293,
iast_FULL (5.827 ms) : 5785, 5869
.   : milestone, 5827,
iast_GLOBAL (3.657 ms) : 3619, 3695
.   : milestone, 3657,
profiling (2.05 ms) : 2038, 2063
.   : milestone, 2050,
tracing (1.895 ms) : 1883, 1907
.   : milestone, 1895,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.234 ms [1.226 ms, 1.242 ms]	-
iast	3.305 ms [3.276 ms, 3.335 ms]	2.072 ms (167.9%)
iast_FULL	6.022 ms [5.978 ms, 6.065 ms]	4.788 ms (388.1%)
iast_GLOBAL	3.639 ms [3.598 ms, 3.68 ms]	2.406 ms (195.0%)
profiling	2.068 ms [2.055 ms, 2.081 ms]	834.33 µs (67.6%)
tracing	1.842 ms [1.831 ms, 1.852 ms]	608.316 µs (49.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.241 ms [1.232 ms, 1.249 ms]	-
iast	3.293 ms [3.263 ms, 3.324 ms]	2.053 ms (165.4%)
iast_FULL	5.827 ms [5.785 ms, 5.869 ms]	4.586 ms (369.6%)
iast_GLOBAL	3.657 ms [3.619 ms, 3.695 ms]	2.416 ms (194.8%)
profiling	2.05 ms [2.038 ms, 2.063 ms]	809.421 µs (65.2%)
tracing	1.895 ms [1.883 ms, 1.907 ms]	654.542 µs (52.8%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~99cfd101b5, baseline=1.61.0-SNAPSHOT~b7fb6fc123
    dateFormat X
    axisFormat %s
section baseline
no_agent (17.788 ms) : 17663, 17913
.   : milestone, 17788,
appsec (19.702 ms) : 19560, 19843
.   : milestone, 19702,
code_origins (17.986 ms) : 17862, 18110
.   : milestone, 17986,
iast (17.873 ms) : 17747, 17998
.   : milestone, 17873,
profiling (19.739 ms) : 19600, 19879
.   : milestone, 19739,
tracing (17.809 ms) : 17686, 17932
.   : milestone, 17809,
section candidate
no_agent (17.077 ms) : 16959, 17194
.   : milestone, 17077,
appsec (18.559 ms) : 18427, 18691
.   : milestone, 18559,
code_origins (17.771 ms) : 17646, 17896
.   : milestone, 17771,
iast (17.996 ms) : 17869, 18122
.   : milestone, 17996,
profiling (18.866 ms) : 18734, 18998
.   : milestone, 18866,
tracing (17.771 ms) : 17647, 17895
.   : milestone, 17771,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.788 ms [17.663 ms, 17.913 ms]	-
appsec	19.702 ms [19.56 ms, 19.843 ms]	1.914 ms (10.8%)
code_origins	17.986 ms [17.862 ms, 18.11 ms]	198.162 µs (1.1%)
iast	17.873 ms [17.747 ms, 17.998 ms]	85.015 µs (0.5%)
profiling	19.739 ms [19.6 ms, 19.879 ms]	1.952 ms (11.0%)
tracing	17.809 ms [17.686 ms, 17.932 ms]	21.305 µs (0.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.077 ms [16.959 ms, 17.194 ms]	-
appsec	18.559 ms [18.427 ms, 18.691 ms]	1.482 ms (8.7%)
code_origins	17.771 ms [17.646 ms, 17.896 ms]	694.546 µs (4.1%)
iast	17.996 ms [17.869 ms, 18.122 ms]	918.847 µs (5.4%)
profiling	18.866 ms [18.734 ms, 18.998 ms]	1.789 ms (10.5%)
tracing	17.771 ms [17.647 ms, 17.895 ms]	694.557 µs (4.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/benchmarks
git_commit_date	1774925260	1774943903
git_commit_sha	`b7fb6fc`	`99cfd10`
release_version	1.61.0-SNAPSHOT~b7fb6fc123	1.61.0-SNAPSHOT~99cfd101b5

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1774945984	1774945984
ci_job_id	1553241980	1553241980
ci_pipeline_id	105198286	105198286
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-3-wiybm2pu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-3-wiybm2pu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:dacapo:tomcat:appsec	better [-1.394ms; -1.053ms] or [-37.145%; -28.053%]	2.529ms	3.752ms

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~99cfd101b5, baseline=1.61.0-SNAPSHOT~b7fb6fc123
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.484 ms) : 1473, 1496
.   : milestone, 1484,
appsec (3.752 ms) : 3535, 3970
.   : milestone, 3752,
iast (2.27 ms) : 2201, 2338
.   : milestone, 2270,
iast_GLOBAL (2.301 ms) : 2231, 2370
.   : milestone, 2301,
profiling (2.114 ms) : 2058, 2170
.   : milestone, 2114,
tracing (2.073 ms) : 2020, 2126
.   : milestone, 2073,
section candidate
no_agent (1.485 ms) : 1474, 1497
.   : milestone, 1485,
appsec (2.529 ms) : 2475, 2584
.   : milestone, 2529,
iast (2.27 ms) : 2202, 2339
.   : milestone, 2270,
iast_GLOBAL (2.312 ms) : 2242, 2381
.   : milestone, 2312,
profiling (2.102 ms) : 2048, 2157
.   : milestone, 2102,
tracing (2.093 ms) : 2040, 2147
.   : milestone, 2093,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.484 ms [1.473 ms, 1.496 ms]	-
appsec	3.752 ms [3.535 ms, 3.97 ms]	2.268 ms (152.8%)
iast	2.27 ms [2.201 ms, 2.338 ms]	785.284 µs (52.9%)
iast_GLOBAL	2.301 ms [2.231 ms, 2.37 ms]	816.239 µs (55.0%)
profiling	2.114 ms [2.058 ms, 2.17 ms]	629.891 µs (42.4%)
tracing	2.073 ms [2.02 ms, 2.126 ms]	588.253 µs (39.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.485 ms [1.474 ms, 1.497 ms]	-
appsec	2.529 ms [2.475 ms, 2.584 ms]	1.044 ms (70.3%)
iast	2.27 ms [2.202 ms, 2.339 ms]	785.109 µs (52.9%)
iast_GLOBAL	2.312 ms [2.242 ms, 2.381 ms]	826.3 µs (55.6%)
profiling	2.102 ms [2.048 ms, 2.157 ms]	616.969 µs (41.5%)
tracing	2.093 ms [2.04 ms, 2.147 ms]	608.017 µs (40.9%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~99cfd101b5, baseline=1.61.0-SNAPSHOT~b7fb6fc123
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.551 s) : 15551000, 15551000
.   : milestone, 15551000,
appsec (15.045 s) : 15045000, 15045000
.   : milestone, 15045000,
iast (18.512 s) : 18512000, 18512000
.   : milestone, 18512000,
iast_GLOBAL (17.64 s) : 17640000, 17640000
.   : milestone, 17640000,
profiling (14.892 s) : 14892000, 14892000
.   : milestone, 14892000,
tracing (14.932 s) : 14932000, 14932000
.   : milestone, 14932000,
section candidate
no_agent (15.467 s) : 15467000, 15467000
.   : milestone, 15467000,
appsec (14.784 s) : 14784000, 14784000
.   : milestone, 14784000,
iast (17.98 s) : 17980000, 17980000
.   : milestone, 17980000,
iast_GLOBAL (17.862 s) : 17862000, 17862000
.   : milestone, 17862000,
profiling (14.971 s) : 14971000, 14971000
.   : milestone, 14971000,
tracing (14.821 s) : 14821000, 14821000
.   : milestone, 14821000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.551 s [15.551 s, 15.551 s]	-
appsec	15.045 s [15.045 s, 15.045 s]	-506.0 ms (-3.3%)
iast	18.512 s [18.512 s, 18.512 s]	2.961 s (19.0%)
iast_GLOBAL	17.64 s [17.64 s, 17.64 s]	2.089 s (13.4%)
profiling	14.892 s [14.892 s, 14.892 s]	-659.0 ms (-4.2%)
tracing	14.932 s [14.932 s, 14.932 s]	-619.0 ms (-4.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.467 s [15.467 s, 15.467 s]	-
appsec	14.784 s [14.784 s, 14.784 s]	-683.0 ms (-4.4%)
iast	17.98 s [17.98 s, 17.98 s]	2.513 s (16.2%)
iast_GLOBAL	17.862 s [17.862 s, 17.862 s]	2.395 s (15.5%)
profiling	14.971 s [14.971 s, 14.971 s]	-496.0 ms (-3.2%)
tracing	14.821 s [14.821 s, 14.821 s]	-646.0 ms (-4.2%)

Improve benchmark stability

99cfd10

amarziali closed this Mar 31, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve benchmark stability#11008

Improve benchmark stability#11008
amarziali wants to merge 1 commit intomasterfrom
andrea.marziali/benchmarks

amarziali commented Mar 31, 2026

Uh oh!

pr-commenter bot commented Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

amarziali commented Mar 31, 2026

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Mar 31, 2026

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant