Adds crashtracking addresses and error thread name by bric3 · Pull Request #10984 · DataDog/dd-trace-java

bric3 · 2026-03-27T15:11:34Z

What Does This Do

The hotspot crash logs missed some useful addresses to locate instructions, it was also missing the frame type. This change also adds the missing thread name that crashed :

https://github.com/DataDog/libdatadog/blob/main/docs/RFCs/0011-crashtracker-structured-log-format-V1_X.md

Moreover, this PR fixes some frames that were incorrectly parsed, e.g. this function parsed (0 :

J 602  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00000001091a1f54 [0x00000001091a1ec0+0x94]

Motivation

Additional Notes

Frame type marker
The frame type is computed only for HotSpot methods, it maps the HotSpot marker character to a string, now emitted as type in the event.

Marker	`type` value
`J`	`"compiled"`
`A`	`"aot_compiled"`
`j`	`"interpreted"`
`V`	`"vm"`
`v`	`"stub"`
`C`	`"native"`

Note that A, may appear on JDK 11, and was removed later.

Addresses

J/A compiled Java frames

J 602  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00000001091a1f54 [0x00000001091a1ec0+0x94]
  ↑ compilation ID                                        ↑ ip                ↑ symbol_address   ↑ relative_address

Field	Value in example	Meaning
`function`	`java.util.zip.ZipFile.getEntry(J[BZ)J`	JVM method signature
(not captured)	`602`	JIT compilation ID (internal JIT counter for this method)
`ip`	`0x00000001091a1f54`	Program counter at the time of the crash
`symbol_address`	`0x00000001091a1ec0`	Start address of the JIT-compiled method body
`relative_address`	`0x94`	Byte offset from the method start to the crash instruction

C  [libzip.dylib+0x39b8]  Java_java_util_zip_ZipFile_getEntry+0xa8
                 ↑ relative_address

Field	Value in example	Meaning
`relative_address`	`0x39b8`	Byte offset of the call site within the library

Other C/V native frames and j interpreted / v stub frames were not changed: the relative_address from [lib.so+offset] was already captured in master.

FYI, I created this crash reproducer on JDK8 based on this report https://ops.java/troubleshooting/articles/errors-zip-getentry-newentry/

ZipFileMmapCrashRepro

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.Random;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipOutputStream;

public class ZipFileMmapCrashRepro {
  private static final int ENTRY_COUNT = 4096;
  private static final int PAYLOAD_SIZE = 8 * 1024;
  private static final int READER_THREADS = Math.max(4, Runtime.getRuntime().availableProcessors());

  public static void main(String[] args) throws Exception {
    final Path dir = Files.createTempDirectory("zipfile-mmap-crash");
    final Path jar = dir.resolve("target.jar");
    final String[] names = entryNames();
    final AtomicBoolean stop = new AtomicBoolean(false);
    final CountDownLatch started = new CountDownLatch(READER_THREADS + 1);

    writeJarInPlace(jar, names, 0);

    final ZipFile zipFile = new ZipFile(jar.toFile());
    System.out.println("Jar: " + jar);
    System.out.println("Readers: " + READER_THREADS);
    System.out.println("Press Ctrl+C to stop. A JVM crash will terminate the process.");

    for (int i = 0; i < READER_THREADS; i++) {
      final int readerId = i;
      final Thread reader =
          new Thread(
              () -> {
                final Random random = new Random(17L + readerId);
                started.countDown();
                while (!stop.get()) {
                  final String name = names[random.nextInt(names.length)];
                  try {
                    final ZipEntry entry = zipFile.getEntry(name);
                    if (entry == null) {
                      continue;
                    }
                    try (InputStream in = zipFile.getInputStream(entry)) {
                      drain(in);
                    }
                  } catch (IOException ignored) {
                    // Expected when the writer lands mid-update.
                  }
                }
              },
              "zip-reader-" + i);
      reader.setDaemon(true);
      reader.start();
    }

    final Thread writer =
        new Thread(
            () -> {
              int version = 1;
              started.countDown();
              while (!stop.get()) {
                try {
                  writeJarInPlace(jar, names, version++);
                } catch (IOException ignored) {
                }
              }
            },
            "zip-writer");
    writer.setDaemon(true);
    writer.start();

    started.await();
    Thread.sleep(Long.MAX_VALUE);
  }

  private static void writeJarInPlace(Path jar, String[] names, int version) throws IOException {
    final byte[] content = buildJar(names, version);

    try (OutputStream out =
        Files.newOutputStream(
            jar,
            StandardOpenOption.CREATE,
            StandardOpenOption.TRUNCATE_EXISTING,
            StandardOpenOption.WRITE)) {
      // Write in smaller chunks to widen the inconsistent window seen by the readers.
      int offset = 0;
      while (offset < content.length) {
        final int chunk = Math.min(1024, content.length - offset);
        out.write(content, offset, chunk);
        offset += chunk;
        if ((offset & 8191) == 0) {
          Thread.yield();
        }
      }
    }
  }

  private static byte[] buildJar(String[] names, int version) throws IOException {
    final ByteArrayOutputStream buffer = new ByteArrayOutputStream();
    final byte[] payload = payload(version);

    try (ZipOutputStream zip = new ZipOutputStream(buffer)) {
      for (int i = 0; i < names.length; i++) {
        final ZipEntry entry = new ZipEntry(names[i]);
        entry.setTime(1_700_000_000_000L + version + i);
        zip.putNextEntry(entry);
        zip.write(payload);
        zip.write(intToAscii(version));
        zip.closeEntry();
      }
    }

    return buffer.toByteArray();
  }

  private static byte[] payload(int version) {
    final byte[] payload = new byte[PAYLOAD_SIZE + (version & 255)];
    for (int i = 0; i < payload.length; i++) {
      payload[i] = (byte) ('A' + ((i + version) % 26));
    }
    return payload;
  }

  private static byte[] intToAscii(int value) {
    return Integer.toString(value).getBytes();
  }

  private static String[] entryNames() {
    final String[] names = new String[ENTRY_COUNT];
    for (int i = 0; i < names.length; i++) {
      names[i] = String.format("pkg/resource-%04d.txt", i);
    }
    return names;
  }

  private static void drain(InputStream in) throws IOException {
    final byte[] buffer = new byte[4096];
    while (in.read(buffer) >= 0) {
      // drain
    }
  }
}

The RFC 0011 states that https://github.com/DataDog/libdatadog/blob/main/docs/RFCs/0011-crashtracker-structured-log-format-V1_X.md

pr-commenter · 2026-03-27T15:54:52Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/crashlog-method-offset
git_commit_date	1774870153	1774869698
git_commit_sha	17a89d1ec3	`034e3fc`
release_version	1.61.0-SNAPSHOT~517a89d1ec3	1.61.0-SNAPSHOT~034e3fc218

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1774871448	1774871448
ci_job_id	1549631419	1549631419
ci_pipeline_id	105001112	105001112
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-nqkuts2g 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-nqkuts2g 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 64 metrics, 7 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~034e3fc218, baseline=1.61.0-SNAPSHOT~517a89d1ec3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1056589
Total [baseline] (8.834 s) : 0, 8834237
Agent [candidate] (1.058 s) : 0, 1057862
Total [candidate] (8.839 s) : 0, 8838957
section iast
Agent [baseline] (1.229 s) : 0, 1229245
Total [baseline] (9.544 s) : 0, 9543740
Agent [candidate] (1.238 s) : 0, 1238057
Total [candidate] (9.57 s) : 0, 9569800

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.057 s	-
Agent	iast	1.229 s	172.656 ms (16.3%)
Total	tracing	8.834 s	-
Total	iast	9.544 s	709.503 ms (8.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	iast	1.238 s	180.195 ms (17.0%)
Total	tracing	8.839 s	-
Total	iast	9.57 s	730.843 ms (8.3%)

gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~034e3fc218, baseline=1.61.0-SNAPSHOT~517a89d1ec3

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.214 ms) : 0, 1214
BytebuddyAgent [baseline] (628.219 ms) : 0, 628219
BytebuddyAgent [candidate] (629.763 ms) : 0, 629763
AgentMeter [baseline] (29.432 ms) : 0, 29432
AgentMeter [candidate] (29.474 ms) : 0, 29474
GlobalTracer [baseline] (256.498 ms) : 0, 256498
GlobalTracer [candidate] (257.082 ms) : 0, 257082
AppSec [baseline] (31.776 ms) : 0, 31776
AppSec [candidate] (31.823 ms) : 0, 31823
Debugger [baseline] (59.69 ms) : 0, 59690
Debugger [candidate] (59.568 ms) : 0, 59568
Remote Config [baseline] (590.923 µs) : 0, 591
Remote Config [candidate] (594.31 µs) : 0, 594
Telemetry [baseline] (8.044 ms) : 0, 8044
Telemetry [candidate] (8.016 ms) : 0, 8016
Flare Poller [baseline] (5.018 ms) : 0, 5018
Flare Poller [candidate] (4.222 ms) : 0, 4222
section iast
crashtracking [baseline] (1.197 ms) : 0, 1197
crashtracking [candidate] (1.197 ms) : 0, 1197
BytebuddyAgent [baseline] (797.838 ms) : 0, 797838
BytebuddyAgent [candidate] (804.52 ms) : 0, 804520
AgentMeter [baseline] (11.422 ms) : 0, 11422
AgentMeter [candidate] (11.707 ms) : 0, 11707
GlobalTracer [baseline] (247.972 ms) : 0, 247972
GlobalTracer [candidate] (249.072 ms) : 0, 249072
IAST [baseline] (25.401 ms) : 0, 25401
IAST [candidate] (25.567 ms) : 0, 25567
AppSec [baseline] (26.659 ms) : 0, 26659
AppSec [candidate] (27.427 ms) : 0, 27427
Debugger [baseline] (68.363 ms) : 0, 68363
Debugger [candidate] (67.098 ms) : 0, 67098
Remote Config [baseline] (519.135 µs) : 0, 519
Remote Config [candidate] (525.982 µs) : 0, 526
Telemetry [baseline] (10.153 ms) : 0, 10153
Telemetry [candidate] (11.266 ms) : 0, 11266
Flare Poller [baseline] (3.528 ms) : 0, 3528
Flare Poller [candidate] (3.451 ms) : 0, 3451

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~034e3fc218, baseline=1.61.0-SNAPSHOT~517a89d1ec3

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058281
Total [baseline] (11.125 s) : 0, 11125005
Agent [candidate] (1.057 s) : 0, 1057076
Total [candidate] (11.108 s) : 0, 11107607
section appsec
Agent [baseline] (1.262 s) : 0, 1261987
Total [baseline] (11.225 s) : 0, 11225474
Agent [candidate] (1.254 s) : 0, 1253982
Total [candidate] (11.153 s) : 0, 11153344
section iast
Agent [baseline] (1.228 s) : 0, 1228362
Total [baseline] (11.241 s) : 0, 11240506
Agent [candidate] (1.232 s) : 0, 1232224
Total [candidate] (11.292 s) : 0, 11292493
section profiling
Agent [baseline] (1.182 s) : 0, 1182081
Total [baseline] (11.084 s) : 0, 11084469
Agent [candidate] (1.186 s) : 0, 1185618
Total [candidate] (11.076 s) : 0, 11076125

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	appsec	1.262 s	203.705 ms (19.2%)
Agent	iast	1.228 s	170.08 ms (16.1%)
Agent	profiling	1.182 s	123.799 ms (11.7%)
Total	tracing	11.125 s	-
Total	appsec	11.225 s	100.469 ms (0.9%)
Total	iast	11.241 s	115.502 ms (1.0%)
Total	profiling	11.084 s	-40.535 ms (-0.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.057 s	-
Agent	appsec	1.254 s	196.906 ms (18.6%)
Agent	iast	1.232 s	175.148 ms (16.6%)
Agent	profiling	1.186 s	128.542 ms (12.2%)
Total	tracing	11.108 s	-
Total	appsec	11.153 s	45.737 ms (0.4%)
Total	iast	11.292 s	184.885 ms (1.7%)
Total	profiling	11.076 s	-31.482 ms (-0.3%)

gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~034e3fc218, baseline=1.61.0-SNAPSHOT~517a89d1ec3

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.191 ms) : 0, 1191
crashtracking [candidate] (1.205 ms) : 0, 1205
BytebuddyAgent [baseline] (629.749 ms) : 0, 629749
BytebuddyAgent [candidate] (628.835 ms) : 0, 628835
AgentMeter [baseline] (29.436 ms) : 0, 29436
AgentMeter [candidate] (29.419 ms) : 0, 29419
GlobalTracer [baseline] (257.333 ms) : 0, 257333
GlobalTracer [candidate] (256.88 ms) : 0, 256880
AppSec [baseline] (31.863 ms) : 0, 31863
AppSec [candidate] (31.696 ms) : 0, 31696
Debugger [baseline] (60.533 ms) : 0, 60533
Debugger [candidate] (60.259 ms) : 0, 60259
Remote Config [baseline] (586.812 µs) : 0, 587
Remote Config [candidate] (586.7 µs) : 0, 587
Telemetry [baseline] (7.977 ms) : 0, 7977
Telemetry [candidate] (7.994 ms) : 0, 7994
Flare Poller [baseline] (3.542 ms) : 0, 3542
Flare Poller [candidate] (4.228 ms) : 0, 4228
section appsec
crashtracking [baseline] (1.217 ms) : 0, 1217
crashtracking [candidate] (1.208 ms) : 0, 1208
BytebuddyAgent [baseline] (668.276 ms) : 0, 668276
BytebuddyAgent [candidate] (662.39 ms) : 0, 662390
AgentMeter [baseline] (12.258 ms) : 0, 12258
AgentMeter [candidate] (12.207 ms) : 0, 12207
GlobalTracer [baseline] (260.648 ms) : 0, 260648
GlobalTracer [candidate] (259.293 ms) : 0, 259293
IAST [baseline] (24.528 ms) : 0, 24528
IAST [candidate] (24.332 ms) : 0, 24332
AppSec [baseline] (179.107 ms) : 0, 179107
AppSec [candidate] (178.774 ms) : 0, 178774
Debugger [baseline] (66.769 ms) : 0, 66769
Debugger [candidate] (66.621 ms) : 0, 66621
Remote Config [baseline] (628.189 µs) : 0, 628
Remote Config [candidate] (630.792 µs) : 0, 631
Telemetry [baseline] (8.408 ms) : 0, 8408
Telemetry [candidate] (8.508 ms) : 0, 8508
Flare Poller [baseline] (3.637 ms) : 0, 3637
Flare Poller [candidate] (3.609 ms) : 0, 3609
section iast
crashtracking [baseline] (1.189 ms) : 0, 1189
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (797.454 ms) : 0, 797454
BytebuddyAgent [candidate] (799.839 ms) : 0, 799839
AgentMeter [baseline] (11.455 ms) : 0, 11455
AgentMeter [candidate] (11.468 ms) : 0, 11468
GlobalTracer [baseline] (247.202 ms) : 0, 247202
GlobalTracer [candidate] (248.289 ms) : 0, 248289
IAST [baseline] (25.348 ms) : 0, 25348
IAST [candidate] (25.341 ms) : 0, 25341
AppSec [baseline] (27.387 ms) : 0, 27387
AppSec [candidate] (26.433 ms) : 0, 26433
Debugger [baseline] (69.182 ms) : 0, 69182
Debugger [candidate] (70.28 ms) : 0, 70280
Remote Config [baseline] (524.01 µs) : 0, 524
Remote Config [candidate] (534.886 µs) : 0, 535
Telemetry [baseline] (9.132 ms) : 0, 9132
Telemetry [candidate] (9.223 ms) : 0, 9223
Flare Poller [baseline] (3.44 ms) : 0, 3440
Flare Poller [candidate] (3.464 ms) : 0, 3464
section profiling
crashtracking [baseline] (1.167 ms) : 0, 1167
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (682.734 ms) : 0, 682734
BytebuddyAgent [candidate] (685.875 ms) : 0, 685875
AgentMeter [baseline] (8.983 ms) : 0, 8983
AgentMeter [candidate] (8.968 ms) : 0, 8968
GlobalTracer [baseline] (214.819 ms) : 0, 214819
GlobalTracer [candidate] (214.927 ms) : 0, 214927
AppSec [baseline] (32.37 ms) : 0, 32370
AppSec [candidate] (32.198 ms) : 0, 32198
Debugger [baseline] (65.883 ms) : 0, 65883
Debugger [candidate] (65.791 ms) : 0, 65791
Remote Config [baseline] (562.775 µs) : 0, 563
Remote Config [candidate] (565.903 µs) : 0, 566
Telemetry [baseline] (7.698 ms) : 0, 7698
Telemetry [candidate] (7.742 ms) : 0, 7742
Flare Poller [baseline] (3.445 ms) : 0, 3445
Flare Poller [candidate] (3.518 ms) : 0, 3518
ProfilingAgent [baseline] (93.601 ms) : 0, 93601
ProfilingAgent [candidate] (93.734 ms) : 0, 93734
Profiling [baseline] (94.162 ms) : 0, 94162
Profiling [candidate] (94.281 ms) : 0, 94281

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/crashlog-method-offset
git_commit_date	1774870176	1774869698
git_commit_sha	17a89d1ec3	`034e3fc`
release_version	1.61.0-SNAPSHOT~517a89d1ec3	1.61.0-SNAPSHOT~034e3fc218

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1774871946	1774871946
ci_job_id	1549631420	1549631420
ci_pipeline_id	105001112	105001112
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-3-eguguw6l 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-3-eguguw6l 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 5 performance regressions! Performance is the same for 12 metrics, 17 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:iast_GLOBAL:high_load	worse [+170.942µs; +311.963µs] or [+6.348%; +11.584%]	worse [+200.109µs; +594.313µs] or [+2.624%; +7.794%]	unstable [-216.640op/s; +61.765op/s] or [-16.273%; +4.640%]	2.934ms	8.022ms	1253.812op/s	2.693ms	7.625ms	1331.250op/s
scenario:load:insecure-bank:iast_FULL:high_load	worse [+219.565µs; +397.148µs] or [+4.351%; +7.870%]	worse [+263.255µs; +927.323µs] or [+2.171%; +7.646%]	unstable [-119.783op/s; +43.283op/s] or [-14.798%; +5.347%]	5.355ms	12.723ms	771.188op/s	5.046ms	12.128ms	809.438op/s
scenario:load:petclinic:appsec:high_load	worse [+605.118µs; +1280.618µs] or [+3.330%; +7.047%]	same [-264.556µs; +1354.051µs] or [-0.870%; +4.451%]	unstable [-37.549op/s; +14.611op/s] or [-14.962%; +5.822%]	19.115ms	30.969ms	239.500op/s	18.172ms	30.424ms	250.969op/s
scenario:load:petclinic:profiling:high_load	better [-2.541ms; -1.744ms] or [-12.609%; -8.656%]	better [-3.123ms; -1.355ms] or [-9.910%; -4.301%]	unstable [-2.380op/s; +49.442op/s] or [-1.037%; +21.546%]	18.007ms	29.273ms	253.000op/s	20.149ms	31.512ms	229.469op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~034e3fc218, baseline=1.61.0-SNAPSHOT~517a89d1ec3
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.102 ms) : 17920, 18284
.   : milestone, 18102,
appsec (18.594 ms) : 18401, 18787
.   : milestone, 18594,
code_origins (17.782 ms) : 17608, 17955
.   : milestone, 17782,
iast (18.888 ms) : 18696, 19080
.   : milestone, 18888,
profiling (20.351 ms) : 20148, 20555
.   : milestone, 20351,
tracing (17.895 ms) : 17717, 18072
.   : milestone, 17895,
section candidate
no_agent (17.978 ms) : 17795, 18160
.   : milestone, 17978,
appsec (19.493 ms) : 19292, 19693
.   : milestone, 19493,
code_origins (17.685 ms) : 17510, 17860
.   : milestone, 17685,
iast (19.057 ms) : 18863, 19250
.   : milestone, 19057,
profiling (18.443 ms) : 18257, 18630
.   : milestone, 18443,
tracing (17.482 ms) : 17308, 17655
.   : milestone, 17482,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	18.102 ms [17.92 ms, 18.284 ms]	-
appsec	18.594 ms [18.401 ms, 18.787 ms]	492.085 µs (2.7%)
code_origins	17.782 ms [17.608 ms, 17.955 ms]	-319.813 µs (-1.8%)
iast	18.888 ms [18.696 ms, 19.08 ms]	786.713 µs (4.3%)
profiling	20.351 ms [20.148 ms, 20.555 ms]	2.249 ms (12.4%)
tracing	17.895 ms [17.717 ms, 18.072 ms]	-206.957 µs (-1.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.978 ms [17.795 ms, 18.16 ms]	-
appsec	19.493 ms [19.292 ms, 19.693 ms]	1.515 ms (8.4%)
code_origins	17.685 ms [17.51 ms, 17.86 ms]	-292.191 µs (-1.6%)
iast	19.057 ms [18.863 ms, 19.25 ms]	1.079 ms (6.0%)
profiling	18.443 ms [18.257 ms, 18.63 ms]	465.826 µs (2.6%)
tracing	17.482 ms [17.308 ms, 17.655 ms]	-496.001 µs (-2.8%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~034e3fc218, baseline=1.61.0-SNAPSHOT~517a89d1ec3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.182 ms) : 1171, 1193
.   : milestone, 1182,
iast (3.187 ms) : 3144, 3231
.   : milestone, 3187,
iast_FULL (5.71 ms) : 5653, 5767
.   : milestone, 5710,
iast_GLOBAL (3.443 ms) : 3387, 3499
.   : milestone, 3443,
profiling (1.996 ms) : 1978, 2013
.   : milestone, 1996,
tracing (1.816 ms) : 1801, 1832
.   : milestone, 1816,
section candidate
no_agent (1.174 ms) : 1163, 1186
.   : milestone, 1174,
iast (3.247 ms) : 3204, 3291
.   : milestone, 3247,
iast_FULL (5.997 ms) : 5935, 6058
.   : milestone, 5997,
iast_GLOBAL (3.658 ms) : 3598, 3718
.   : milestone, 3658,
profiling (2.035 ms) : 2016, 2053
.   : milestone, 2035,
tracing (1.904 ms) : 1888, 1920
.   : milestone, 1904,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.182 ms [1.171 ms, 1.193 ms]	-
iast	3.187 ms [3.144 ms, 3.231 ms]	2.005 ms (169.6%)
iast_FULL	5.71 ms [5.653 ms, 5.767 ms]	4.528 ms (383.1%)
iast_GLOBAL	3.443 ms [3.387 ms, 3.499 ms]	2.261 ms (191.3%)
profiling	1.996 ms [1.978 ms, 2.013 ms]	813.79 µs (68.8%)
tracing	1.816 ms [1.801 ms, 1.832 ms]	634.369 µs (53.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.174 ms [1.163 ms, 1.186 ms]	-
iast	3.247 ms [3.204 ms, 3.291 ms]	2.073 ms (176.6%)
iast_FULL	5.997 ms [5.935 ms, 6.058 ms]	4.822 ms (410.7%)
iast_GLOBAL	3.658 ms [3.598 ms, 3.718 ms]	2.484 ms (211.5%)
profiling	2.035 ms [2.016 ms, 2.053 ms]	860.553 µs (73.3%)
tracing	1.904 ms [1.888 ms, 1.92 ms]	729.468 µs (62.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/crashlog-method-offset
git_commit_date	1774870251	1774869698
git_commit_sha	17a89d1ec3	`034e3fc`
release_version	1.61.0-SNAPSHOT~517a89d1ec3	1.61.0-SNAPSHOT~034e3fc218

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1774871755	1774871755
ci_job_id	1549631421	1549631421
ci_pipeline_id	105001112	105001112
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-3-k8ysypxx 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-3-k8ysypxx 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~034e3fc218, baseline=1.61.0-SNAPSHOT~517a89d1ec3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (3.808 ms) : 3585, 4030
.   : milestone, 3808,
iast (2.27 ms) : 2200, 2339
.   : milestone, 2270,
iast_GLOBAL (2.303 ms) : 2234, 2373
.   : milestone, 2303,
profiling (2.128 ms) : 2071, 2185
.   : milestone, 2128,
tracing (2.068 ms) : 2014, 2122
.   : milestone, 2068,
section candidate
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (3.807 ms) : 3583, 4031
.   : milestone, 3807,
iast (2.27 ms) : 2200, 2339
.   : milestone, 2270,
iast_GLOBAL (2.309 ms) : 2239, 2379
.   : milestone, 2309,
profiling (2.088 ms) : 2033, 2143
.   : milestone, 2088,
tracing (2.063 ms) : 2009, 2117
.   : milestone, 2063,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	3.808 ms [3.585 ms, 4.03 ms]	2.331 ms (157.9%)
iast	2.27 ms [2.2 ms, 2.339 ms]	793.192 µs (53.7%)
iast_GLOBAL	2.303 ms [2.234 ms, 2.373 ms]	826.653 µs (56.0%)
profiling	2.128 ms [2.071 ms, 2.185 ms]	651.596 µs (44.1%)
tracing	2.068 ms [2.014 ms, 2.122 ms]	591.501 µs (40.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	3.807 ms [3.583 ms, 4.031 ms]	2.331 ms (157.9%)
iast	2.27 ms [2.2 ms, 2.339 ms]	793.506 µs (53.8%)
iast_GLOBAL	2.309 ms [2.239 ms, 2.379 ms]	832.62 µs (56.4%)
profiling	2.088 ms [2.033 ms, 2.143 ms]	611.576 µs (41.4%)
tracing	2.063 ms [2.009 ms, 2.117 ms]	586.863 µs (39.8%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~034e3fc218, baseline=1.61.0-SNAPSHOT~517a89d1ec3
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.353 s) : 15353000, 15353000
.   : milestone, 15353000,
appsec (14.598 s) : 14598000, 14598000
.   : milestone, 14598000,
iast (17.988 s) : 17988000, 17988000
.   : milestone, 17988000,
iast_GLOBAL (17.756 s) : 17756000, 17756000
.   : milestone, 17756000,
profiling (14.858 s) : 14858000, 14858000
.   : milestone, 14858000,
tracing (14.886 s) : 14886000, 14886000
.   : milestone, 14886000,
section candidate
no_agent (14.928 s) : 14928000, 14928000
.   : milestone, 14928000,
appsec (15.191 s) : 15191000, 15191000
.   : milestone, 15191000,
iast (18.407 s) : 18407000, 18407000
.   : milestone, 18407000,
iast_GLOBAL (17.876 s) : 17876000, 17876000
.   : milestone, 17876000,
profiling (15.595 s) : 15595000, 15595000
.   : milestone, 15595000,
tracing (14.888 s) : 14888000, 14888000
.   : milestone, 14888000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.353 s [15.353 s, 15.353 s]	-
appsec	14.598 s [14.598 s, 14.598 s]	-755.0 ms (-4.9%)
iast	17.988 s [17.988 s, 17.988 s]	2.635 s (17.2%)
iast_GLOBAL	17.756 s [17.756 s, 17.756 s]	2.403 s (15.7%)
profiling	14.858 s [14.858 s, 14.858 s]	-495.0 ms (-3.2%)
tracing	14.886 s [14.886 s, 14.886 s]	-467.0 ms (-3.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.928 s [14.928 s, 14.928 s]	-
appsec	15.191 s [15.191 s, 15.191 s]	263.0 ms (1.8%)
iast	18.407 s [18.407 s, 18.407 s]	3.479 s (23.3%)
iast_GLOBAL	17.876 s [17.876 s, 17.876 s]	2.948 s (19.7%)
profiling	15.595 s [15.595 s, 15.595 s]	667.0 ms (4.5%)
tracing	14.888 s [14.888 s, 14.888 s]	-40.0 ms (-0.3%)

...-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/J9JavacoreParser.java

jbachorik

I have just one outstanding question about the J9 non-native frame type. Otherwise, LGTM!

dd-octo-sts · 2026-03-30T14:31:01Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-03-30T14:31:07Z

View all feedbacks in Devflow UI.

2026-03-30 14:31:07 UTC ℹ️ Start processing command /merge

2026-03-30 14:31:12 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-03-30 16:07:57 UTC ℹ️ MergeQueue: This merge request was merged

zhengyu123

Thanks!

bric3 added 4 commits March 27, 2026 10:25

feat(crashtracking): Adds frame type

a1bc213

fix(crashtracking): Incorrect parsing of some frames

8b7e481

feat(crashtracking): Adds addresses on Java frames if available

300662b

feat(crashtracking): Adds the missing error.thread_name

9520985

The RFC 0011 states that https://github.com/DataDog/libdatadog/blob/main/docs/RFCs/0011-crashtracker-structured-log-format-V1_X.md

bric3 requested a review from a team as a code owner March 27, 2026 15:11

bric3 added type: enhancement Enhancements and improvements comp: crash tracking Crash Tracking labels Mar 27, 2026

bric3 requested review from jbachorik and zhengyu123 March 30, 2026 09:45

jbachorik reviewed Mar 30, 2026

View reviewed changes

...-agent/agent-crashtracking/src/main/java/datadog/crashtracking/parsers/J9JavacoreParser.java Outdated Show resolved Hide resolved

jbachorik approved these changes Mar 30, 2026

View reviewed changes

fix(crashtracking): J9 wasn't properly reporting the frame type

2acacde

bric3 enabled auto-merge March 30, 2026 11:04

bric3 added the type: bug Bug report and fix label Mar 30, 2026

chore: make spotless happy

034e3fc

bric3 added this pull request to the merge queue Mar 30, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 30, 2026

zhengyu123 approved these changes Mar 30, 2026

View reviewed changes

gh-worker-dd-mergequeue-cf854d bot merged commit 50402e3 into master Mar 30, 2026
913 of 918 checks passed

gh-worker-dd-mergequeue-cf854d bot deleted the bdu/crashlog-method-offset branch March 30, 2026 16:07

github-actions bot added this to the 1.61.0 milestone Mar 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds crashtracking addresses and error thread name#10984

Adds crashtracking addresses and error thread name#10984
gh-worker-dd-mergequeue-cf854d[bot] merged 6 commits intomasterfrom
bdu/crashlog-method-offset

bric3 commented Mar 27, 2026 •

edited

Loading

Uh oh!

pr-commenter bot commented Mar 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

jbachorik left a comment

Uh oh!

dd-octo-sts bot commented Mar 30, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Mar 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

zhengyu123 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

bric3 commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenter bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!

jbachorik left a comment

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts bot commented Mar 30, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

zhengyu123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bric3 commented Mar 27, 2026 •

edited

Loading

pr-commenter bot commented Mar 27, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Mar 30, 2026 •

edited

Loading