Compare release artifacts with jardiff before publishing to Maven Central by bric3 · Pull Request #11856 · DataDog/dd-trace-java

bric3 · 2026-07-03T18:37:32Z

What Does This Do

Adds a jar comparison between the artifacts produced by the build job and those produced by the deploy_to_maven_central job before the jar gets published to maven central, i.e. before publishMavenPublicationToSonatypeRepository executes.

If a difference is found the deploy_to_maven_central fails. It will also prevent to publish the jar to github releases.

This PR introduces a dd-trace-java.jardiff plugin that uses under the hood the jardiff. The tool is run with --stat --exit-code to compare a freshly built jar against a reference jar from build.
Failing the build if they differ.

Trimmed task graph (--task-graph) when invoking publishToSonatype :

Tasks graph for: :dd-java-agent:publishToSonatype
\--- :dd-java-agent:publishToSonatype (org.gradle.api.DefaultTask)
     \--- :dd-java-agent:publishMavenPublicationToSonatypeRepository (org.gradle.api.publish.maven.tasks.PublishToMavenRepository)
          +--- :initializeSonatypeStagingRepository (io.github.gradlenexus.publishplugin.InitializeNexusStagingRepository)
          +--- :dd-java-agent:compareToReferenceJar (datadog.gradle.plugin.jardiff.JardiffTask)
          |    \--- :dd-java-agent:shadowJar (com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar) (+)
          +--- :dd-java-agent:generatePomFileForMavenPublication (org.gradle.api.publish.maven.tasks.GenerateMavenPom)
          +--- :dd-java-agent:javadocJar (org.gradle.api.tasks.bundling.Jar)
          |    \--- :dd-java-agent:javadoc (org.gradle.api.tasks.javadoc.Javadoc) (+)
          +--- :dd-java-agent:shadowJar (*)
          +--- :dd-java-agent:signMavenPublication (org.gradle.plugins.signing.Sign)
          |    +--- :dd-java-agent:generatePomFileForMavenPublication (*)
          |    +--- :dd-java-agent:javadocJar (*)
          |    +--- :dd-java-agent:shadowJar (*)
          |    \--- :dd-java-agent:sourcesJar (org.gradle.api.tasks.bundling.Jar) (+)
          \--- :dd-java-agent:sourcesJar (*)

Motivation

The build job produces deliverable jar artifacts, those are used during the whole pipeline for all tests, system-tests, benchmark included. It is also used for OCI images. The build job also pushes to the lib gitlab cache, the Gradle build cache.
However, the deploy_to_maven_central depends on build job and reuse the build cache rehydrated from the lib gitlab cache.

That allows to reuse what was computed in the build job, including the produced artifact, and usually that works fine.

However, if for any reason the build script has a bug, it might result in a different artifact being built, possibly a faulty artifact.
That artifact is the one that gets published to customer of maven central or github release.

Note

The above description iswhat happened in release 1.60.0: a faulty jar reached Maven Central and GitHub.

In order to prevent such a faulty artifact to be published, this PR introduces an intermediate tasks whose role is to compare the artifacts produced from the build job with the one from the current deploy_to_maven_central Gradle invocation.

Additional Notes

This task runs on all publication unless explicitly disabled:
- feature-flagging-api dd-openfeature in particular disable the task at this point because build do not build these jars
- deploy_snapshot_with_ddprof_snapshot excludes it (-x compareToReferenceJar) since its ddprof-snapshot jar differs by design
publishToMavenLocal stays ungated (it is a PublishToMavenLocal, not a PublishToMavenRepository), so local installs are unaffected.

Simple tests to compare to another jar

Run it directly, no credentials or network:

Build a "reference" jar (stand-in for the build job's validated artifact)

./gradlew :dd-trace-api:jar
mkdir -p /tmp/jardiff-ref && cp dd-trace-api/build/libs/dd-trace-api-*.jar /tmp/jardiff-ref/

Compare the freshly built jar against it, if identical this succeeds

./gradlew :dd-trace-api:compareToReferenceJar --reference-jar=/tmp/jardiff-ref/dd-trace-api-<version>.jar

The current wiring do not handle publishToMavenLocal. E.g. ./gradlew :dd-trace-api:publishToMavenLocal installs to ~/.m2 without triggering it.

Simulating a divergent rebuild

Reproducing a second build that produces a different jar, with a real subproject change flowing into a published shadow jar.

dd-trace-ot bundles the :dd-trace-ot:correlation-id-injection subproject (relocated under ddtrot/):

Reference: the shadow jar as it stands today.

./gradlew :dd-trace-ot:shadowJar --build-cache
mkdir -p /tmp/jardiff-ref && cp dd-trace-ot/build/libs/dd-trace-ot-*.jar /tmp/jardiff-ref/

Patch a class in the bundled subproject, e.g. add a method to

dd-trace-ot/correlation-id-injection/src/main/java/datadog/trace/correlation/Slf4jCorrelationIdInjector.java
Any change to a bundled class works: add/remove/rename a method or class)

Rebuild and compare, the changed class is recompiled and re-bundled, as compareToReferenceJar has the jar task as dependency.

The comparison should fail and should pinpoint what's changed.

./gradlew :dd-trace-ot:compareToReferenceJar -PjardiffReferenceDir=/tmp/jardiff-ref  --build-cache
...
> Built jar differs from the reference jar — refusing to publish.
    ddtrot/dd/trace/correlation/Slf4jCorrelationIdInjector.class | 9 +++++++++
    1 files changed, 9 insertions(+), 0 deletions(-)

Other alternatives

During the previous attempt in #11733, an idea emerged to use a non Gradle tool to make the publication, like jreleaser. However, this require a deeper change (build has to make all the necessary artifacts, they need to be in a proper shape to be handed over to jreleaser, configuring jreleaser, make team aware of jreleaser failure modes).

datadog-datadog-prod-us1 · 2026-07-03T18:53:27Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 56.94% (-0.02%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 23288b0 | Docs | Datadog PR Page | Give us feedback!}

dd-octo-sts · 2026-07-03T19:01:34Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.06 s	13.96 s	[-0.0%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent	12.90 s	12.96 s	[-1.2%; +0.2%] (no difference)
startup:petclinic:appsec:Agent	16.95 s	16.68 s	[+0.7%; +2.5%] (maybe worse)
startup:petclinic:iast:Agent	16.34 s	16.94 s	[-8.0%; +0.8%] (no difference)
startup:petclinic:profiling:Agent	16.85 s	16.54 s	[-2.8%; +6.5%] (no difference)
startup:petclinic:sca:Agent	16.89 s	16.81 s	[-0.5%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	16.11 s	15.92 s	[+0.1%; +2.2%] (maybe worse)

Commit: 23288b07 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

`deploy_to_maven_central` rebuilds the artifacts (reusing the build cache) before publishing, so a fault in the build script could make the rebuild to produce a faulty jar that differs from the one the build job. Which was validated across the whole test chain. This add a `compareToReferenceJar` task (via a new `dd-trace-java.jardiff` plugin) that runs jardiff `--stat --exit-code` against a reference jar (the one from `build` job) and fails on any difference. This allows to gate maven publication (and github releases).

mhdatie · 2026-07-06T15:48:52Z

@bric3

However, if for any reason the build script has a bug, it might result in a different artifact being built, possibly a faulty artifact

Is the motivation here that if the build was faulty, then it would result in a cache miss during the deploy to maven step, and that's why we're comparing the two artifacts?

.. Or would the maven deployment step use the faulty build cache anyways? Because that would result in the same build, no?

bric3 · 2026-07-06T16:07:27Z

Is the motivation here that if the build was faulty, then it would result in a cache miss during the deploy to maven step, and that's why we're comparing the two artifacts?

Not necessarily a cache miss, but anything that alters the output in an unintended way, that could include how gradle uses it's build cache and if the build infrastructure we have has "bugs" that surface in these conditions.

.. Or would the maven deployment step use the faulty build cache anyways? Because that would result in the same build, no?

It's supposed to result in the same, however, if Gradle detects out-of-date items, it can re-execute parts of the build to account for that, this is the whole idea behind that feature, only rebuild what changed.

It's unlikely to happen, but if our customization to the build script have an error this can lead to unintended bad interaction. This is what happened during 1.60.0 release where parts of the classes disappeared.

sarahchen6

Cool idea to check the artifacts in a separate Gradle task! I wonder if it'd be better to extract this logic to a separate job?

For example, we could have a separate verify_artifacts job that runs after the build job and before the publishing jobs. This way, we decouple the artifact check logic from the publish command itself, and we can test on master and catch failures before the release itself.

bric3 · 2026-07-07T08:38:53Z

I wonder if it'd be better to extract this logic to a separate job?

Actually no, because the goal is to catch build issues that can happen in this job, because this job is running Gradle and can exercise faulty execution paths that could happen during this job specifically.

I'm planning to have a job that does that on each pipeline though. But if it isn't done in this job the issue could still surface there, in deploy_to_maven_central.

mcculls

LGTM - can you get a second approval from the platform team before merging

sarahchen6

Sorry I may be a bit late for this review to be included in the release..... I think it'd be nice to try this out on PRs first, but otherwise pre-approving !

sarahchen6 · 2026-07-07T14:03:01Z

+ * [jardiff](https://github.com/bric3/jardiff) CLI and fails the build if they differ.
+ *
+ * This guards Maven Central publication against non-deterministic rebuilds: the artifact about to
+ * be published (that maybe rebuilt with faults in a later job) is compared


Suggested change

* be published (that maybe rebuilt with faults in a later job) is compared

* be published (that was possibly rebuilt with faults) is compared

Is this correct? The "reference artifact" is what was published by the build job, and we want to ensure that "the artifact about to be published" wasn't rebuilt with faults?

sarahchen6 · 2026-07-07T14:03:08Z

+      JardiffComparison.Outcome.IDENTICAL ->
+        logger.lifecycle("✓ ${candidate.name} is identical to the reference jar ${reference.name}")
+
+      JardiffComparison.Outcome.DIFFERENT ->


I think it'd help to include instructions in the logs on how to proceed in this case. For example, I assume that if the jars are different, the build and then publish jobs should be re-run.

Same for the ERROR case?

sarahchen6 · 2026-07-07T14:03:13Z

+    assertThat(buildFile("reports/jardiff/comparison.txt")).content()
+      .doesNotContain("**/*.version")
+  }
+


could help to include another test (maybe I missed it) that covers the case when compareToReferenceJar is disabled and no check is done

sarahchen6 · 2026-07-07T14:08:33Z

+    # Preserve the `build` job artifacts (validated across the whole test chain) before Gradle
+    # rebuilds and overwrites them under workspace/**/build/libs. jardiff then compares the jars
+    # about to be published against these references (see https://app.datadoghq.com/incidents/50455).


I don't think we need to include the link to the incident publicly; the logic stands on its own.

Suggested change

# Preserve the `build` job artifacts (validated across the whole test chain) before Gradle

# rebuilds and overwrites them under workspace/**/build/libs. jardiff then compares the jars

# about to be published against these references (see https://app.datadoghq.com/incidents/50455).

# Preserve the `build` job artifacts (validated across the whole test chain) before Gradle

# rebuilds and overwrites them under workspace/**/build/libs.

# jardiff then compares the jars about to be published against these references to ensure artifacts match.

AlexeyKuznetsov-DD · 2026-07-07T14:37:53Z

+ * Pure, Gradle-free helpers for driving the [jardiff](https://github.com/bric3/jardiff) CLI.
+ *
+ * Kept separate from [JardiffTask] so the argument construction and exit-code interpretation can
+ * be unit-tested without spinning up a Gradle build or resolving the jardiff jar.


Not a blocker, just thinking out loud.
What if instead of using external tool, keep it as part of dd-trace-java?
Just take only needed part? WDYT?

mhdatie · 2026-07-07T18:52:20Z

@bric3 what would be the remediation steps after detecting that a previous build does not match the one being released?

If we could actually fall into this state and I suppose on a very rare occasion then we should probably update the release docs with a runbook and bake it into the CI failure logs.

bric3 added type: feature Enhancements and improvements tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jul 3, 2026

bric3 force-pushed the bdu/jardiff-task-before-publish branch 2 times, most recently from c8a141d to 2b41fdf Compare July 3, 2026 21:12

bric3 force-pushed the bdu/jardiff-task-before-publish branch from 2b41fdf to 23288b0 Compare July 3, 2026 21:33

bric3 marked this pull request as ready for review July 3, 2026 21:39

bric3 requested review from a team as code owners July 3, 2026 21:39

bric3 requested review from AlexeyKuznetsov-DD, PerfectSlayer, mcculls, mhlidd, randomanderson, sameerank, sarahchen6 and typotter and removed request for a team July 3, 2026 21:39

sarahchen6 reviewed Jul 6, 2026

View reviewed changes

mcculls approved these changes Jul 7, 2026

View reviewed changes

sarahchen6 approved these changes Jul 7, 2026

View reviewed changes

AlexeyKuznetsov-DD reviewed Jul 7, 2026

View reviewed changes

	* be published (that maybe rebuilt with faults in a later job) is compared
	* be published (that was possibly rebuilt with faults) is compared

Uh oh!

Conversation

bric3 commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Simple tests to compare to another jar

Simulating a divergent rebuild

Other alternatives

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 3, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

mhdatie commented Jul 6, 2026

Uh oh!

bric3 commented Jul 6, 2026

Uh oh!

sarahchen6 left a comment

Choose a reason for hiding this comment

Uh oh!

bric3 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mcculls left a comment

Choose a reason for hiding this comment

Uh oh!

sarahchen6 left a comment

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

AlexeyKuznetsov-DD Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

mhdatie commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bric3 commented Jul 3, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 3, 2026 •

edited by datadog-official Bot

Loading

dd-octo-sts Bot commented Jul 3, 2026 •

edited

Loading

bric3 commented Jul 7, 2026 •

edited

Loading

mhdatie commented Jul 7, 2026 •

edited

Loading