Skip to content

Skip context extraction for Kafka records without headers - #13177

Draft
dougqh wants to merge 3 commits into
masterfrom
dougqh/kafka-empty-headers-fast-path
Draft

dougqh wants to merge 3 commits into
masterfrom
dougqh/kafka-empty-headers-fast-path

Conversation

@dougqh

@dougqh dougqh commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Kafka consumers and producers (kafka-clients 0.11 and 3.8) now skip context extraction for a record with no headers. Such a record can't carry a context, but extraction still built its carrier caches and ran every propagator over nothing, for every record consumed and every record sent.

Behavior doesn't change: with no headers, extraction always returned no context, so a record without headers still gets no extracted context.

Motivation

Records without headers are common. A consumer sees them from untraced producers, and a producer sees them on most of what it sends, since its own headers are only added afterward.

KafkaConsumeScenariosBenchmark gains a traceHeaders parameter; false gives records with no headers. 500-record batches through TracingIterator (JDK 17, 3 forks, per record):

dataStreams traceHeaders before ns after ns before B after B
false false 597.0 ± 4.2 533.9 ± 3.0 1912 1528
true false 955.7 ± 15.2 882.9 ± 3.1 2912 2512
false true 1639.5 ± 16.2 1632.2 ± 10.7 3424 3425
true true 2091.5 ± 39.6 2088.6 ± 36.4 4669 4657

For a record without headers that's about 390 B and 65 ns less, about a fifth of the tracer's allocation for that record. Records with trace headers are unchanged. The producer path makes the same extraction call on each send, so it should save about the same, though only the consumer is benchmarked here.

Additional Notes

  • How "no headers" is detected: Kafka's Headers has no size() or isEmpty(). toArray() would copy every traced record's headers just to test for none, so hasHeaders checks iterator().hasNext(). The benchmark shows the iterator doesn't allocate on traced records.
  • Kafka Streams is unchanged. Its advice always attaches the extracted context as a scope, so skipping extraction there isn't a pure no-op and needs its own look.

🤖 Generated with Claude Code

dougqh and others added 3 commits November 3, 2026 09:01
Extracting from a record with no headers can't find a context, but it still
built the carrier caches and ran every propagator, for every record consumed
and every record sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh dougqh added tag: performance Performance related changes type: refactoring inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM labels Nov 3, 2026
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 64.24% (+4.48%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: de3a7c9 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.92 s 13.94 s [-0.8%; +0.4%] (no difference)
startup:insecure-bank:tracing:Agent 12.94 s 12.95 s [-1.1%; +0.9%] (no difference)
startup:petclinic:appsec:Agent 17.15 s 17.04 s [-0.4%; +1.7%] (no difference)
startup:petclinic:iast:Agent 17.06 s 17.01 s [-0.5%; +1.1%] (no difference)
startup:petclinic:profiling:Agent 16.67 s 16.73 s [-1.3%; +0.5%] (no difference)
startup:petclinic:sca:Agent 17.17 s 17.03 s [-0.0%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.38 s 16.18 s [+0.2%; +2.3%] (maybe worse)

Commit: de3a7c95 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@pr-commenter

pr-commenter Bot commented Nov 3, 2026

Copy link
Copy Markdown

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/kafka-empty-headers-fast-path
git_commit_date 1793706372 1793714463
git_commit_sha fb71020 de3a7c9
See matching parameters
Baseline Candidate
ci_job_date 1793715912 1793715912
ci_job_id 2206454008 2206454008
ci_pipeline_id 149775025 149775025
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-odd93kw5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-odd93kw5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce unsure
[-8032.532op/s; -1166.767op/s] or [-4.808%; -0.698%]
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter

pr-commenter Bot commented Nov 3, 2026

Copy link
Copy Markdown

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/kafka-empty-headers-fast-path
git_commit_date 1793706372 1793714463
git_commit_sha fb71020 de3a7c9
See matching parameters
Baseline Candidate
ci_job_date 1793715963 1793715963
ci_job_id 2206454012 2206454012
ci_pipeline_id 149775025 149775025
cpu_model Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion 11.0.31 11.0.31
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost *.localhost
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-f7zlnnxs 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-f7zlnnxs 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu 11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume unsure
[-12367.930op/s; -196.055op/s] or [-4.996%; -0.079%]
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: kafka Kafka instrumentation tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant