Skip to content

feat(tracing): add native OTLP export marker to trace chunks - #378

Open
mhlidd wants to merge 3 commits into
mainfrom
matthew.li/otlp-adoption-markers
Open

mhlidd wants to merge 3 commits into
mainfrom
matthew.li/otlp-adoption-markers

Conversation

@mhlidd

@mhlidd mhlidd commented Sep 29, 2026 •

Copy link
Copy Markdown

Description

As of today, we have no way of measuring OTLP export adoption throughout our tracers. As a result, we want the tracers to emit an _dd.sdk.otlp_export tag as a source of truth for the export mode that is used. This tag will be used in the backend with the ingested spans metric to determine the overall adoption of OTLP export. Despite dd-trace-cpp not supporting OTLP Export today, we still want an explicit tag to be set in the tracer, so only the value of false is relevant here.

Motivation

Additional Notes

Tests:

  • New section in test/test_trace_segment.cpp ("TraceSegment finalization of spans"). It builds a root with one child and checks there's one chunk, the marker is "false" on chunk[0], and it's absent on chunk[1].
  • Ran locally with cmake -B build -DDD_TRACE_BUILD_TESTING=1 . && cmake --build build -j && ./build/test/tests: 134 test cases, 199,859 assertions, all passing.
  • Formatting: bin/format wasn't run because clang-format-14 isn't installed. Homebrew clang-format 20.1.6 (--dry-run --Werror) reports the changed source files clean.

Benchmarks: check-big-regressions flags BM_TraceID_ParseHex (+9–12%) plus smaller BM_Hex* moves in both runs. Those benchmarks don't run the changed code; BM_TraceTinyCCSource, which does, is ~4% faster. The same benchmarks moved 5–20% on unrelated PRs (#366, #374), which points to binary layout or heap-state effects rather than this change.

Related changes:

This change was written with AI assistance and reviewed before submission.

Jira ticket: N/A

🤖 Generated with Claude Code

Set _dd.sdk.otlp_export=false on the local root span of each trace
chunk sent to the agent, so the backend can tell native export apart
from OTLP export. dd-trace-cpp has no OTLP trace export, so the marker
is unconditional.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mhlidd mhlidd added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Sep 29, 2026
@pr-commenter

pr-commenter Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-10-01 18:40:53

Comparing candidate commit 95a0890 in PR branch matthew.li/otlp-adoption-markers with baseline commit 7059aed in branch main.

Found 1 performance improvements and 5 performance regressions! Performance is the same for 2 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:BM_HexPadded_uint64/NoPadding

  • 🟥 execution_time [+1.018µs; +1.155µs] or [+2.402%; +2.724%]

scenario:BM_HexPadded_uint64/WorstCasePadding

  • 🟥 execution_time [+1.310µs; +1.359µs] or [+2.845%; +2.952%]

scenario:BM_Hex_uint64

  • 🟥 execution_time [+907.068ns; +933.998ns] or [+3.887%; +4.002%]

scenario:BM_TraceID_ParseHex/128bit

  • 🟥 execution_time [+14.175µs; +14.327µs] or [+11.555%; +11.679%]

scenario:BM_TraceID_ParseHex/64bit

  • 🟥 execution_time [+5.856µs; +5.875µs] or [+8.410%; +8.437%]

scenario:BM_TraceTinyCCSource

  • 🟩 execution_time [-3.760ms; -3.224ms] or [-4.481%; -3.843%]

@datadog-official

datadog-official Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Pipelines

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-cpp | check-big-regressions — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

Development | build-windows-cmake (arm64)

View more details · View in GitHub Actions

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 95a0890 | Docs | View more details | Give us feedback!

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mhlidd
mhlidd marked this pull request as ready for review September 30, 2026 15:46
@mhlidd
mhlidd requested review from a team as code owners September 30, 2026 15:46
@mhlidd
mhlidd requested review from zacharycmontoya and a balanced review from Copilot and removed request for a team September 30, 2026 15:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation matches the stated wire-format requirement and has focused test coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds the native OTLP export marker to each trace chunk’s first span.

Changes:

  • Defines the _dd.sdk.otlp_export internal tag.
  • Sets the marker to "false" during trace-segment finalization.
  • Tests that only the first span carries the marker.
File Description
src/​datadog/​tags.h Declares the marker constant.
src/​datadog/​tags.cpp Defines the marker key.
src/​datadog/​trace_segment.cpp Adds the marker to the local root span.
test/​test_trace_segment.cpp Verifies marker placement and value.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@xlamorlette-datadog

Copy link
Copy Markdown
Collaborator

@mhlidd
Could you please replace the automatically generated PR description with a clear explanation of the change and its motivation, notably so that, as a non feature expert, I can review it more easily and later explain in very simple terms what it brings to users.
I'm sorry, I plan to finalize clearer contribution instructions for this repository soon, I have been short on time…

@zacharycmontoya zacharycmontoya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mhlidd

mhlidd commented Oct 1, 2026

Copy link
Copy Markdown
Author

@zacharycmontoya @xlamorlette-datadog The check-big-regressions CI job is consistently failing but I don't see how my PR changes any code that would affect the failing benchmarks. Do you have any idea as to why this is failing? What would be the best solution here? 🤔

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants