chore(runtime): add explicit identity refresh

[litianningdatadog]

Stacked PRs:

• 👉 This chore(runtime): add explicit identity refresh #19778 — runtime identity foundation
  • chore(serverless): publish web request-start event #19898 — WSGI/ASGI request-start hook
    • feat(runtime): refresh MicroVM identity #19939 — central MicroVM /run identity refresh
      • fix(tracer): isolate MicroVM identity refresh #19820 — tracer/native writer exporter refresh
        • fix(telemetry): rebuild worker on identity refresh #19821 — telemetry worker refresh
          • chore(runtime): refresh runtime metric identity and collector state #19822 — runtime metrics runtime-id tags

Description

Adds ddtrace.internal.runtime.refresh_identity() for runtimes that need a fresh runtime ID without going through an OS fork.

This introduces on_runtime_identity_refresh(), a dedicated callback registry for consumers that should react only to explicit identity refreshes. Existing on_runtime_id_change() behavior remains available for fork-related and general runtime-ID change handling.

refresh_identity() rotates the runtime ID without recording parent or ancestor lineage because a resumed MicroVM run is not a real fork. Existing fork behavior and lineage tracking remain unchanged.

Refresh callbacks are isolated by default so one failed consumer does not block others. Callers coordinating a refresh transaction can opt into error propagation with raise_on_error=True and retry the same identity refresh when a rebuild fails.

Reference

• RFC

Testing

Added coverage for:

• runtime ID rotation through refresh_identity()
• preservation of parent and ancestor fork lineage
• explicit refresh subscriber notification
• refresh subscribers not being invoked by fork handling
• callback failure isolation by default
• optional refresh callback failure propagation
• successful callbacks under raise_on_error=True

Validation:

• scripts/run-tests -s --venv 190fcc7 -- -q tests/tracer/runtime/test_runtime_id.py — 21 passed
• scripts/lint checks — passed

Risks

Low. The new path is opt-in and is not wired to traffic in this PR. Existing fork behavior is preserved.

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

No remaining files require a CODEOWNERS review.

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 201 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 201 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=132)
ddtrace.llmobs._integrations.vertexai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=130)
ddtrace.appsec._listeners -×-> ddtrace.trace  (product:appsec -> product:tracing, score=130)
ddtrace.internal.ci_visibility.git_client -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=130)
ddtrace.llmobs._integrations.langchain -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=130)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[datadog-datadog-prod-us1-2]

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a444344 | Docs | View more details | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-09-25 22:05:56

Comparing candidate commit a444344 in PR branch tianning.li/1-runtime-identity-refresh with baseline commit fb06160 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 6 performance regressions! Performance is the same for 350 metrics, 9 unstable metrics, 4 known flaky benchmarks, 4 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationextract-b3_headers

• 🟥 execution_time [+1.070µs; +1.154µs] or [+14.353%; +15.480%]

scenario:httppropagationextract-empty_headers

• 🟥 execution_time [+84.856ns; +101.951ns] or [+11.626%; +13.968%]

scenario:msgpackencoderscenario-simple_one_span

• 🟥 execution_time [+517.792ns; +571.798ns] or [+12.697%; +14.021%]

scenario:otelspan-start

• 🟥 execution_time [+1.735ms; +2.563ms] or [+7.028%; +10.385%]

scenario:recursivecomputation-shallow

• 🟥 execution_time [+50.106µs; +53.501µs] or [+7.145%; +7.629%]

scenario:span-start-finish-telemetry

• 🟥 execution_time [+4.478ms; +4.651ms] or [+10.950%; +11.373%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-741.612ns; +746.205ns] or [-7.129%; +7.173%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-33.805ns; +45.234ns] or [-5.107%; +6.833%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-1962.009ns; +1858.256ns] or [-9.873%; +9.351%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-2061.291ns; +1615.072ns] or [-10.731%; +8.408%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-386.624ns; +376.310ns] or [-9.093%; +8.851%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-236.685ns; +223.504ns] or [-8.846%; +8.354%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-107.330ns; +78.034ns] or [-7.871%; +5.723%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-4481.895ns; +4823.075ns] or [-9.336%; +10.047%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-915.932ns; +998.212ns] or [-9.043%; +9.856%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.947µs; +3.076µs] or [+20.912%; +21.832%]

scenario:span-start

• 🟥 execution_time [+1.357ms; +1.783ms] or [+10.870%; +14.287%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+240.194ns; +273.967ns] or [+12.605%; +14.378%]

scenario:tracer-small

• 🟥 execution_time [+44.248µs; +45.652µs] or [+18.400%; +18.984%]

Known flaky benchmarks without significant changes:

• scenario:errortrackingflasksqli-baseline
• scenario:flasksimple-iast-get
• scenario:sethttpmeta-all-enabled
• scenario:telemetryaddmetric-record-100-metrics

[Copilot]

Pull request overview

Adds an explicit, non-fork “process identity refresh” pathway for runtimes that can resume from snapshots without an OS-level fork, along with a weakly-referenced subscriber mechanism so long-lived components can rebuild identity-bound state without leaking instances.

Changes:

• Introduces ddtrace.internal.runtime.refresh_identity() to rotate the runtime id without recording fork lineage and to notify registered subscribers.
• Refactors on_runtime_id_change() to store subscribers via weak references and isolates subscriber exceptions during notification.
• Expands runtime-id test coverage for refresh semantics, lineage invariants, subscriber notification, exception isolation, and weakref cleanup.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
ddtrace/internal/runtime/__init__.py	Adds refresh_identity(), weakref-based subscriber registry, and subscriber notification isolation; keeps fork behavior on the forksafe hook without subscriber notification.
tests/tracer/runtime/test_runtime_id.py	Adds subprocess-based tests for refresh behavior and subscriber semantics; updates imports for runtime module usage.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2fa8088e77

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[brettlangdon]

This feels very over-engineered, I don't feel like it is clear why we are moving everything to be weakrefs? it adds a ton of machinery that isn't clear the value we are getting from it.

I am also surprised that removing notifications from _set_runtime_id doesn't break a bunch of things, or how it wouldn't break a bunch of things.

[Copilot]

🟡 Changes recommended

Callback failures can leave explicit-refresh consumers with stale identity-dependent state.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

tests/tracer/runtime/test_runtime_id.py:81

• This only verifies that the fork itself does not invoke the refresh callback. It would still pass if the child lost the callback registry entirely, so it does not cover the stated guarantee that refresh subscribers remain registered after a fork. Invoke refresh_identity() in the child and assert that the inherited subscriber receives the new ID.

    child = os.fork()
    if child == 0:
        assert seen == []
        os._exit(42)

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

[litianningdatadog]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-09-24 20:11:03 UTC ℹ️ Start processing command /merge

---

2026-09-24 20:11:14 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

• Run /code blockers to see what is blocking it.
• Run /remove to cancel it.

---

2026-09-25 00:14:12 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

[litianningdatadog]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-09-25 15:26:47 UTC ℹ️ Start processing command /merge

---

2026-09-25 15:26:52 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 56m (p90).

---

2026-09-25 15:50:43 UTC ❌ MergeQueue: This merge request was updated

This PR is rejected because it was updated