feat(runtime): refresh MicroVM identity - #19939
litianningdatadog wants to merge 1 commit into
Conversation
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🔗 Commit SHA: 25e8bb8 | Docs | View more details | Give us feedback! |
Dependency direction analysis📈 Existing violations got worse4 pre-existing violation(s) increased in severity (e.g. their target became more depended-on, or got pulled into an import cycle), though the edge itself isn't new:
|
Circular import analysis
|
Codeowners resolved asResolved from the full PR diff against |
BenchmarksBenchmark execution time: 2026-10-02 01:55:23 Comparing candidate commit 25e8bb8 in PR branch Found 0 performance improvements and 6 performance regressions! Performance is the same for 561 metrics, 10 unstable metrics, 7 known flaky benchmarks, 17 flaky benchmarks without significant changes.
|
There was a problem hiding this comment.
Pull request overview
This PR wires AWS Lambda MicroVM “/run” lifecycle detection into ddtrace startup so the runtime identity can be refreshed after import-time when a MicroVM reuses the same Python process for a new logical runtime.
Changes:
- Adds a MicroVM
/runrequest listener (web.request.starting) that refreshes runtime identity once per process, with a forksafe/threadsafe guard. - Centralizes MicroVM
/runmethod/path constants and reuses them from the contrib web dispatch helper. - Adds subprocess + integration test coverage to ensure refresh happens before root span creation and is thread-safe.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
ddtrace/internal/runtime/__init__.py |
Adds MicroVM /run hook constants and request-event listener that refreshes identity once per process. |
ddtrace/__init__.py |
Registers the MicroVM identity-refresh hook during ddtrace import. |
ddtrace/contrib/internal/web.py |
Uses the shared MicroVM /run constants when dispatching web.request.starting. |
ddtrace/contrib/_events/web_framework.py |
Minor formatting-only change. |
tests/tracer/runtime/test_runtime_id.py |
Adds focused subprocess tests for hook registration behavior, matching logic, ordering vs. root span, thread-safety, and fork reset. |
tests/contrib/asgi/test_asgi.py |
Adds ASGI integration coverage that /run refreshes identity exactly once. |
tests/contrib/wsgi/test_wsgi.py |
Adds WSGI integration coverage that /run refreshes identity exactly once. |
releasenotes/notes/aws-lambda-microvm-identity-refresh-3a672cd6bcbad16d.yaml |
Adds a release note entry for the MicroVM identity refresh feature. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
2593fa8 to
4aebb33
Compare
73d9d64 to
53a921c
Compare
4aebb33 to
206cd22
Compare
53a921c to
7b4a217
Compare
206cd22 to
f3ea773
Compare
2c58627 to
99e2b4e
Compare
f3ea773 to
68bf61e
Compare
676adf4 to
18c03c4
Compare
8e26c22 to
cdc53d4
Compare
6e5799f to
75def6a
Compare
0d2b84e to
988ed14
Compare
bfe3d91 to
a44c5ce
Compare
988ed14 to
81e11ca
Compare
a44c5ce to
7398b18
Compare
01e37a4 to
3bb8821
Compare
Stacked PRs: - 👉 This #19778 — runtime identity foundation - #19898 — WSGI/ASGI request-start hook - #19939 — central MicroVM /run identity refresh - #19820 — tracer/native writer exporter refresh - #19821 — telemetry worker refresh - #19822 — runtime metrics runtime-id tags ## Description Adds `ddtrace.internal.runtime.refresh_identity()` for runtimes that need a fresh runtime ID without going through an OS fork. This introduces `on_runtime_identity_refresh()`, a dedicated callback registry for consumers that should react only to explicit identity refreshes. Existing `on_runtime_id_change()` behavior remains available for fork-related and general runtime-ID change handling. `refresh_identity()` rotates the runtime ID without recording parent or ancestor lineage because a resumed MicroVM run is not a real fork. Existing fork behavior and lineage tracking remain unchanged. Refresh callbacks are isolated by default so one failed consumer does not block others. Callers coordinating a refresh transaction can opt into error propagation with `raise_on_error=True` and retry the same identity refresh when a rebuild fails. ## Reference - [RFC](https://docs.google.com/document/d/17lde4Zak2YRBuDvf32KFanXj9TdpOv24GYaEY8swA1w/edit?tab=t.0#heading=h.wwu37decnk8j) ## Testing Added coverage for: - runtime ID rotation through `refresh_identity()` - preservation of parent and ancestor fork lineage - explicit refresh subscriber notification - refresh subscribers not being invoked by fork handling - callback failure isolation by default - optional refresh callback failure propagation - successful callbacks under `raise_on_error=True` Validation: - `scripts/run-tests -s --venv 190fcc7 -- -q tests/tracer/runtime/test_runtime_id.py` — 21 passed - `scripts/lint checks` — passed ## Risks Low. The new path is opt-in and is not wired to traffic in this PR. Existing fork behavior is preserved. Co-authored-by: tianning.li <tianning.li@datadoghq.com>
60390da to
3b930a9
Compare
emmettbutler
left a comment
emmettbutler
left a comment
There was a problem hiding this comment.
Deferring review because the base branch is not main
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 875cd07953
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ec6286041
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e08be7d996
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cddc13b4a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09ba3d964b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1063bc02b4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 25e8bb8778
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if first_error is None: | ||
| first_error = e | ||
| else: | ||
| log.exception("Runtime ID callback %r failed and stays pending", cb) |
There was a problem hiding this comment.
When a refresh subscriber fails in production, this branch saves the first exception without logging it and later re-raises it; however, the /run publisher calls core.dispatch() without allow_raise, and the native event hub silently suppresses ordinary exceptions when the default config._raise=False. With a single failing subscriber, the transition therefore remains incomplete and retries the callback on every subsequent observation without emitting any diagnostic, leaving partially refreshed components difficult to detect or troubleshoot. Log the first failure before propagating it to the isolating dispatcher.
Useful? React with 👍 / 👎.
Stacked PRs:
Description
This PR owns the MicroVM policy for the generic request-start event published
by PR #19898. At ddtrace initialization it registers
maybe_refresh_identityonly whenin_aws_lambda_microvm()is true. Thelistener also guards direct registration outside a MicroVM.
maybe_refresh_identitymatches onlyPOST /aws/lambda-microvms/runtime/v1/run,then refreshes the runtime identity once per transition. The refresh lock,
pending-callback state, and retry behavior remain in the runtime-ID
implementation.
When a refresh callback fails, the transition still runs the remaining pending
callbacks and then re-raises the first error. Failed callbacks stay pending, so
the next observation retries only those against the same refreshed identity.
Previously the first failure stopped the loop, so a callback that failed on every
retry also blocked every callback queued behind it. This applies only to the
MicroVM pending queue; direct
refresh_identity()callers and the non-raisingpath keep their existing behavior.
The generic publisher dispatches without
allow_raise, so refresh callbackfailures do not fail the lifecycle request. The refresh state still leaves failed
callbacks pending, allowing the next observation to retry against the same
refreshed identity.
This completes the MicroVM side of review comment r4146791727.
Reference
Testing
refresh_identity(raise_on_error=True)still stops at the first failuretests/tracer/runtime/test_runtime_id.py: 39 passed (Python 3.12); the first two new tests failwithout the change;
detect_global_lockspassedgit diff --checkRisks
Runtime identity refresh changes only for the MicroVM
/runlifecycle request.No listener is registered in non-MicroVM processes.
The error-deferral change is keyed on the MicroVM pending queue, which only
maybe_refresh_identitysupplies, and that function returns immediately outside aMicroVM.
🤖 Generated with Claude Code