demos: add actor telemetry continuity reference demo (#503) - #750
Open
sanskar singh bhardwaj (sanskar-singh-2403) wants to merge 1 commit into
Open
Conversation
…#503) Add a reference demo for actor telemetry continuity across suspend/resume, the follow-up work requested in agent-substrate#503. The demo is a small Go HTTP server that emits OpenTelemetry metrics and trace spans over OTLP on every request. It reproduces the agent-substrate#503 loss: an OTel push exporter batches telemetry in memory and flushes on a timer, so when the push interval (default 60s) is longer than the active window before idle suspension, telemetry is still buffered in the SDK when the actor is suspended. With snapshotsConfig.onPause: Data (the mode the template uses) Substrate drops process RAM on pause, so that buffered telemetry is lost. The mitigation is isolated behind a flushTelemetry(ctx) seam that ForceFlushes both the MeterProvider and TracerProvider. Today it is called from the self-suspend path via an idle watcher; when the PreSuspend hook (agent-substrate#450) lands, flushTelemetry does not change, only its call-site moves into the PreSuspend handler. This keeps agent-substrate#450 a call-site move rather than a rewrite. The ActorTemplate sets OTEL_EXPORTER_OTLP_ENDPOINT to the in-cluster collector, since the SDK default (localhost:4317) has nothing listening inside the actor sandbox netns. Includes install-demo-actor-telemetry.sh wired into install-ate.sh, a rendered-and-validated template, unit tests for the idle-flush watcher, and a README documenting the reproduce/verify flow against Jaeger and Prometheus on Kind.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a reference demo for actor telemetry continuity across suspend/resume, the
follow-up work requested in #503.
The demo is a small Go HTTP server that emits OpenTelemetry metrics and trace
spans over OTLP on every request. It reproduces the #503 loss: an OTel push
exporter batches telemetry in memory and flushes on a timer, so when the push
interval (default 60s) is longer than the active window before idle suspension,
telemetry is still buffered in the SDK when the actor is suspended. With
snapshotsConfig.onPause: Data(the mode the template uses) Substrate dropsprocess RAM on pause, so that buffered telemetry is lost.
The mitigation is isolated behind a
flushTelemetry(ctx)seam that ForceFlushesboth the MeterProvider and TracerProvider. Today it is called from the
self-suspend path via an idle watcher; when the PreSuspend hook (#450) lands,
flushTelemetrydoes not change, only its call-site moves into the PreSuspendhandler. This keeps #450 a call-site move rather than a rewrite.
The ActorTemplate sets
OTEL_EXPORTER_OTLP_ENDPOINTto the in-clustercollector, since the SDK default (
localhost:4317) has nothing listening insidethe actor sandbox netns.
Includes
install-demo-actor-telemetry.shwired intoinstall-ate.sh, arendered-and-validated template, unit tests for the idle-flush watcher, and a
README documenting the reproduce/verify flow against Jaeger and Prometheus on
Kind.
Scope
This covers the "reference demo for actor telemetry continuity" action item in
#503 only. The
docs/observability.mdupdate is tracked separately, and thePreSuspend/PostResumealignment is #450.Part of #503
Testing
gofmt,go vet,go build, andgo test ./demos/actor-telemetry/all pass(4/4 unit tests on the idle-flush watcher).
Template renders via the install script's
sedsubstitution and parses asvalid Kubernetes manifests.
End-to-end suspend/resume verification against a live Kind cluster with
ate-system is described in the README.
Tests pass
Appropriate changes to documentation are included in the PR