Skip to content

demos: add actor telemetry continuity reference demo (#503) - #750

Open
sanskar singh bhardwaj (sanskar-singh-2403) wants to merge 1 commit into
agent-substrate:mainfrom
sanskar-singh-2403:demo/actor-telemetry-503
Open

demos: add actor telemetry continuity reference demo (#503)#750
sanskar singh bhardwaj (sanskar-singh-2403) wants to merge 1 commit into
agent-substrate:mainfrom
sanskar-singh-2403:demo/actor-telemetry-503

Conversation

@sanskar-singh-2403

Copy link
Copy Markdown

Add a reference demo for actor telemetry continuity across suspend/resume, the
follow-up work requested in #503.

The demo is a small Go HTTP server that emits OpenTelemetry metrics and trace
spans over OTLP on every request. It reproduces the #503 loss: an OTel push
exporter batches telemetry in memory and flushes on a timer, so when the push
interval (default 60s) is longer than the active window before idle suspension,
telemetry is still buffered in the SDK when the actor is suspended. With
snapshotsConfig.onPause: Data (the mode the template uses) Substrate drops
process RAM on pause, so that buffered telemetry is lost.

The mitigation is isolated behind a flushTelemetry(ctx) seam that ForceFlushes
both the MeterProvider and TracerProvider. Today it is called from the
self-suspend path via an idle watcher; when the PreSuspend hook (#450) lands,
flushTelemetry does not change, only its call-site moves into the PreSuspend
handler. This keeps #450 a call-site move rather than a rewrite.

The ActorTemplate sets OTEL_EXPORTER_OTLP_ENDPOINT to the in-cluster
collector, since the SDK default (localhost:4317) has nothing listening inside
the actor sandbox netns.

Includes install-demo-actor-telemetry.sh wired into install-ate.sh, a
rendered-and-validated template, unit tests for the idle-flush watcher, and a
README documenting the reproduce/verify flow against Jaeger and Prometheus on
Kind.

Scope

This covers the "reference demo for actor telemetry continuity" action item in
#503 only. The docs/observability.md update is tracked separately, and the
PreSuspend/PostResume alignment is #450.

Part of #503

Testing

  • gofmt, go vet, go build, and go test ./demos/actor-telemetry/ all pass
    (4/4 unit tests on the idle-flush watcher).

  • Template renders via the install script's sed substitution and parses as
    valid Kubernetes manifests.

  • End-to-end suspend/resume verification against a live Kind cluster with
    ate-system is described in the README.

  • Tests pass

  • Appropriate changes to documentation are included in the PR

…#503)

Add a reference demo for actor telemetry continuity across suspend/resume,
the follow-up work requested in agent-substrate#503.

The demo is a small Go HTTP server that emits OpenTelemetry metrics and
trace spans over OTLP on every request. It reproduces the agent-substrate#503 loss: an
OTel push exporter batches telemetry in memory and flushes on a timer, so
when the push interval (default 60s) is longer than the active window
before idle suspension, telemetry is still buffered in the SDK when the
actor is suspended. With snapshotsConfig.onPause: Data (the mode the
template uses) Substrate drops process RAM on pause, so that buffered
telemetry is lost.

The mitigation is isolated behind a flushTelemetry(ctx) seam that
ForceFlushes both the MeterProvider and TracerProvider. Today it is called
from the self-suspend path via an idle watcher; when the PreSuspend hook
(agent-substrate#450) lands, flushTelemetry does not change, only its call-site moves into
the PreSuspend handler. This keeps agent-substrate#450 a call-site move rather than a
rewrite.

The ActorTemplate sets OTEL_EXPORTER_OTLP_ENDPOINT to the in-cluster
collector, since the SDK default (localhost:4317) has nothing listening
inside the actor sandbox netns.

Includes install-demo-actor-telemetry.sh wired into install-ate.sh, a
rendered-and-validated template, unit tests for the idle-flush watcher, and
a README documenting the reproduce/verify flow against Jaeger and
Prometheus on Kind.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant