Production Build: 2.1.0-beta.3 - #321
Open
github-actions[bot] wants to merge 16 commits into
Open
github-actions[bot] wants to merge 16 commits into
github-actions[bot] wants to merge 16 commits into
Conversation
… flag Adds a logs-first demo scenario where orders complete successfully at the wrong price, so APM/RUM golden signals stay green and the revenue loss is only discoverable in the checkout logs. - New flagd flag promoDiscountBug sets the fraction of orders that match the leftover internal test promo code PROMOTEST100 (100% off). Orders that don't match get the live SPRING10 campaign discount (10% off). - checkout applies the discount only to the amount sent to payment. The order items, shipping cost and the OrderResult returned to the customer all keep the full price, so the confirmation page, confirmation email and accounting DB are unaffected. - The 'order placed' log record gains app.promo.code, app.order.discount.pct and app.order.amount.charged alongside the existing app.order.amount, which stays the pre-discount total. Deliberately not added to span attributes: keeping the signal log-only is what makes this a logs-first scenario.
The kubelet probes payment's gRPC health service on every liveness (10s) and readiness (5s) tick, and auto-instrumentation turns each probe into a SERVER span. Splunk APM counts those as service requests: ~18/min/pod, so ~360 per 5 minutes across four pods against ~15 real Charge calls. The paymentFailure 50/50 A/B split therefore read as a ~2% error rate instead of ~50%. Add a sampler that drops grpc.health.v1.Health spans and delegates everything else to the default parent-based sampler. This overlaps the collector-side filter added later in this branch; keeping both means payment stops exporting the spans at the source rather than paying to ship them and drop them centrally. Separately, make the manual `charge` span INTERNAL instead of SERVER and mirror its attributes onto the auto-instrumented entry span. This does NOT change the request rate: Splunk APM counts service entry spans only, and the manual span has a same-service parent, so it never entered the count. It is span-volume hygiene -- one less exported span per charge, attribute promotion preserved. Requires @opentelemetry/sdk-trace-base as a direct dependency, pinned to the 2.9.0 already resolved for @splunk/otel so no second copy is installed. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
Register the payment hotfix in .hotfix.yaml via manage-hotfix.py so stitch-manifests.sh resolves the tag, and point the vA/vB manifests (plus the legacy payment-k8s.yaml fallback) at ghcr.io/splunk/opentelemetry-demo/otel-payment:2.0.8-hotfix-payment-1. Carries the health-probe sampler and INTERNAL charge span fix. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
Every gRPC service in the demo carries a livenessProbe (10s) and a readinessProbe (5s) against grpc.health.v1.Health/Check, and the language auto-instrumentation turns each probe into a SERVER span. APM counts those as service requests: ~18/min/pod of pure noise. On payment (4 pods) that read as ~390 requests per 5 minutes against ~15 real Charge calls, so the paymentFailure 50/50 A/B split showed up as a ~2% error rate instead of ~50%. The same distortion applies to cart, checkout, product-catalog, ad, recommendation and currency. Add filter/drop_health_checks to the agent traces pipeline, matching the gRPC health service on both the RPC attributes and the span name (instrumentations differ on which they populate), plus anchored HTTP liveness/readiness paths. error_mode: ignore so IsMatch against an absent attribute is a no-op. Validated with `otelcol validate` against splunk-otel-collector 0.158.0. This is the fleet-wide fix; the payment-side sampler in this branch covers the same probes in-process for that one service. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
Splunk APM renders the endpoint as grpc.grpc.health.v1.Health/Check, so the raw span name carries a prefix in at least some instrumentations. Match the substring instead of anchoring at the start; the rpc.service condition already covers the attribute-based case. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
Copy of the 2.0.8 values, which now carries filter/drop_health_checks, so the health-probe fix is inherited by the next release rather than depending on the "Ensure values.yaml exists" clone step in prod-release.yml. Only the `# Version:` line and the Deploy example's -f filename are changed; the config itself is identical to 2.0.8. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
Per-hotfix copies of the values file are redundant: prod-release.yml clones one when a version has none, and the per-version files that matter (2.0.8, 2.0.9) are carried explicitly. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
The message thrown after all retries are exhausted named the rejected credential: Payment failed after retries: Invalid API Token (test-20e26e90-...) That string crosses the gRPC boundary, so it surfaced on every failed trace as grpc.error_message and otel.status_description in checkout's oteldemo.PaymentService/Charge span, and as exception.message and exception.stacktrace on the exception events in both checkout and payment. Use 'Invalid request' instead, matching InvalidRequestError, which is what the simulated gateway already raises per attempt for a 401. The token is still logged in the two payment ERROR records that accompany the failure; those are log records rather than span attributes, and they are the breadcrumb the demo expects an attendee to follow to root cause. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
…iption
The charge span's status description was the bare status code ("401"), so it
read differently from the auto-instrumented entry span, whose
otel.status_description comes from the thrown error's message.
Mirror the thrown message instead. Both spans now show 'Invalid request' for a
401, and the status code stays available as http.status_code and
rpc.grpc.status_code.
Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
This reverts commit 267b4ea. The pin and its .hotfix.yaml entry were for shipping the payment fixes as a hotfix on top of 2.0.8. This branch releases 2.0.9 instead, where the pipeline sets every image to the release version and clears hotfix tracking, so both are stale here: the manifests would ship a 2.0.8 hotfix tag, and manage-hotfix.py would still report a 2.0.8 hotfix against a 2.0.9 base. The payment code fixes themselves are untouched. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
Brings in the promoDiscountBug silent revenue leak scenario (checkout + flagd) alongside the payment health-probe, span-noise and error-message fixes. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
…tics payment-vB ran with OTEL_LOG_LEVEL=debug, which enables the OpenTelemetry JS diag logger. That pretty-prints every exported span to stdout as a multi-line object dump, plus instrumentation patch traces and a machine-id ENOENT on every start. The filelog receiver ingests one record per line, and none of those lines carry a parseable severity, so Splunk showed hundreds of UNKNOWN entries per minute whose _raw was a fragment like "traceId: '89a4cae...'". On a sample of the last 200 lines from a vB pod, 200 were SDK noise and 0 were application logs. Drop OTEL_LOG_LEVEL to info, matching payment-vA. LOG_LEVEL stays at debug so vB remains the more verbose variant where it is useful: pino emits single-line JSON with severity and trace context, which parses correctly. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
…own"
Every payment log record carried deployment.environment: "unknown". The mixin's
fallback chain missed on all three steps:
- OTEL_RESOURCE_ATTRIBUTES on these pods is
"service.name=payment,service.version=...,payment.variant=B" with no
deployment.environment key, so the regex never matched. In k8s the
environment is injected downstream by the collector's
resource/add_environment processor, not passed to the app.
- ENVIRONMENT is not set; the pods get WORKSHOP_ENV, from workshop-secret
key "env".
- So it fell through to the literal string "unknown".
The collector cannot repair this: it adds deployment.environment as a resource
attribute with action: insert, which does not touch a field the app already
wrote into the log body.
Add WORKSHOP_ENV to the chain, and accept deployment.environment.name as well
as deployment.environment so the regex matches current semconv.
Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
…ot end on the error
For the demo the payment ERROR was the last log line in the trace, so the root
cause was the first thing a viewer saw before investigating anything.
checkout returned from a failed chargeCard straight to the caller with no
logging at all, and payment stopped at its ERROR records. Add breadcrumbs on
the unwind path:
payment Charge attempt sequence complete. (charge.js finally)
Charge request rejected, returning to caller.
Leaving payment service. (index.js catch, after callback)
checkout payment declined, unwinding order
no transaction id returned by payment service
cart left intact for retry
shipping order not placed
order confirmation email not sent
order not published to orders topic
order total not recorded for reporting
leaving PlaceOrder
Ten INFO records after the failure. Each states something that is actually true
about the abandoned order rather than inventing rollback work the service does
not do, so the trail stays honest while the failure sits mid-stream.
checkout uses InfoContext so the records carry trace context and correlate to
the same trace; several existing logger.Info calls in this file use the
context-free form and do not correlate.
Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
…rder path Builds on the unwind logging so the payment errors sit in the middle of a long log stream rather than at the top of the page, making the search exercise in the demo worth doing. checkout now brackets every downstream call, so a failed order emits roughly a dozen INFO records before the error as well: entering/leaving cart service (with item count) entering/leaving product catalog service entering/leaving shipping service entering/leaving currency service entering/leaving order preparation order total calculated / promo code resolved entering/leaving payment service (leaving carries result=declined) and the unwind gained one line per order item plus the promo, order id and return-code records. payment gains "Entering payment service." on the way in and, on the way out, "Charge not authorized, no funds captured." and "Payment gateway session closed." The success path gets a matching leaving pair so the brackets are symmetric rather than only appearing on failures. Every line still states something true about the order; none of them invent rollback work the services do not perform. Claude-Session: https://claude.ai/code/session_019CsicxrWPhBhfFNmhdkTaM
Version bump: custom Services: checkout Updated: - SPLUNK-VERSION - Source k8s manifests with new image references
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production Build 2.1.0-beta.3
Version Bump: custom
Services Built: checkout
Changes
2.1.0-beta.3Note
This is a partial build. Other services will use their existing image versions.
Next Steps
Run the Build Demo Manifest - PRODUCTION workflow to stitch the manifest.
🤖 Generated automatically by GitHub Actions