Increase p50 SLO threshold from 0.122 to 0.126 ms after post-2026-09-28 shift

[faydef]

Summary

Since 2026-09-28 13:39 UTC, normal_operation/only-tracing agg_http_req_duration_p50 has jumped from a stable ~0.11884 ms mean (48 runs over ~29 days) to ~0.12293 ms (6 consecutive matched runs on master and one feature branch), a +3.4 % / +0.00409 ms shift. The current threshold of < 0.122 ms fails on every run since the shift began; five master-branch check-slo-breaches jobs have errored with 90 % CIs reaching [123.487 µs; 123.685 µs]. This PR raises the threshold to < 0.126 ms so the SLO gate reflects the new steady state and stops blocking release branches.

high_load/only-tracing throughput is unchanged (mean 4036.9 → 4037.7 op/s, always well above the 3700 op/s SLO), so its threshold is left as-is.

Changes

• .gitlab/bp-runner.fail-on-breach.yml: raise agg_http_req_duration_p50 threshold for normal_operation/only-tracing from < 0.122 ms to < 0.126 ms.
• warning_range: 2 is preserved, so the warning band tracks the new SLO.
• No other file touched.

Sizing rationale

Post-shift 90 % CI upper bound peaked at 123.685 µs; 0.126 ms leaves ~2.3 µs of buffer, matching the ~2–3 µs buffer used in prior threshold bumps (PRs #354, #385, #429).

Testing/Validation

• YAML validated with python3 -c "import yaml; yaml.safe_load(open('.gitlab/bp-runner.fail-on-breach.yml'))".
• Inspected the final diff with git diff from the pre-session base — the only functional change is the threshold value.

Evidence

Historical daily-mean apm_sdk_slo.v1.measurement{project:nginx-datadog,metric:agg_http_req_duration_p50,scenario:normal_operation/only-tracing,statistic:mean} for the last 30 days:

| Window | Runs | Min | Max | Mean | | --- | --- | --- | --- | --- | | 2026-08-30 → 2026-09-27 | 48 | 0.11763 ms | 0.11994 ms | 0.11884 ms | | 2026-09-28 → 2026-09-29 | 6 | 0.12208 ms | 0.12359 ms | 0.12293 ms |

Failing check-slo-breaches jobs on master with reported 90 % CIs on p50:

| Job | Pipeline | Start (UTC) | 90 % CI on p50 | | --- | --- | --- | --- | | 2085294589 | 140553788 | 2026-09-28 13:39 | [123.487 µs; 123.685 µs] | | 2086161863 | 140619512 | 2026-09-28 16:06 | [123.001 µs; 123.288 µs] | | 2087791446 | 140763707 | 2026-09-28 23:33 | [122.829 µs; 123.003 µs] | | 2088668144 | 140840356 | 2026-09-29 09:21 | [122.927 µs; 123.124 µs] | | 2088790411 | 140850056 | 2026-09-29 09:53 | [122.731 µs; 122.937 µs] |

Job URLs follow the pattern https://gitlab.ddbuild.io/DataDog/apm-reliability/nginx-datadog/-/jobs/<job-id> (e.g. .../jobs/2085294589, .../jobs/2086161863, .../jobs/2087791446, .../jobs/2088668144, .../jobs/2088790411, .../jobs/2089008535).

The shift is confined to per-request latency (throughput is flat) and starts one minute after commit b7d2e16 (Upgrade C++ Tracer from v2.1.1 to v2.3.0, 2026-09-28 13:38 UTC) landed on master. Given the tracer-version alignment, the underlying cause should be investigated separately; this PR only unblocks CI at the observed steady state.

Marking as draft for review.

🤖 Generated with Claude Code

NOTE: some AI-generated links and images were removed to ensure safety.

---

PR by Bits - View session in Datadog

[datadog-official]

View session in Datadog

Bits Code status: ✅ Done

Comment @DataDog to request changes

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-30T12:07:40.546551Z	b3c7217	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-30T12:09:48.654263Z	b3c7217	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[xlamorlette-datadog]

For future similar automated PR, I would like to get a better PR description:

• For such a simple change, the summary should be only one sentence (max 20 words I guess).
• Stating what was not changed is noise.
• Given the change is a one-liner, explaining the change is useless, the summary should be enough.
• Evidence should give links to failing jobs.
• Important: Evidence must contain links/logs demonstrating that the increase was observed in the baseline.
• The table and numbers in the Evidence are not readable.
• The final 'Generated with Claude Code' and 'some AI-generated links and images were removed […]' sentences are just noise and useless ad (as 'Sent from my [telephone brand name]').