Fix repeated web server error log failures by cgcote · Pull Request #4254 · DataDog/dd-trace-php

cgcote · 2026-09-30T17:29:14Z

Summary

consume newly written web-server error log entries after each check
reset the read offset when the log is truncated or rotated
add regression coverage for one-time consumption and truncation

Root cause

WebServer::checkErrors() always read from the offset captured when the server started, but never advanced that offset. A single transient error was therefore reported by every retry and every subsequent test sharing the server. In Symfony 6.2, this amplified one error into synchronized failures across the four automated login-event tests.

CI Visibility evidence

Aggregated default-branch data for the previous 14 days showed 416 Symfony 6.2 known-flaky failures across 8 commits:

testLoggedInCalls: 104 failures
testUserLoginFailureEvent: 104 failures
testUserLoginSuccessEvent: 104 failures
testUserSignUp: 104 failures

Each affected commit had the same failure count for all four tests. The failures spanned every PHP 8.3 web-server SAPI (148 apache2handler, 128 cgi-fcgi, and 140 cli-server) and shared the Got error from webserver: signature at WebFrameworkTestCase.php:74. This synchronized shape supports a shared harness-state problem rather than four independent Symfony assertions.

The same signature also appeared at much lower volume in Swoole (68 events) and FrankenPHP (6 events), confirming that the underlying behavior is generic to the web-server harness and is simply most visible in Symfony.

CI Visibility for this PR reports all instrumented tests passing, no new flaky tests, 100% patch coverage, and no code-quality or security findings.

Validation

Red before / green after

The same deterministic PHP 8.3 check was run against the committed pre-fix implementation and this change:

RED — pre-fix
first='[ddtrace] [error] transient error'
second='[ddtrace] [error] transient error'
exit=1

GREEN — fixed
first='[ddtrace] [error] transient error'
second=NULL
exit=0

Tests

focused unit test: 2 tests, 6 assertions
Symfony 6.2: 30/30 suite runs passed
10 runs each with cli-server, cgi-fcgi, and apache2handler
120/120 executions of the four historically flaky tests passed (570 assertions)
PHP syntax checks and git diff --check passed

datadog-datadog-prod-us1 · 2026-09-30T17:29:26Z

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 5 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | test_integrations_frankenphp: [8.2] — ❌ 1 test failed

View more details · View in GitLab
❌ DDTrace\Tests\Integrations\Frankenphp\CommonScenariosTest::testScenario from frankenphp-test.DDTrace\Tests\Integrations\Frankenphp\CommonScenariosTest
DDTrace\Tests\Integrations\Frankenphp\CommonScenariosTest::testScenario
Exception: install-frankenphp.sh failed (exit 1):
go: downloading github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761
go: downloading github.com/x448/float16 v0.8.4
go: downloading golang.org/x/exp v0.0.0-20260529124908-c761662dc8c9
go: downloading filippo.io/bigmod v0.1.0
go: downloading golang.org/x/oauth2 v0.36.0
go: downloading github.com/go-jose/go-jose/v4 v4.1.4
go: downloading github.com/google/certificate-transparency-go v1.3.3
go: downloading github.com/urfave/cli v1.22.17
...
DataDog/apm-reliability/dd-trace-php | ASAN test_c with multiple observers: [8.5] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-php | randomized tests: [amd64, no-asan, 3]

View more details · View in GitLab
View all 5 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 68.38% (+0.00%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a0a383c | Docs | View more details | Give us feedback!}

chatgpt-codex-connector · 2026-09-30T17:29:27Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-02T07:49:25.473402Z	`5df4f42`	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-02T07:49:22.495140Z	`5df4f42`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5df4f429e0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-30T17:35:06Z

+        $currentSize = (int) @filesize($errorLog);
+        if ($currentSize < $this->errorLogSize) {


Detect log replacement independently of file size

When the log is truncated or rotated and the replacement grows to at least the previous offset before the next check, this size-only condition does not reset the cursor. file_get_contents() then starts partway through the new file, so an error marker near its beginning can be skipped and the associated test can incorrectly pass. Track the file identity or otherwise detect replacement/truncation rather than relying solely on currentSize < errorLogSize.

Useful? React with 👍 / 👎.

@cgcote This is a proper issue, and a mere size check isn't enough for that.
A retry might abort earlier, the successful run run for longer.

@bwoebi addressed

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5df4f429e0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cgcote · 2026-10-02T08:02:03Z

/gitlab retry-pipeline

gh-worker-devflow-routing-ef8351 · 2026-10-02T08:02:06Z

View all feedbacks in Devflow UI.

2026-10-02 08:02:05 UTC ℹ️ Start processing command /gitlab retry-pipeline

2026-10-02 08:03:34 UTC ℹ️ Devflow: /gitlab retry-pipeline

All failed jobs have been retried successfully. The following job IDs have been retried: 2102078658, 2102078822, 2102079357, 2102079579

cgcote · 2026-10-02T08:03:33Z

/gitlab retry-pipeline

gh-worker-devflow-routing-ef8351 · 2026-10-02T08:03:36Z

View all feedbacks in Devflow UI.

2026-10-02 08:03:36 UTC ℹ️ Start processing command /gitlab retry-pipeline

2026-10-02 08:04:50 UTC ℹ️ Devflow: /gitlab retry-pipeline

All failed jobs have been retried successfully. The following job IDs have been retried:

A size-only comparison misses a rotated/replaced error log whose replacement has already grown to at least the previous offset, causing checkErrors() to skip entries at the beginning of the new file. Track the log's inode in addition to its size and reset the cursor whenever either changes.

Fix repeated web server error log failures

5df4f42

cgcote requested review from a team as code owners September 30, 2026 17:29

chatgpt-codex-connector Bot reviewed Sep 30, 2026

View reviewed changes

cgcote marked this pull request as draft September 30, 2026 17:49

cgcote marked this pull request as ready for review October 2, 2026 07:45

chatgpt-codex-connector Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread tests/WebServer.php

cgcote and others added 3 commits October 2, 2026 09:23

Merge branch 'master' into fix/consume-webserver-error-log

7c8e961

Merge branch 'master' into fix/consume-webserver-error-log

a0a383c

Navigation Menu

Fix repeated web server error log failures - #4254

Fix repeated web server error log failures#4254
cgcote wants to merge 4 commits into
masterfrom
fix/consume-webserver-error-log

cgcote commented Sep 30, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Sep 30, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Sep 30, 2026

Uh oh!

bwoebi Oct 2, 2026

Uh oh!

cgcote Oct 2, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

cgcote commented Oct 2, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

cgcote commented Oct 2, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		$currentSize = (int) @filesize($errorLog);
		if ($currentSize < $this->errorLogSize) {

Conversation

cgcote commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

CI Visibility evidence

Validation

Red before / green after

Tests

Uh oh!

datadog-datadog-prod-us1 Bot commented Sep 30, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

bwoebi Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

cgcote Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

cgcote commented Oct 2, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgcote commented Oct 2, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cgcote commented Sep 30, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Sep 30, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

chatgpt-codex-connector Bot commented Sep 30, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 2, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 2, 2026 •

edited

Loading