Use the Copilot PAT pool for Build Failure Analysis

[YuliiaKovalova]

Summary

Move both Build Failure Analysis entry points to the Copilot PAT-pool design proposed for update-default-versions in #17050.

• Reuse shared/pat_pool.md unchanged from Use the agentic workflows PAT pool and run in an isolated environment #17050; keep BFA's token binding in its existing shared import.
• Declare the protected copilot-pat-pool environment in both callers. Environment configuration is not inherited through a shared import.
• Bind the selected COPILOT_PAT_0..9 slot to both analysis and threat detection, with direct pat_pool job dependencies. Only the slot number crosses job outputs.
• Fail explicitly before inference if the pool supplied no valid slot number or the selected secret is empty. Do not fall back to the legacy COPILOT_GITHUB_TOKEN or the Actions token.
• Run gh-aw's existing OAuth-token validator in a protected-environment prerequisite job. Both analysis and threat detection directly depend on it and explicitly require its success. A detection pre-step alone is insufficient because the generated CLI step uses always(). This is a format check, not proof of credential validity or entitlement.
• Skip the entire detection job, including image downloads, when the agent produced neither output nor a patch. Preserve detection after an agent failure with partial output, provided token validation succeeded.
• Preserve the existing per-phase model overrides, claude-sonnet-4.6 fallback, binlog-fetch jobs, trigger declarations, concurrency, and read-only inference permissions.

This is a companion to #17050, which is still unmerged, rather than a separate pool implementation. It does not change update-default-versions or incorporate the independent download-hardening work in #17436.

Failure evidence and limits

Arcade run 35669930318 failed in Execute GitHub Copilot CLI: Copilot returned HTTP 401 for /models and /chat/completions, with AuthenticateToken authentication failed. The existing secret was present and PAT-shaped; the same runtime configuration had previously performed successful inference. The other workflow using that secret reference now fails similarly.

Expiry or revocation of the old credential has not been established. This PR implements the approved move to the managed pool; it does not establish that any pool credential is currently valid, or that the production 401 is resolved. The isolated AWF setup-download HTTP 504 is separate from this authentication change.

Generated toolchain and behavior changes

Both locks were regenerated with gh-aw v0.86.2. The checksum-verified v0.79.8 Windows compiler hung during startup, so the installed compiler was used instead. The generated diff therefore includes framework/action/container updates, including Copilot CLI 1.0.79 and AWF v0.27.44; it is not an authentication-only edit to the old YAML.

Recompile using the explicit setup SHA in the command below. gh-aw v0.86.2's release-version cache cleanup otherwise removes the updater's setup@v0.77.5 pin even during a targeted BFA compile. The supported SHA-based action selection preserves both setup versions while using the same BFA action code. The shared source documents the required flags.

The new compiler defaults Copilot to auto; this PR explicitly preserves BFA's previous model precedence for both inference phases. Its generated slash-command activation also checks a literal command prefix: post /analyze-build-failure as the unindented first line. This behavior change is documented and covered by expression checks. The binlog MCP image and read-only mount are unchanged.

Credential and action review

The BFA manifests replace the legacy inference-secret reference with COPILOT_PAT_0 through COPILOT_PAT_9. No credential values were supplied, replaced, printed, or committed. The selector exports only an index; inference credentials remain excluded from the agent container and are supplied to the AWF API-proxy sidecar. Repository/API write credentials remain separate from inference credentials.

Regeneration updates the SHA-pinned github/gh-aw-actions/setup, actions/checkout, and actions/setup-node actions, and adds SHA-pinned actions/cache/restore and actions/cache/save. These are framework-generated changes, not a new custom credential transport. Compilation's action/schema validation and actionlint pass. Pool-secret contents, validity, and entitlement still require maintainer verification.

The shared actions-lock.json includes all action refs used by all three generated workflows, including the older checkout/setup-node entries and setup@v0.77.5 used by update-default-versions. Cache saving uses the Actions runtime token; no actions: write permission was added.

Validation

• Compile both workflows with gh aw compile build-failure-analysis build-failure-analysis-command --approve --no-check-update --schedule-seed dotnet/arcade --action-mode action --action-tag 6aab9e5b5c91c615506061f09bedd81a23babe3c --validate --actionlint.
• Verify reproducible lock generation and unchanged binlog-fetch jobs, trigger declarations, concurrency, and inference permissions.
• Verify all 11 distinct action pins across all three generated workflow locks exactly match actions-lock.json; every original upstream pin is retained. Release SHAs were checked, including the restored annotated v0.77.5 tag. Repeating the documented compilation leaves both BFA locks, the untouched updater lock, and the shared action lock byte-identical.
• Exercise six synthetic pool-selection cases; verify that outputs/logs do not contain the synthetic credential values.
• Accept all ten slot numbers and explicitly reject five empty/invalid selections before inference.
• Exercise 132 cases against the generated prerequisite validator: both PAT formats accepted, OAuth tokens and empty secrets rejected in all ten slots, invalid slots rejected, and missing validator scripts fail closed. No synthetic token values appear in logs, summaries, or outputs.
• Exercise 256 combinations of prerequisite result, agent result, quota status, output presence, and patch presence against the generated consumer conditions, plus eight detection-output guard cases. Failed, cancelled, or skipped validation blocks both inference jobs despite detection's always() condition. Empty-output runs skip detection entirely; failed agents with partial output remain eligible.
• Check the generated dependencies, environment scope, credential bindings, and original model precedence in both analysis and detection.
• Exercise ten generated command-activation cases, including PR, activation, binlog, and literal-prefix gates.
• Validate/provision the protected environment's pool credentials.
• Run the changed workflow end-to-end in dotnet/arcade and confirm a delivered analysis report.

The exact PR head (7a2c999d943afc3a18572dbf7f2a70cb5f6812e7) is now published on the dedicated upstream branch users/YuliiaKovalova/bfa-pat-pool-e2e. It awaits normal branch protection, requested from the maintainers in this comment, before it can use the existing protected-branches-only PAT-pool environment. No production branch or environment restriction was changed. Actual upstream run 35994258685 fetched and retained the binlogs, but GitHub rejected pre_activation before a runner started: the test branch is not allowed to deploy to copilot-pat-pool. Token validation, inference, detection, and safe outputs were all skipped. Authentication and report delivery remain unverified.

Once the test branch is eligible, run workflow 313838244 on that ref against a genuine failed arcade-pr build whose head and merge revisions still match the PR, then confirm analysis, threat detection, safe outputs, and the resulting PR report. Revalidate the target build immediately before dispatch. A fork run, successful compilation, skipped agent, or run of the unchanged default-branch workflow does not satisfy this check. Do not relax the environment's branch protection to test a fork.

To double check:

• The right tests are in and the right validation has happened. Guidance: https://github.com/dotnet/arcade/blob/main/Documentation/Validation.md

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved token-validation and detection-scope findings affect both generated workflows.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Activation jobs cannot validate environment-scoped PAT tokens · New

What changed in this PR

Moves both Build Failure Analysis entry points to the shared Copilot PAT pool and regenerates their gh-aw workflows.

Changes:

• Adds pooled PAT selection and protected-environment bindings.
• Preserves workflow triggers, models, permissions, and concurrency.
• Regenerates workflow locks and pinned action metadata with gh-aw v0.86.2.

File	Description
.github/​workflows/​shared/​pat_pool.md	Defines shared PAT-pool selection.
.github/​workflows/​shared/​build-failure-analysis-shared.md	Adds shared authentication and validation.
.github/​workflows/​build-failure-analysis.md	Configures the automatic BFA workflow.
.github/​workflows/​build-failure-analysis.lock.yml	Generated automatic workflow.
.github/​workflows/​build-failure-analysis-command.md	Configures the command-triggered BFA workflow.
.github/​aw/​actions-lock.json	Updates gh-aw action lock metadata.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

• Missing Copilot PAT validation before threat-detection inference · New
• Missing Copilot PAT validation before threat-detection inference · New

Resolved since last review (1)

• Activation jobs cannot validate environment-scoped PAT tokens

[Copilot]

Copilot review overview

🔵 Needs a closer look

Moderate findings remain for action-lock synchronization and cache-save permissions.

Review effort: Lite
Findings: None

Resolved since last review (2)

• Missing Copilot PAT validation before threat-detection inference
• Missing Copilot PAT validation before threat-detection inference

Previously missed (1)

In code that hasn't changed since last review

Regenerate actions-lock.json with current workflow action refs

.github/​aw/​actions-lock.json:31

actions-lock.json is now out of sync with both regenerated workflow locks: it still pins actions/checkout@v6.0.2 and actions/setup-node@v6.4.0, while the locks use v7.0.1/v7.0.0, and it has no entries for the newly generated actions/cache/restore and actions/cache/save actions. This leaves the repository's compiler action lock unable to reproduce or validate the generated workflows; regenerate this file with all action refs and SHAs used by the locks.

[YuliiaKovalova]

Addressed the latest review's action-lock note in d93279e. actions-lock.json now includes actions/checkout@v7.0.1, actions/setup-node@v7.0.0, and actions/cache/restore / actions/cache/save at v6.1.0, using the exact SHAs already present in both BFA locks. The older checkout/setup-node entries are retained because the unchanged update-default-versions lock still uses them.

All eight unique action refs in each BFA lock now have matching shared-lock pins. Release-tag SHAs were verified; recompilation with schema/action validation and actionlint leaves both BFA locks, the updater lock, and the shared action lock unchanged. gh-aw v0.86.2 had been resolving the missing refs from its embedded pins, which explains why earlier reproduction checks succeeded despite the incomplete shared lock.

For the cache-save permissions note, no permission increase is warranted: the pinned cache action authenticates to the cache service with ACTIONS_RUNTIME_TOKEN, not GITHUB_TOKEN (pinned implementation). actions: read remains unchanged. Actual upstream BFA E2E verification is still pending.

[Copilot]

Copilot review overview

🔵 Needs a closer look

A critical action-lock entry remains unresolved, and protected credentials and end-to-end behavior still require validation.

Review effort: Lite
Findings: 1

Open (1)

• Retain setup action lock entry used by update-default-versions workflow · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Preserve the existing detection output guards while adding validator-success checks.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Retain setup action lock entry used by update-default-versions workflow

Previously missed (1)

In code that hasn't changed since last review

Preserve output guard before downloading AWF images

.github/​workflows/​shared/​build-failure-analysis-shared.md:94

This replacement of the caller's detection condition drops the output_types/has_patch guard. As a result, every non-skipped agent run now enters threat detection and downloads the AWF images before detection_guard can decide there is nothing to analyze, even when the agent produced no output. Preserve the guard in this shared condition while adding the validator-success check; keep always() and allow runs with partial output after an agent failure.

[YuliiaKovalova]

Addressed the latest review-summary item in 7a2c999: the detection job now requires both successful PAT validation and (needs.agent.outputs.output_types != '' || needs.agent.outputs.has_patch == 'true') before any runner setup or AWF image downloads. The compiler-owned always() and agent-not-skipped checks remain, so an agent that fails after emitting partial output still gets detection.

Both BFA entry points were regenerated. Reproduced the previous no-output startup and verified its removal, exercised 256 consumer-condition combinations plus the existing 132 token-validation and eight detection-output cases, and checked that the only executable workflow change is the detection job condition. Compilation/actionlint and byte-reproducible generation pass; all 11 shared action pins, including the updater's legacy setup pin, remain intact.

The remaining blocker is actual upstream E2E, not a running CI job: the current account has read-only access to dotnet/arcade. A maintainer needs to deploy the reviewed workflow to an approved protected upstream test branch and dispatch workflow 313838244 against a currently failed arcade-pr build with matching PR revisions. Completion requires successful pool authentication, analysis, detection, safe outputs, and a delivered report. No personal inference credential or branch-protection change is requested.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Protected credentials and end-to-end behavior remain unvalidated, with a fail-closed documentation issue unresolved.

Review effort: Lite
Findings: None

Previously missed (1)

In code that hasn't changed since last review

Update comment to document unconditional auth failure

.github/​workflows/​shared/​pat_pool.md:43

This comment still tells future consumers to define a COPILOT_GITHUB_TOKEN fallback, which contradicts the shared authentication contract above and this PR's requirement to fail closed without falling back to the legacy secret or Actions token. Please document the empty pool as an unconditional failure instead, so a future caller does not reintroduce the bypass.

[YuliiaKovalova]

Actual upstream attempt: deployment rejected before inference

I dispatched the real Arcade automatic workflow 313838244 on dotnet/arcade:users/YuliiaKovalova/bfa-pat-pool-e2e, at exact PR commit 7a2c999d943afc3a18572dbf7f2a70cb5f6812e7:

Run: https://github.com/dotnet/arcade/actions/runs/35994258685

• Inputs: failed arcade-pr build 1609556, PR Use renewable Azure CLI authentication for Helix #17592. Both PR head and merge revisions were revalidated immediately before dispatch.
• Fetch binlogs (Azure Pipelines) succeeded and retained build-failure-analysis-data (19,476,913 bytes).
• pre_activation failed before a runner started; it has no executed steps. GitHub's exact annotation is:

Branch "users/YuliiaKovalova/bfa-pat-pool-e2e" is not allowed to deploy to copilot-pat-pool due to environment protection rules.

pat_pool, validate_copilot_pat, activation, the agent, detection, and safe outputs were all skipped. This is an observed environment-policy rejection, not a Copilot HTTP 401 result and not successful E2E verification. No inference request or report delivery was exercised.

The outstanding administrator action is the existing request to protect the dedicated upstream test branch, without relaxing the environment policy. Deployment through a normal reviewed merge to already-protected main is the other route; it would still require runtime verification afterward.

This particular input build failed in Helix monitoring, so even an eventual legitimate no-op on these binlogs would not establish compilation-report delivery. A report-producing E2E must use a suitable current failure and revalidate its revisions before dispatch. The PR's E2E acceptance check remains unchecked.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Protected credentials and end-to-end report delivery remain unverified.

Review effort: Lite
Findings: None

[Evangelink]

Reviewed the source definitions and generated workflows for both Build Failure Analysis entry points. The PAT selector and validator remain scoped to the protected environment, both inference phases fail closed on validation, the detection output guard is preserved, and the action pins are consistent.

I also recompiled both workflows with checksum-verified gh-aw v0.86.2 and ran actionlint successfully. The protected-environment end-to-end credential/report-delivery validation remains pending as documented, but I do not see a blocking code issue.