Retry AWS SSI across zones by nccatoni · Pull Request #7714 · DataDog/system-tests

nccatoni · 2026-09-14T13:20:40Z

Motivation

AWS SSI runs can fail intermittently when EC2 has no capacity for the requested instance type in the randomly selected availability zone. This was observed for AlmaLinux 8 arm64 with t4g.medium capacity in us-east-1c, forcing an expensive full CI retry even when capacity is available in another configured zone.

Changes

Normalize comma-separated subnet and security-group configuration so multiple network IDs are handled reliably.
Shuffle configured subnets once per run and, on InsufficientInstanceCapacity, retry the Pulumi update with each remaining subnet at most once.
Preserve the existing same-subnet retry behavior for IdempotentParameterMismatch and keep exhausted capacity errors classified for the existing CI retry fallback.
Document that AWS SSI should be configured with subnet IDs from multiple availability zones.

Testing

Added focused tests covering multi-subnet normalization, successful zone rotation after repeated capacity failures, exhaustion after all zones, and unchanged idempotency retry behavior.
DD_SITE=datadoghq.com venv/bin/python -m pytest --confcutdir=tests/test_the_test tests/test_the_test/test_aws_provider.py -q: 4 passed.
DD_SITE=datadoghq.com ./format.sh --check passed mypy, Ruff, whitespace, yamlfmt, yamllint, manifest validation, AI Guard fixture validation, and shellcheck. The final Node.js linter image pull could not run because the sandbox proxy returned Bad Gateway for Docker Hub.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

PR by Bits - View session in Datadog

Comment @DataDog to request changes

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

datadog-datadog-prod-us1 · 2026-09-14T13:20:41Z

View session in Datadog

Bits Code status: ✅ Done

Comment @DataDog to request changes

datadog-datadog-prod-us1 · 2026-09-14T13:20:45Z

I can only run on private repositories.

github-actions · 2026-09-14T13:21:13Z

CODEOWNERS have been resolved as:

tests/test_the_test/test_aws_provider.py                                @DataDog/system-tests-reviewers
.gitlab/ssi_gitlab-ci.yml                                               @DataDog/system-tests-core
docs/understand/scenarios/onboarding.md                                 @DataDog/system-tests-core
tests/test_the_test/test_gitlab_pipeline_structure.py                   @DataDog/system-tests-reviewers
utils/scripts/ssi_wizards/aws_onboarding_wizard.sh                      @DataDog/system-tests-core
utils/virtual_machine/aws_infra_exceptions.json                         @DataDog/system-tests-core
utils/virtual_machine/aws_provider.py                                   @DataDog/system-tests-core
utils/virtual_machine/virtual_machines.py                               @DataDog/system-tests-core

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

datadog-datadog-prod-us1 · 2026-10-02T09:49:03Z

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 8 Pipeline jobs failed

Testing the test | System Tests (ruby, dev) / End-to-end #1 / rails52 1 — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled.test_sdk_disabled[rails52] from system_tests_suite

assert 500 == 200
 &#43;  where 500 = HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}).status_code
 &#43;    where HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}) = &lt;tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled object at 0x7f4284a37920&gt;.request

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled object at 0x7f4284a37920&gt;

    def test_sdk_disabled(self):
        &#34;&#34;&#34;Test AI Guard disabled by default, it should always return ALLOW and no span should be generated&#34;&#34;&#34;
&gt;       assert self.request.status_code == 200
E       assert 500 == 200
...

Testing the test | System Tests (ruby, dev) / End-to-end #1 / rails61 1 — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled.test_sdk_disabled[rails61] from system_tests_suite

assert 500 == 200
 &#43;  where 500 = HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}).status_code
 &#43;    where HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}) = &lt;tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled object at 0x7f0954c7b830&gt;.request

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled object at 0x7f0954c7b830&gt;

    def test_sdk_disabled(self):
        &#34;&#34;&#34;Test AI Guard disabled by default, it should always return ALLOW and no span should be generated&#34;&#34;&#34;
&gt;       assert self.request.status_code == 200
E       assert 500 == 200
...

Testing the test | System Tests (ruby, dev) / End-to-end #1 / rails72 1 — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

View all 8 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ee6a8d1 | Docs | View more details | Give us feedback!}

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

…-ssi-across-zones Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com> # Conflicts: # tests/test_the_test/test_gitlab_pipeline_structure.py

chatgpt-codex-connector · 2026-10-02T13:24:29Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-02T13:28:01.362272Z	`ee6a8d1`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

APMSP-3972 Retry AWS SSI across zones

1c6740e

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

datadog-prod-us1-5 Bot added the Bits AI label Sep 14, 2026

nccatoni changed the title ~~APMSP-3972 Retry AWS SSI across zones~~ Retry AWS SSI across zones Sep 14, 2026

APMSP-3972 Configure AWS SSI subnets across zones

bd7a199

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

datadog-bits and others added 3 commits October 2, 2026 09:51

APMSP-3972 Configure AWS SSI CI subnets

82a9ac5

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into dd/nccatoni/retry-aws…

3a511db

…-ssi-across-zones Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com> # Conflicts: # tests/test_the_test/test_gitlab_pipeline_structure.py

Fix format

ee6a8d1

nccatoni marked this pull request as ready for review October 2, 2026 13:24

nccatoni requested review from a team as code owners October 2, 2026 13:24

nccatoni requested a review from robertomonteromiguel October 2, 2026 13:25

nccatoni closed this Oct 2, 2026

Navigation Menu

Retry AWS SSI across zones - #7714

Retry AWS SSI across zones#7714
nccatoni wants to merge 5 commits into
mainfrom
dd/nccatoni/retry-aws-ssi-across-zones

nccatoni commented Sep 14, 2026

Uh oh!

datadog-datadog-prod-us1 Bot commented Sep 14, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Oct 2, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nccatoni commented Sep 14, 2026

Motivation

Changes

Testing

Workflow

Reviewer checklist

Uh oh!

datadog-datadog-prod-us1 Bot commented Sep 14, 2026 • edited by datadog-prod-us1-5 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Oct 2, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-datadog-prod-us1 Bot commented Sep 14, 2026 •

edited by datadog-prod-us1-5 Bot

Loading

github-actions Bot commented Sep 14, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Oct 2, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

chatgpt-codex-connector Bot commented Oct 2, 2026 •

edited

Loading