Pin injector for tracer SSI by nccatoni · Pull Request #7720 · DataDog/system-tests

nccatoni · 2026-09-14T16:08:01Z

Motivation

Tracer SSI pipelines currently test custom library packages with the latest injector unless callers coordinate an injector override themselves. That can expose tracer validation to an injector release before it has been deliberately promoted. APMSP-3754 establishes a repository-owned injector pin that DataDog/auto_inject can advance after successful default-branch OCI publication.

Changes

When DD_INSTALLER_LIBRARY_VERSION is set and DD_INSTALLER_INJECTOR_VERSION is not already supplied, SSI framework entry points load the trimmed version from auto_inject.lock and export it as DD_INSTALLER_INJECTOR_VERSION. Explicit injector overrides retain precedence. When the lock supplies the version, the framework logs the selected pinned version.

The shared version helper lives with the framework scripts under utils/scripts. The same rule is applied to Docker SSI, AWS VM SSI, AWS container-app SSI, and generated external pipelines. Default SSI runs remain unchanged when no custom library version is present. The lock contains one immutable package version followed by a newline so automated updates remain a minimal one-line change. The onboarding guidance documents the automatic pin and override behavior.

Testing

Added focused tests for the pinned injector and its log output, explicit injector override, default-library branch, and lock-file format invariant.
pytest --scenario TEST_THE_TEST -q tests/test_the_test/test_installer_versions.py tests/test_the_test/test_external_gitlab_pipeline.py tests/test_the_test/test_gitlab_pipeline_structure.py — 12 passed.
Repository-wide mypy and Ruff checks passed. The aggregate formatter stopped only while downloading the absent yamlfmt binary through the sandbox network tunnel; no YAML is changed by this PR.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

PR by Bits - View session in Datadog

Comment @DataDog to request changes

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

datadog-official · 2026-09-14T16:08:02Z

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix: Disabled | Enable

Comment @DataDog to request changes

datadog-datadog-prod-us1 · 2026-09-14T16:08:06Z

I can only run on private repositories.

github-actions · 2026-09-14T16:08:28Z

CODEOWNERS have been resolved as:

tests/test_the_test/test_installer_versions.py                          @DataDog/system-tests-reviewers
utils/build/auto_inject.lock                                            @DataDog/system-tests-core
utils/build/ssi/base/installer_versions.sh                              @DataDog/system-tests-core
docs/understand/scenarios/onboarding.md                                 @DataDog/system-tests-core
utils/_context/_scenarios/docker_ssi.py                                 @DataDog/system-tests-core
utils/build/ssi/base/base_ssi.Dockerfile                                @DataDog/system-tests-core
utils/build/ssi/base/install_script_ssi.sh                              @DataDog/system-tests-core
utils/build/virtual_machine/provisions/auto-inject/auto-inject_installer_manual.yml  @DataDog/system-tests-core
utils/build/virtual_machine/provisions/auto-inject/repositories/autoinstall/execute_install_script.sh  @DataDog/system-tests-core
utils/build/virtual_machine/provisions/container-auto-inject-install-script/auto-inject_container_script.yml  @DataDog/system-tests-core
utils/build/virtual_machine/provisions/host-auto-inject-install-script/auto-inject_host_script.yml  @DataDog/system-tests-core

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

datadog-official · 2026-09-18T14:23:42Z

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 320 Pipeline jobs failed

Testing the test | System Tests (ruby, dev) / End-to-end #1 / rails61 1 — ❌ 15 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.ai_guard.test_ai_guard_sdk.Test_AIGuardEvent_Tag.test_ai_guard_event[rails61] from system_tests_suite

assert 500 == 200
 &#43;  where 500 = HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}).status_code
 &#43;    where HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}) = &lt;tests.ai_guard.test_ai_guard_sdk.Test_AIGuardEvent_Tag object at 0x7f814c66fa40&gt;.r

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_AIGuardEvent_Tag object at 0x7f814c66fa40&gt;

    def test_ai_guard_event(self):
        &#34;&#34;&#34;Test AI Guard sets ai_guard.event:true tag in the local root span of the trace.&#34;&#34;&#34;
&gt;       assert self.r.status_code == 200
E       assert 500 == 200
...

❌ tests.ai_guard.test_ai_guard_sdk.Test_AIGuardStandalone.test_standalone_keeps_ai_guard_trace[rails61] from system_tests_suite

assert 500 == 200
 &#43;  where 500 = HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}).status_code
 &#43;    where HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}) = &lt;tests.ai_guard.test_ai_guard_sdk.Test_AIGuardStandalone object at 0x7efed6507a40&gt;.r

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_AIGuardStandalone object at 0x7efed6507a40&gt;

    def test_standalone_keeps_ai_guard_trace(self):
&gt;       assert self.r.status_code == 200
E       assert 500 == 200
E        &#43;  where 500 = HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}).status_code
...

❌

tests.ai_guard.test_ai_guard_sdk.Test_AIGuardStandalone_APMDisabledMarker.test_all_spans_have_apm_disabled_marker[rails61]

from system_tests_suite

assert 500 == 200
 &#43;  where 500 = HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}).status_code
 &#43;    where HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}) = &lt;tests.ai_guard.test_ai_guard_sdk.Test_AIGuardStandalone_APMDisabledMarker object at 0x7efed6507a70&gt;.r

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_AIGuardStandalone_APMDisabledMarker object at 0x7efed6507a70&gt;

    def test_all_spans_have_apm_disabled_marker(self) -&gt; None:
&gt;       assert self.r.status_code == 200
E       assert 500 == 200
E        &#43;  where 500 = HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}).status_code
...

↳ and 12 more — View all

Testing the test | System Tests (ruby, dev) / End-to-end #2 / rails61 2 — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled.test_sdk_disabled[rails61] from system_tests_suite

assert 500 == 200
 &#43;  where 500 = HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}).status_code
 &#43;    where HttpResponse(status_code:500, headers:{&#39;X-Frame-Options&#39;: &#39;SAMEORIGIN&#39;, &#39;X-XSS-Protection&#39;: &#39;1; mode=block&#39;, &#39;X-Conten...&#39;Transfer-Encoding&#39;: &#39;chunked&#39;}, text:{&#34;error&#34;:&#34;unknown keywords: :id, :tool_name, :arguments&#34;,&#34;type&#34;:&#34;ArgumentError&#34;}) = &lt;tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled object at 0x7fbef9eb13d0&gt;.request

self = &lt;tests.ai_guard.test_ai_guard_sdk.Test_SDK_Disabled object at 0x7fbef9eb13d0&gt;

    def test_sdk_disabled(self):
        &#34;&#34;&#34;Test AI Guard disabled by default, it should always return ALLOW and no span should be generated&#34;&#34;&#34;
&gt;       assert self.request.status_code == 200
E       assert 500 == 200
...

DataDog/system-tests | Ubuntu_22_arm64.HOS: [test-app-dotnet] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

View all 320 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 7637e4a | Docs | View more details | Give us feedback!}

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

Co-authored-by: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

chatgpt-codex-connector · 2026-09-25T13:53:39Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-25T13:56:40.427766Z	`74634c9`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

robertomonteromiguel

LGTM. Let's wait for the CI

APMSP-3754 pin injector for tracer SSI

26faee5

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

datadog-datadog-prod-us1-2 Bot added the Bits AI label Sep 14, 2026

Move installer version helper under scripts

a11cb81

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

nccatoni changed the title ~~Pin injector for tracer SSI~~ APMSP-3754 Pin injector for tracer SSI Sep 18, 2026

datadog-bits and others added 2 commits September 18, 2026 14:59

Preserve explicit injector version

9e38c43

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

Document and log pinned injector

73a0287

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

nccatoni changed the title ~~APMSP-3754 Pin injector for tracer SSI~~ Pin injector for tracer SSI Sep 18, 2026

datadog-bits and others added 5 commits September 18, 2026 15:35

Apply injector pin at SSI install paths

6065f25

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

Keep Docker SSI build context narrow

0682365

Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

Merge branch 'main' into dd/nccatoni/pin-ssi-injector-202609141700

dc14c75

Loud failure when a lock is missing

4ec6b7e

Pin injector for tracer SSI (#7826)

74634c9

Co-authored-by: datadog-bits <263423550+datadog-bits@users.noreply.github.com> Co-authored-by: nccatoni <222672590+nccatoni@users.noreply.github.com>

nccatoni marked this pull request as ready for review September 25, 2026 13:53

nccatoni requested review from a team as code owners September 25, 2026 13:53

nccatoni requested a review from robertomonteromiguel September 25, 2026 13:53

robertomonteromiguel reviewed Sep 28, 2026

View reviewed changes

robertomonteromiguel requested changes Sep 28, 2026

View reviewed changes

nccatoni added 2 commits September 29, 2026 11:18

Using install.datadoghq.com for locked auto installer versions

3f7d9cd

Move lock file to utils/build

2c92321

nccatoni requested a review from robertomonteromiguel September 29, 2026 09:52

robertomonteromiguel requested changes Sep 29, 2026

View reviewed changes

Comment thread utils/build/ssi/base/installer_versions.sh Outdated

nccatoni added 2 commits September 29, 2026 16:51

Fix unconditionally setting installtesting.datad0g.com

6936c35

Merge branch 'main' into dd/nccatoni/pin-ssi-injector-202609141700

420df16

nccatoni requested a review from robertomonteromiguel September 30, 2026 11:52

robertomonteromiguel approved these changes Sep 30, 2026

View reviewed changes

nccatoni added 2 commits October 1, 2026 14:36

Merge branch 'main' into dd/nccatoni/pin-ssi-injector-202609141700

5d70910

Merge branch 'main' into dd/nccatoni/pin-ssi-injector-202609141700

7637e4a

Navigation Menu

Pin injector for tracer SSI - #7720

Pin injector for tracer SSI#7720
nccatoni wants to merge 15 commits into
mainfrom
dd/nccatoni/pin-ssi-injector-202609141700

nccatoni commented Sep 14, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Sep 14, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Sep 18, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertomonteromiguel left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nccatoni commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Testing

Workflow

Reviewer checklist

Uh oh!

datadog-official Bot commented Sep 14, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Sep 14, 2026

Uh oh!

github-actions Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertomonteromiguel left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nccatoni commented Sep 14, 2026 •

edited

Loading

datadog-official Bot commented Sep 14, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

github-actions Bot commented Sep 14, 2026 •

edited

Loading

datadog-official Bot commented Sep 18, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 25, 2026 •

edited

Loading