Skip to content

[APPSEC-69389] Add AI Guard sensetive data redaction #6394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Strech merged 41 commits into from
Oct 2, 2026
Merged

[APPSEC-69389] Add AI Guard sensetive data redaction #6394

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
11f5389
Initial commit. Sensetive data redaction
Strech Sep 2, 2026
a121480
First round of refactoring without code quality
Strech Sep 2, 2026
f5cf03b
Refactor Evaluation::Result initialize interface
Strech Sep 4, 2026
b114391
Refactor Evaluation::Outcome interface
Strech Sep 4, 2026
d04da3b
Add telemetry reporting
Strech Sep 7, 2026
38ce760
Add DD_AI_GUARD_REDACTION_ENABLED configuration
Strech Sep 7, 2026
07e873a
Fix existing tests to pass for new code
Strech Sep 7, 2026
77cedf5
Rename AIGuard::Transport into AIGuard::HTTPClient
Strech Sep 7, 2026
2031afd
Add tests for evaluation outcome, result, response
Strech Sep 7, 2026
5708346
Add redaction tests
Strech Sep 7, 2026
fa0a966
Add telemetry tests
Strech Sep 8, 2026
810d0ef
Add evaluation tests
Strech Sep 8, 2026
dc26c55
Refactor redaction module and its interface
Strech Sep 10, 2026
19044d9
Add new and correct old RBS
Strech Sep 15, 2026
9895d28
Add sensetive data redaction to RubyLLM
Strech Sep 15, 2026
aaa7867
Extract message converter and redactor in RubyLLM
Strech Sep 16, 2026
5fb80cf
Fix tool call argument conversion to be JSON-based
Strech Sep 16, 2026
affffc2
Rework arguments to be JSON/Hash
Strech Sep 22, 2026
54574bb
Introduce multiple tool calls
Strech Sep 22, 2026
d9938da
Upgrade RubyLLM minimal dependency
Strech Sep 24, 2026
b112c49
Fix error propagation from instrumentation
Strech Sep 25, 2026
a73e770
Change instrumentation point for redacted messages
Strech Sep 25, 2026
d7a06ac
Add AI Guard documentation section
Strech Sep 25, 2026
33cc254
Fix late messages truncation
Strech Sep 25, 2026
de29276
Replace error counting with reporting
Strech Sep 25, 2026
1e2b067
Warn on redactions with raw content present
Strech Sep 25, 2026
496a22d
Fix double-evaluation when rendering chat messages
Strech Sep 25, 2026
ea06d79
Add note about double-encoding of images
Strech Sep 25, 2026
e555fca
Update RBS definitions
Strech Sep 25, 2026
c9ace46
Add changelog entries
Strech Sep 28, 2026
02def1d
Remove obsolete RubyLLM version
Strech Sep 28, 2026
607c47e
[🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/r…
dd-octo-sts[bot] Sep 28, 2026
ea00d14
Fix rubocop violations
Strech Sep 28, 2026
5259de8
Expose redaction replacements in evaluation result
Strech Sep 28, 2026
d03ab00
Update reference to ST with updated AI Guard setup
Strech Sep 28, 2026
cd19236
[🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/r…
dd-octo-sts[bot] Sep 28, 2026
7e7f47e
Address review feedback
Strech Sep 28, 2026
b918575
Rename with_* methods to copy
Strech Sep 29, 2026
2dda3a9
Update system-tests reference
Strech Sep 29, 2026
50b640e
Merge branch 'master' into appsec-69389-add-ai-guard-sensetive-data-r…
Strech Oct 1, 2026
03f8e22
[🤖] Lock Dependency: https://github.com/DataDog/dd-trace-rb/actions/r…
dd-octo-sts[bot] Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/system-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ jobs:
test:
needs:
- build
uses: DataDog/system-tests/.github/workflows/system-tests.yml@70f36a2a612d0f9e98af5f4882142e941ae09717 # Automated: Updated by .github/workflows/update-system-tests.yml.
uses: DataDog/system-tests/.github/workflows/system-tests.yml@7624ea2919a72738a2e923adbdf9c2d538bc4ae2 # Automated: Updated by .github/workflows/update-system-tests.yml.
permissions:
contents: read
id-token: write
Expand All @@ -80,7 +80,7 @@ jobs:
binaries_artifact: dd-trace-rb
desired_execution_time: 300 # 5 minutes
scenarios_groups: tracer_release
ref: 70f36a2a612d0f9e98af5f4882142e941ae09717 # Automated: Updated by .github/workflows/update-system-tests.yml.
ref: 7624ea2919a72738a2e923adbdf9c2d538bc4ae2 # Automated: Updated by .github/workflows/update-system-tests.yml.

@vpellan vpellan Oct 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ref is not merged yet on system-tests. We had a case yesterday of merging such a thing by accident, causing the CI to fail on every system-tests PR. Please revert this or merge it on system-tests before merging this here. There will be a job to prevent this in the future but it's not merged yet:
DataDog/system-tests#7874

@Strech Strech Oct 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a dead-lock, this PR heavily changing AI Guard interface and gem requirements due to non GA status, ST will be never green on prod environment because I have to change weblogs, hence I can't merge it

force_execute: ${{ needs.build.outputs.forced_tests }}
parametric_job_count: 8
push_to_test_optimization: true
Expand Down
2 changes: 1 addition & 1 deletion .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ requirements_json_test:

configure_system_tests:
variables:
SYSTEM_TESTS_REF: 70f36a2a612d0f9e98af5f4882142e941ae09717 # Automated: Updated by .github/workflows/update-system-tests.yml.
SYSTEM_TESTS_REF: 7624ea2919a72738a2e923adbdf9c2d538bc4ae2 # Automated: Updated by .github/workflows/update-system-tests.yml.
SYSTEM_TESTS_SCENARIOS_GROUPS: "simple_onboarding,simple_onboarding_appsec,lib-injection,docker_ssi"
SYSTEM_TEST_BUILD_ATTEMPTS: 3

Expand Down
5 changes: 2 additions & 3 deletions Matrixfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -399,9 +399,8 @@
"" => "✅ 2.5 / ✅ 2.6 / ✅ 2.7 / ✅ 3.0 / ✅ 3.1 / ✅ 3.2 / ✅ 3.3 / ✅ 3.4 / ✅ 4.0",
},
"ai_guard:ruby_llm" => {
# FIXME: ruby-llm-latest locks ruby_llm 2.x, which removed RubyLLM::Content; flip to ✅ when the integration supports it
"ruby-llm-1" => "❌ 2.5 / ❌ 2.6 / ❌ 2.7 / ❌ 3.0 / ✅ 3.1 / ✅ 3.2 / ✅ 3.3 / ✅ 3.4 / ✅ 4.0",
"ruby-llm-latest" => "❌ 2.5 / ❌ 2.6 / ❌ 2.7 / ❌ 3.0 / ❌ 3.1 / ❌ 3.2 / ❌ 3.3 / ❌ 3.4 / ❌ 4.0",
"ruby-llm-min" => "❌ 2.5 / ❌ 2.6 / ❌ 2.7 / ❌ 3.0 / ✅ 3.1 / ✅ 3.2 / ✅ 3.3 / ✅ 3.4 / ✅ 4.0",
"ruby-llm-latest" => "❌ 2.5 / ❌ 2.6 / ❌ 2.7 / ❌ 3.0 / ✅ 3.1 / ✅ 3.2 / ✅ 3.3 / ✅ 3.4 / ✅ 4.0"
},
"ai_guard:rack" => {
"rack-latest" => "✅ 2.5 / ✅ 2.6 / ✅ 2.7 / ✅ 3.0 / ✅ 3.1 / ✅ 3.2 / ✅ 3.3 / ✅ 3.4 / ✅ 4.0",
Expand Down
2 changes: 1 addition & 1 deletion appraisal/ruby-3.1.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@
'opentelemetry-sdk' => '~> 1.1',
'opentelemetry-metrics-sdk' => '>= 0.8',
})
build_coverage_matrix('ruby-llm', [1], gem: 'ruby_llm')
build_coverage_matrix('ruby-llm', gem: 'ruby_llm', min: '2.0.0')
build_coverage_matrix('kicks', min: '3.0.0')

appraise 'sneakers' do
Expand Down
2 changes: 1 addition & 1 deletion appraisal/ruby-3.2.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@
'opentelemetry-sdk' => '~> 1.1',
'opentelemetry-metrics-sdk' => '>= 0.8',
})
build_coverage_matrix('ruby-llm', [1], gem: 'ruby_llm')
build_coverage_matrix('ruby-llm', gem: 'ruby_llm', min: '2.0.0')
build_coverage_matrix('kicks', min: '3.0.0')

appraise 'sneakers' do
Expand Down
2 changes: 1 addition & 1 deletion appraisal/ruby-3.3.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@
'opentelemetry-sdk' => '~> 1.1',
'opentelemetry-metrics-sdk' => '>= 0.8',
})
build_coverage_matrix('ruby-llm', [1], gem: 'ruby_llm')
build_coverage_matrix('ruby-llm', gem: 'ruby_llm', min: '2.0.0')
build_coverage_matrix('kicks', min: '3.0.0')

appraise 'sneakers' do
Expand Down
2 changes: 1 addition & 1 deletion appraisal/ruby-3.4.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@
'opentelemetry-sdk' => '~> 1.1',
'opentelemetry-metrics-sdk' => '>= 0.8',
})
build_coverage_matrix('ruby-llm', [1], gem: 'ruby_llm')
build_coverage_matrix('ruby-llm', gem: 'ruby_llm', min: '2.0.0')
build_coverage_matrix('kicks', min: '3.0.0')

appraise 'sneakers' do
Expand Down
2 changes: 1 addition & 1 deletion appraisal/ruby-4.0.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@
'opentelemetry-sdk' => '~> 1.1',
'opentelemetry-metrics-sdk' => '>= 0.8',
})
build_coverage_matrix('ruby-llm', [1], gem: 'ruby_llm')
build_coverage_matrix('ruby-llm', gem: 'ruby_llm', min: '2.0.0')
build_coverage_matrix('kicks', min: '3.0.0')

appraise 'sneakers' do
Expand Down
2 changes: 1 addition & 1 deletion gemfiles/ruby_3.1_ruby_llm_1.gemfile → gemfiles/ruby_3.1_ruby_llm_min.gemfile
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading