test(ffe): enable .NET agentless configuration source by pavlokhrebto · Pull Request #7496 · DataDog/system-tests

pavlokhrebto · 2026-08-12T13:02:46Z

Motivation

DataDog/dd-trace-dotnet#9017 implements the .NET agentless Feature Flagging configuration source (FFL-2703). This enables the shared configuration-source contract for .NET, following the Node.js (#7355), Java (#7300), and Python (#7411) activations.

Changes

Enable tests/parametric/test_ffe/test_configuration_sources.py for .NET from >=3.54.0, the first release with the agentless source.
Add /ffe/start and /ffe/evaluate endpoints to the .NET parametric test app (FfeTestApi.cs), mirroring the Python and Java controllers.
/ffe/start creates a DatadogProvider and awaits Api.Instance.SetProviderAsync(). Only the async call runs the provider's InitializeAsync, which the tracer's CallTarget instrumentation uses to start agentless/RC delivery and wait for the first configuration.
/ffe/evaluate evaluates a flag through the OpenFeature client (Get*DetailsAsync) and returns {value, reason, errorCode}, with errorCode converted from the ErrorType enum to UPPER_SNAKE_CASE (e.g. ProviderNotReady → PROVIDER_NOT_READY) to match the test contract.
Add the Datadog.FeatureFlags.OpenFeature package reference (2.3.1) to the parametric test app.

Disabled tests

Five cold-start cases are marked bug (FFL-3325) for .NET:

Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery::test_bad_to_unchanged_cold_preserves_not_ready
Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery::test_malformed_cold
Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery::test_request_timeout_cold
Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery::test_unauthorized_cold
Test_Feature_Flag_Configuration_Source_Selection::test_remote_config_without_rc_does_not_fallback_to_agentless

They found a real .NET bug on their first run. Before configuration arrives, the provider returned null as the flag value instead of the caller's default: {'errorCode': 'PROVIDER_NOT_READY', 'value': None} where the tests expect 'default'. The error code and reason were already correct. Go, Java, Python and Node return the default and pass these cases.

The fix is in DataDog/dd-trace-dotnet#9299. Once a tracer release includes it, the bug markers get replaced with that version.

Decisions

Keep released configuration-source activation independent of EVP fallback test(dotnet): enable agentless EVP exposure egress #7707.
Preserve the five FFL-3325 exclusions pending the release containing dd-trace-dotnet#9299.
Refresh this existing PR rather than creating a duplicate activation.

Validation (2026-09-23 main refresh)

Base: 8770fe3d914d8f7028c1d4177292acc5b5498a34
Candidate: 4657f3bac00edab8adbb2f5f8cefd7e4a068547f (signed; GitHub verified).

./format.sh --check — passed.
python -m pytest -S TEST_THE_TEST tests/test_the_test/test_manifest.py tests/test_the_test/test_version.py -q --disable-warnings — 171 passed, Python 3.12, with PYTHONPATH=$PWD.
No fresh .NET parametric E2E run in this refresh. Current-head CI must validate the adapter with the released SDK; the manifest/version self-tests are not runtime proof.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-08-12T13:03:14Z

CODEOWNERS have been resolved as:

utils/build/docker/dotnet/parametric/Endpoints/FfeTestApi.cs            @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
.github/CODEOWNERS                                                      @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/system-tests-reviewers
utils/build/docker/dotnet/parametric/ApmTestApi.csproj                  @DataDog/system-tests-reviewers
utils/build/docker/dotnet/parametric/Program.cs                         @DataDog/system-tests-reviewers

datadog-datadog-prod-us1 · 2026-08-12T13:28:39Z

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ac712c1 | Docs | View more details | Give us feedback!}

…endpoint derivation (#9040) ## Summary of changes Adds the foundational configuration layer for agentless Feature Flags delivery: new configuration keys, a `FeatureFlagsSettings` class with source resolution and validation, and an `AgentlessEndpoint` struct for deriving the CDN URL from the Datadog site or a custom base URL. This is PR 1 of a stacked PR series implementing agentless Feature Flags configuration delivery (FFL-2703), porting functionality from [dd-trace-py#19331](DataDog/dd-trace-py#19331) and [dd-trace-java#11892](DataDog/dd-trace-java#11892). ## Reason for change The .NET tracer currently delivers Feature Flags configuration exclusively through the Datadog Agent's Remote Configuration. To support agentless (CDN-backed) delivery, we need a configuration layer that: - Selects between `agentless` (new default), `remote_config`, and `disabled` sources - Derives the managed CDN endpoint from `DD_SITE` or accepts a custom `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL` - Validates poll interval and request timeout settings - Grandfathers existing `DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED` users onto Remote Configuration ## Implementation details - **`FeatureFlagsSource` enum** — `Disabled`, `Agentless`, `RemoteConfig` - **`FeatureFlagsSettings`** — reads all new env vars, resolves the source with the cross-SDK precedence contract (kill switch → explicit source → fail-closed → legacy grandfathering → default agentless), validates poll interval (1–3600s) and request timeout (>0s) - **`AgentlessEndpoint`** — derives `https://ufc-server.ff-cdn.<site>/api/v2/feature-flagging/config/rules-based/server` from the site, appends `dd_env` when configured, accepts custom HTTP/HTTPS URLs, and never echoes URLs in error messages (credentials safety) - **New config keys** in `supported-configurations.yaml` + generated `ConfigurationKeys.FeatureFlags.g.cs`: - `DD_FEATURE_FLAGS_ENABLED` (default `true`, supersedes `DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED`) - `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE` (default `agentless`) - `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL` - `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS` (default 30, type `int`) - `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS` (default 5, type `int`) - `DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS` (default 10000, implementation B) - **`[Obsolete]`** on `FlaggingProviderEnabled` with pragma suppression in `FeatureFlagsSettings` This PR does not wire anything to `FeatureFlagsModule` yet — that happens in a later PR in the stack. ## Test coverage - `FeatureFlagsSettingsTests` — 43 test cases covering source resolution (kill switch, explicit source, legacy grandfathering, `offline` sentinel, invalid values), blank/whitespace handling, casing normalization, defaults, poll interval validation, request timeout validation, initialization timeout, and base URL reading - `AgentlessEndpointTests` — 18 test cases covering managed endpoint derivation (site lowercasing, staging, govcloud), `dd_env` query parameter (addition, null, escaping), custom endpoint path handling (origin-only gets canonical path, custom path used verbatim), HTTP acceptance for custom endpoints, invalid URL rejection, empty/whitespace site rejection, and credentials-in-error safety ## Other details  Stacked PRs: - PR 1 (this): Configuration keys, settings, and endpoint derivation - PR 2: UFC parser and agentless poller - PR 3: Module wiring and OpenFeature activation - PR 4: Manual API activation  System tests: [DataDog/system-tests#7496](DataDog/system-tests#7496)

## Summary of changes Adds the agentless delivery mechanism: a `UfcConfigurationParser` that validates the JSON:API envelope returned by the CDN endpoint, and an `AgentlessConfigurationSource` HTTP poller with retries, ETag caching, gzip decompression, and shutdown safety. This is PR 2 of a stacked PR series implementing agentless Feature Flags configuration delivery (FFL-2703), porting functionality from [dd-trace-py#19331](DataDog/dd-trace-py#19331) and [dd-trace-java#11892](DataDog/dd-trace-java#11892). Stacked on [PR 1 #9040](#9040). ## Reason for change PR 1 added the configuration layer (settings, endpoint derivation). This PR adds the actual HTTP delivery mechanism that polls the CDN endpoint for UFC flag configuration. It is self-contained: the poller and parser are fully testable in isolation and are not yet wired to the `FeatureFlagsModule` (that happens in PR 3). ## Implementation details - **`UfcConfigurationParser`** — validates the JSON:API Universal Flag Configuration response envelope, extracts `data.attributes` into `ServerConfiguration`. Rejects raw UFC documents (without the JSON:API wrapper) so all delivery sources agree on one wire format. Returns descriptive errors; never throws. - **`AgentlessConfigurationSource`** — HTTP poller with: - Fixed-delay polling (no overlap) - In-tick retries (max 3) for 408/429/5xx/network failures, with jittered backoff derived from the poll interval (`clamp(P/6, 2s, 10s)` then `clamp(P/3, 5s, 30s)`) - ETag / `If-None-Match` conditional polling; ETag advances only after parse + apply succeed - Gzip decompression (neither transport decompresses automatically) - Self-tracing suppression (`x-datadog-tracing-enabled: false` header) - API key sent only to managed (Datadog-derived) endpoints, never custom ones - Warn-once per failure category (authentication, http, request) to prevent log flooding - Shutdown safety: `CancellationTokenSource` cancels the loop; shutdown checks after request and before `ApplyAsync` prevent use-after-dispose - `Create()` factory returns `null` when endpoint can'"'t be built or API key is missing - Idempotent `Start()` via `Interlocked.CompareExchange` ## Test coverage - **`UfcConfigurationParserTests`** (16 tests) — valid envelope parsing, malformed JSON rejection, invalid envelope rejection (wrong type, missing data, raw UFC), invalid attributes rejection (missing format/createdAt/environment/flags, wrong types), flag parsing - **`AgentlessConfigurationSourceTests`** (12 tests) — 200 success, ETag forwarding, 304/401/malformed handling, disposal before/after request, retry on 500 then success, max retries on 500, no retry on 400, gzip response, network error, idempotent start ## Other details  Stacked PRs: - PR 1 [#9040](#9040): Configuration keys, settings, and endpoint derivation - PR 2 (this): UFC parser and agentless HTTP poller - PR 3: Module wiring and OpenFeature activation - PR 4: Manual API activation  System tests: [DataDog/system-tests#7496](DataDog/system-tests#7496)

…InitializeAsync (#9044) ## Summary of changes Wires the agentless poller and RC subscription into `FeatureFlagsModule` with source-agnostic activation, and adds `InitializeAsync` through the OpenFeature provider lifecycle so delivery starts when application code initializes the provider. This is PR 3 of a stacked PR series implementing agentless Feature Flags configuration delivery (FFL-2703), porting functionality from [dd-trace-py#19331](DataDog/dd-trace-py#19331) and [dd-trace-java#11892](DataDog/dd-trace-java#11892). Stacked on [PR 1 #9040](#9040) and [PR 2 #9042](#9042). ## Reason for change PR 1 added the configuration layer and PR 2 added the poller and parser. This PR connects them: the module now selects the delivery source at construction time, starts polling only when the provider is initialized, and waits for the first configuration before returning from `InitializeAsync`. ## Implementation details - **`TracerSettings`** — replaces `IsFlaggingProviderEnabled` with `FeatureFlagsSettings FeatureFlags` property - **`FeatureFlagsModule`** — source-agnostic constructor: - RC subscription only when `Source == RemoteConfig` (gated, no longer always-on) - `Create()` checks `settings.FeatureFlags.Enabled` - `Activate()` starts the agentless poller or marks RC intent, with `_deliveryStarted` tracking - `InitializeAsync()` calls `Activate()`, waits for `FirstConfigReceived` with timeout, skips the wait if no delivery started - `ApplyConfiguration()` — public method used by both agentless and RC paths, signals `FirstConfigReceived` - `Dispose()` now also disposes the agentless source and unsubscribes RC - **`OpenFeatureSdkInitializeAsyncIntegration`** (new CallTarget) — intercepts `FeatureFlagsSdk.InitializeAsync` stub and calls `FeatureFlagsModule.InitializeAsync` - **`DatadogProvider.InitializeAsync`** — override that delegates to `FeatureFlagsSdk.InitializeAsync` - **`FeatureFlagsSdk.InitializeAsync`** — no-op stub in the OpenFeature package - **`supported_calltargets.g.json`** — registers the new integration - **`SpanEnrichmentIntegrationTests`** — suppresses `[Obsolete]` warning on `FlaggingProviderEnabled` ## Test coverage - **`FeatureFlagsModuleTests`** (9 tests) — RC config update/reset, agentless source doesn't subscribe to RC, RC source subscribes and advertises capability, disabled returns null, `InitializeAsync` returns immediately when config already applied, returns when config arrives while waiting, returns on timeout without throwing, returns on cancellation without throwing, returns immediately when agentless source cannot start (no API key) ## Other details  Stacked PRs: - PR 1 [#9040](#9040): Configuration keys, settings, and endpoint derivation - PR 2 [#9042](#9042): UFC parser and agentless HTTP poller - PR 3 (this): Module wiring and OpenFeature activation - PR 4: Manual API activation  System tests: [DataDog/system-tests#7496](DataDog/system-tests#7496)

…ebto/FFL-2703/dotnet-agentless-configuration

…e 2.x

…ebto/FFL-2703/dotnet-agentless-configuration

Refresh the existing configuration activation without folding in EVP fallback. Environment: Datadog workspace

…ror (#9299) ## Summary of changes - `FeatureFlagsModule.Evaluate` returns the caller's default instead of `null` when no configuration has arrived. - `FeatureFlagsSdk.GetResolutionDetails` returns the caller's default for every evaluation that carries an error, and when the tracer is not attached. - `Datadog.FeatureFlags.OpenFeature` 2.3.1 → 2.3.2. ## Reason for change Before configuration arrives, `GetStringValueAsync("flag", "fallback")` returned `null` instead of `"fallback"`, with `PROVIDER_NOT_READY`. That is the startup window every application passes through. OpenFeature substitutes the caller's default only when a provider throws. This provider reports errors as resolution details instead, so the default must travel with the details. Three layers could have supplied it, and none did: 1. **Tracer** — the module's not-ready shortcut returned `null`. It came in with the original SDK in #7896, where the error string was `No config loaded`. #8367 changed the string to `PROVIDER_NOT_READY` and kept the `null`. The evaluator's own not-ready path already returned `defaultValue`, so the two paths disagreed. 2. **Package** — `GetResolutionDetails` never received the default. It copied whatever value the tracer returned, and returned `default!` when the tracer is not attached. For a string flag that is `null`. It came in with the package in #8077. 3. **OpenFeature client** — it does not step in, because the provider does not throw. The same gap also lost JSON defaults. On error the tracer hands back the caller's OpenFeature `Value`, which `JsonToValue` cannot convert, so the caller got an empty `Value`. ## How we found it system-tests [#7496](DataDog/system-tests#7496) enabled the Feature Flags configuration-source tests for .NET for the first time. On the first run, the five cold-start cases failed with `{'errorCode': 'PROVIDER_NOT_READY', 'value': None}` where they expect the default. Those tests had been `missing_feature` for .NET, and the unit tests on this path asserted only the error, never the value, so nothing had covered it. ## Implementation details The package is the layer that owns the rule: any evaluation with an error type resolves to the caller's default. The tracer fix is still needed on its own, because it ships with the tracer, while the package reaches customers only when they upgrade it. `Resolve<T>` now takes `T defaultValue` instead of `object?`. No CallTarget instruments `Resolve`, so changing the signature is safe. ## Test coverage `Evaluate_WhenNoConfigurationYet_ReturnsTheDefaultValue` covers string, boolean, integer and numeric flags, and asserts the value. The package has no unit-test project. The system-tests cold-start cases cover the not-ready path end to end. ## Other details Jira: [FFL-3325](https://datadoghq.atlassian.net/browse/FFL-3325) Delivery: the tracer fix ships in the next tracer release. The package fix needs `Datadog.FeatureFlags.OpenFeature` 2.3.2. [FFL-3325]: https://datadoghq.atlassian.net/browse/FFL-3325?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

nccatoni

LGTM but you should probably get a review from someone from dotnet

andrewlock

The manifest changes LGTM, I'm deffering on the API as I think that's entirely up to FFE how they wish to write it (but I would def suggest updating the package version

…of github.com:DataDog/system-tests into pavlo.khrebto/FFL-2703/dotnet-agentless-configuration

test(ffe): enable .NET agentless configuration source

00aa4b2

pavlokhrebto mentioned this pull request Aug 12, 2026

[FFE] Add agentless Feature Flags configuration delivery DataDog/dd-trace-dotnet#9017

Closed

[FeatureFlags] Trigger system-tests base image build

a5f255a

pavlokhrebto added 4 commits September 22, 2026 18:30

Merge branch 'main' of github.com:DataDog/system-tests into pavlo.khr…

1defabe

…ebto/FFL-2703/dotnet-agentless-configuration

test(ffe): await SetProviderAsync and require dotnet 3.54.0

8cba818

test(ffe): fix the dotnet parametric FFE endpoints against OpenFeatur…

1f992d7

…e 2.x

Merge branch 'main' of github.com:DataDog/system-tests into pavlo.khr…

5a64d4d

…ebto/FFL-2703/dotnet-agentless-configuration

pavlokhrebto mentioned this pull request Sep 23, 2026

[FeatureFlags] Return the caller's default value on every provider error DataDog/dd-trace-dotnet#9299

Merged

pavlokhrebto added 2 commits September 23, 2026 16:27

test(ffe): mark dotnet cold-start cases as bug until FFL-3325 ships

5fe83f9

test(ffe): use long-key form for the dotnet cold-start manifest entries

29010b9

pavlokhrebto marked this pull request as ready for review September 23, 2026 15:18

pavlokhrebto requested a review from a team as a code owner September 23, 2026 15:18

Merge main into .NET agentless configuration tests

4657f3b

Refresh the existing configuration activation without folding in EVP fallback. Environment: Datadog workspace

pavlokhrebto requested a review from leoromanovsky September 24, 2026 08:20

nccatoni approved these changes Sep 24, 2026

View reviewed changes

andrewlock approved these changes Sep 24, 2026

View reviewed changes

Comment thread utils/build/docker/dotnet/parametric/ApmTestApi.csproj Outdated

Comment thread utils/build/docker/dotnet/parametric/Endpoints/FfeTestApi.cs

Comment thread utils/build/docker/dotnet/parametric/Endpoints/FfeTestApi.cs Outdated

pavlokhrebto added 2 commits September 24, 2026 17:24

test(ffe): address review on the dotnet parametric FFE endpoints

03c4320

Merge branch 'pavlo.khrebto/FFL-2703/dotnet-agentless-configuration' …

ac712c1

…of github.com:DataDog/system-tests into pavlo.khrebto/FFL-2703/dotnet-agentless-configuration

pavlokhrebto requested a review from a team as a code owner September 24, 2026 14:24

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: waiting label Sep 25, 2026

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: removed and removed mergequeue-status: waiting labels Sep 25, 2026

leoromanovsky approved these changes Sep 25, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: removed mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: removed mergequeue-status: waiting mergequeue-status: queued labels Sep 25, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 23eaea6 into main Sep 28, 2026
447 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the pavlo.khrebto/FFL-2703/dotnet-agentless-configuration branch September 28, 2026 06:52

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Sep 28, 2026

pavlokhrebto mentioned this pull request Sep 30, 2026

chore(codeowners): make FFE own its test endpoints #7861

Merged

Navigation Menu

test(ffe): enable .NET agentless configuration source - #7496

test(ffe): enable .NET agentless configuration source#7496
gh-worker-dd-mergequeue-cf854d[bot] merged 11 commits into
mainfrom
pavlo.khrebto/FFL-2703/dotnet-agentless-configuration

pavlokhrebto commented Aug 12, 2026 •

edited by leoromanovsky

Loading

Uh oh!

github-actions Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Aug 12, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

nccatoni left a comment

Uh oh!

andrewlock left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pavlokhrebto commented Aug 12, 2026 • edited by leoromanovsky Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Disabled tests

Decisions

Validation (2026-09-23 main refresh)

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Aug 12, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nccatoni left a comment

Choose a reason for hiding this comment

Uh oh!

andrewlock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pavlokhrebto commented Aug 12, 2026 •

edited by leoromanovsky

Loading

github-actions Bot commented Aug 12, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Aug 12, 2026 •

edited by datadog-official Bot

Loading