From 3210de6472c495872774c1fad0ae356999644b45 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 24 Aug 2026 14:18:27 +0200 Subject: [PATCH 01/22] test(ofrep): adopt the conformance suite in the OFREP provider OFREP is a protocol, not a vendor, so the suite needs no new infrastructure: flagd already serves the OFREP HTTP API on 8016 inside the flagd-testbed image that the flagd TCK suites use, alongside the launchpad control API on 8080. The Compose stack is therefore the same image with a different port exposed, and the whole adoption is one test class plus one dependency. Four capabilities are withheld, all traceable to the same fact: OfrepProvider implements FeatureProvider rather than extending EventProvider and overrides no lifecycle method, so it has no state, no stream, no poll loop and no initialize(). It cannot emit events (EVENTS), cannot observe the backend going away (STALE) or changing (CONFIGURATION_CHANGE), and cannot fail initialisation against a dead port (UNAVAILABLE_INIT). Each omission is justified against specific lines of the provider in the capabilities() javadoc. events.feature and lifecycle.feature are both tagged @events at feature level, so 5 scenarios are reported as skipped and 24 run. OBJECT and STRICT_NUMERIC_TYPING are both declared. Unlike the flagd provider, OFREP does not silently narrow a float to an integer: values are deserialised by a plain Jackson ObjectMapper into an untyped Object, so a JSON fraction arrives as Double and a JSON integer as Integer, and handleResolved admits a value only on an exact type.isInstance check. float-flag requested as an integer is reported as TYPE_MISMATCH with the code default rather than truncated to 0. Signed-off-by: Simon Schrottner --- providers/ofrep/pom.xml | 17 +++ .../providers/ofrep/e2e/OfrepTckTest.java | 125 ++++++++++++++++++ .../test/resources/tck/docker-compose.yaml | 17 +++ 3 files changed, 159 insertions(+) create mode 100644 providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java create mode 100644 providers/ofrep/src/test/resources/tck/docker-compose.yaml diff --git a/providers/ofrep/pom.xml b/providers/ofrep/pom.xml index 3f2fd6695e..66064d472b 100644 --- a/providers/ofrep/pom.xml +++ b/providers/ofrep/pom.xml @@ -17,6 +17,11 @@ OFREP Provider https://openfeature.dev + + + [0.0.1,) + + Rahul-Baradol @@ -81,5 +86,17 @@ 4.12.0 test + + + + dev.openfeature.contrib.tools + provider-tck + ${provider-tck.version} + test + diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java new file mode 100644 index 0000000000..b4153c2404 --- /dev/null +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -0,0 +1,125 @@ +package dev.openfeature.contrib.providers.ofrep.e2e; + +import dev.openfeature.contrib.providers.ofrep.OfrepProvider; +import dev.openfeature.contrib.providers.ofrep.OfrepProviderOptions; +import dev.openfeature.contrib.tools.providertck.AbstractProviderTckTest; +import dev.openfeature.contrib.tools.providertck.BackendEndpoint; +import dev.openfeature.contrib.tools.providertck.Capability; +import dev.openfeature.sdk.FeatureProvider; +import java.io.File; +import java.time.Duration; +import java.util.Collections; +import java.util.EnumSet; +import java.util.List; +import java.util.Set; + +/** + * Runs the OpenFeature Provider TCK against the OFREP provider. + * + *

OFREP is a protocol rather than a vendor, so the backend under test is simply something that + * speaks it. flagd does, on port {@value #OFREP_PORT}, which means this suite reuses the flagd + * testbed image and its launchpad control API unchanged — see + * {@code src/test/resources/tck/docker-compose.yaml}. + */ +public class OfrepTckTest extends AbstractProviderTckTest { + + /** The container-internal port flagd serves the OFREP HTTP API on. */ + private static final int OFREP_PORT = 8016; + + /** + * A port nothing listens on, for the initialisation-failure scenarios. + * + *

Deliberately not a port on the Compose stack: the stack must stay up for the whole suite, + * and simulated outages belong to the control API. + */ + private static final int UNAVAILABLE_PORT = 9999; + + @Override + public File composeFile() { + return new File("src/test/resources/tck/docker-compose.yaml"); + } + + @Override + public List backendPorts() { + return Collections.singletonList(OFREP_PORT); + } + + @Override + public FeatureProvider createProvider(BackendEndpoint endpoint) { + return OfrepProvider.constructProvider(OfrepProviderOptions.builder() + .baseUrl("http://" + endpoint.host() + ":" + endpoint.port(OFREP_PORT)) + .build()); + } + + /** + * {@inheritDoc} + * + *

Short timeouts on purpose: the {@code @unavailable} scenarios assert that failure is + * reported promptly. They are skipped for this provider — see + * {@link #capabilities()} — but the deadlines stay correct so that the scenarios start passing + * on their own the day the provider grows an {@code initialize()}. + */ + @Override + public FeatureProvider createUnavailableProvider() { + return OfrepProvider.constructProvider(OfrepProviderOptions.builder() + .baseUrl("http://localhost:" + UNAVAILABLE_PORT) + .connectTimeout(Duration.ofMillis(500)) + .requestTimeout(Duration.ofMillis(500)) + .build()); + } + + /** + * {@inheritDoc} + * + *

Four capabilities are withheld, and all four come from the same root cause: {@code + * OfrepProvider} is a bare {@link dev.openfeature.sdk.FeatureProvider} (OfrepProvider.java:19) + * with no lifecycle of its own. It holds no state, opens no stream, runs no poll loop and does + * not override {@code initialize()} — every evaluation is a fresh, independent HTTP POST + * (Resolver.java:54-97, OfrepApi.java:93-118). There is nothing in it that could observe a + * backend transition, let alone report one. + * + *

    + *
  • {@link Capability#EVENTS} — the class declares {@code implements FeatureProvider}, + * not {@code extends EventProvider} (OfrepProvider.java:19), so it has no {@code emit*} + * method available and calls none. The whole file contains no reference to + * {@code ProviderEvent}. The {@code PROVIDER_READY} that a client does observe is + * synthesised by the SDK on successful initialisation and would appear for + * {@code NoOpProvider} just the same; it is not the provider participating in the event + * system, so declaring the capability would be claiming behaviour the provider does not + * have — and would silently assert an untestable {@code PROVIDER_ERROR}. + *
  • {@link Capability#STALE} — requires noticing that the backend went away between + * evaluations. Nothing survives a call: {@code Resolver.resolve} builds its result purely + * from the current response and, on {@code IOException}, returns + * {@code ErrorCode.GENERAL} without recording anything (Resolver.java:93-96, + * OfrepApi.java:114-115). No state, no transition, no {@code PROVIDER_STALE}. + *
  • {@link Capability#CONFIGURATION_CHANGE} — needs a subscription to the backend. + * The only outbound call in the provider is the per-evaluation + * {@code POST /ofrep/v1/evaluate/flags/{key}} (OfrepApi.java:27, 93-109). There is no + * bulk endpoint, no ETag handling and no watch, so a change is never detected as an + * event — merely reflected by the next evaluation. + *
  • {@link Capability#UNAVAILABLE_INIT} — {@code OfrepProvider} does not override + * {@code initialize(EvaluationContext)}, so the interface default runs and initialisation + * cannot fail. {@code constructProvider} only validates its arguments; it never touches + * the network (OfrepProvider.java:38-68). A provider pointed at a dead port therefore + * reaches {@code READY}, which is the opposite of what the scenario asserts. + *
+ * + *

{@link Capability#OBJECT} and {@link Capability#STRICT_NUMERIC_TYPING} are both declared, + * and the second is worth spelling out because the flagd provider cannot declare it. + * Deserialisation goes through a plain Jackson {@code ObjectMapper} into an untyped + * {@code Object value} (OfrepResponse.java:16, OfrepApi.java:109), which maps a JSON integer to + * {@link Integer} and a JSON fraction to {@link Double}. {@code handleResolved} then admits the + * value only on an exact {@code type.isInstance(responseValue)} check and otherwise returns + * {@code TYPE_MISMATCH} with the code default (Resolver.java:183-190). Nothing anywhere widens + * or narrows between the two numeric types, so {@code float-flag} (0.5) requested as an integer + * is rejected rather than truncated to {@code 0}. The same exact-instance check is what makes + * the {@code @object} mismatch matrix work; the structured happy path passes through + * {@code resolve(Object.class, ...)}, which every non-null value satisfies, and is converted + * with {@code Value.objectToValue} (Resolver.java:125-136). + */ + @Override + public Set capabilities() { + return EnumSet.complementOf(EnumSet.of( + Capability.EVENTS, Capability.STALE, Capability.CONFIGURATION_CHANGE, Capability.UNAVAILABLE_INIT)); + } +} diff --git a/providers/ofrep/src/test/resources/tck/docker-compose.yaml b/providers/ofrep/src/test/resources/tck/docker-compose.yaml new file mode 100644 index 0000000000..93aa4cf800 --- /dev/null +++ b/providers/ofrep/src/test/resources/tck/docker-compose.yaml @@ -0,0 +1,17 @@ +# Backend stack for the OpenFeature Provider TCK, wrapping the unmodified flagd testbed image. +# +# OFREP is a vendor-neutral protocol, so the TCK does not need an OFREP-specific testbed: any +# backend that speaks OFREP and exposes the control API will do. flagd serves the OFREP HTTP API +# on 8016 alongside its own gRPC surfaces, and the testbed image already ships the "launchpad" +# control API on 8080 — the same stack the flagd provider's TCK suite uses, seeded with the same +# canonical flag set. No new image, no new control API. +# +# Note there are no host port bindings. The TCK requires dynamically mapped ports and discovers +# them after startup — a pinned host port would make the suite unrunnable in parallel and would +# collide with a developer's local flagd. +services: + backend: + image: ghcr.io/open-feature/flagd-testbed:v3.8.0 + ports: + - 8016 # flagd OFREP evaluation (HTTP) + - 8080 # launchpad control API From 02e5b3aef48e83813b0db1b9797d70f2ce37fcea Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 12:33:41 +0200 Subject: [PATCH 02/22] test(ofrep): follow the base's capability model The base renamed the containerised base class, retired @strict-numeric-typing in favour of @numeric-coercion, and now refuses a declaration that names a reserved or not-applicable capability. EnumSet.complementOf swept up @large-integers, @targeting and @caching, so the suite would have stopped at startup; Capability.declarableExcept leaves those out on its own. LIFECYCLE is withheld as well, which the complement had quietly claimed since the capability appeared: OfrepProvider has no initialize(), so the readiness scenario passed exactly as it does for NoOpProvider, and the new shutdown scenarios gated by the same tag are skipped rather than passed vacuously. NUMERIC_COERCION is withheld because the tag now requires the lossless direction too, and Resolver.handleResolved admits a value only on an exact type.isInstance check: integer-flag requested as a float arrives from Jackson as an Integer and is refused. Strict typing in both directions is a choice under the suite's model, not a defect, so no KnownDeviation goes with it. Read from the source, not from a run. The class Javadoc records that flagd-testbed v3.8.0 serves none of the six new canonical flags, so the four untagged scenarios that read them fail FLAG_NOT_FOUND until the testbed is updated; that is the stack's gap, not the provider's, so it is documented rather than declared. Signed-off-by: Simon Schrottner --- .../providers/ofrep/e2e/OfrepTckTest.java | 97 +++++++++++++------ 1 file changed, 66 insertions(+), 31 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index b4153c2404..6952c6ed4d 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -2,14 +2,13 @@ import dev.openfeature.contrib.providers.ofrep.OfrepProvider; import dev.openfeature.contrib.providers.ofrep.OfrepProviderOptions; -import dev.openfeature.contrib.tools.providertck.AbstractProviderTckTest; import dev.openfeature.contrib.tools.providertck.BackendEndpoint; import dev.openfeature.contrib.tools.providertck.Capability; +import dev.openfeature.contrib.tools.providertck.ContainerizedProviderTckTest; import dev.openfeature.sdk.FeatureProvider; import java.io.File; import java.time.Duration; import java.util.Collections; -import java.util.EnumSet; import java.util.List; import java.util.Set; @@ -20,8 +19,19 @@ * speaks it. flagd does, on port {@value #OFREP_PORT}, which means this suite reuses the flagd * testbed image and its launchpad control API unchanged — see * {@code src/test/resources/tck/docker-compose.yaml}. + * + *

The testbed does not yet serve the whole canonical flag set. The suite's + * assets added six flags — {@code large-integer-flag}, {@code huge-integer-flag}, + * {@code integral-float-flag}, {@code false-flag}, {@code zero-flag} and + * {@code empty-string-flag} — and {@code flagd-testbed} v3.8.0 serves none of them (its + * {@code zero-flags.json} keys are {@code integer-zero-flag} and so on, not the canonical names). + * Until open-feature/flagd-testbed is updated, the four untagged scenarios that read them — the + * three falsy-value rows and the 32-bit precision scenario — fail with {@code FLAG_NOT_FOUND}. + * That is a gap in the stack, not in the provider, so it is recorded here rather than declared as + * a {@code KnownDeviation}: a deviation says the provider is wrong, and the provider was never + * given the flag to get wrong. */ -public class OfrepTckTest extends AbstractProviderTckTest { +public class OfrepTckTest extends ContainerizedProviderTckTest { /** The container-internal port flagd serves the OFREP HTTP API on. */ private static final int OFREP_PORT = 8016; @@ -71,22 +81,30 @@ public FeatureProvider createUnavailableProvider() { /** * {@inheritDoc} * - *

Four capabilities are withheld, and all four come from the same root cause: {@code - * OfrepProvider} is a bare {@link dev.openfeature.sdk.FeatureProvider} (OfrepProvider.java:19) - * with no lifecycle of its own. It holds no state, opens no stream, runs no poll loop and does - * not override {@code initialize()} — every evaluation is a fresh, independent HTTP POST + *

Five capabilities are withheld for the same root cause: {@code OfrepProvider} is a bare + * {@link dev.openfeature.sdk.FeatureProvider} (OfrepProvider.java:19) with no lifecycle of its + * own. It holds no state, opens no stream, runs no poll loop and does not override + * {@code initialize()} — every evaluation is a fresh, independent HTTP POST * (Resolver.java:54-97, OfrepApi.java:93-118). There is nothing in it that could observe a * backend transition, let alone report one. * *

    + *
  • {@link Capability#LIFECYCLE} — there is no initialisation to observe. + * {@code constructProvider} only validates its arguments and never touches the network + * (OfrepProvider.java:38-68), and the interface default {@code initialize()} does + * nothing, so the {@code PROVIDER_READY} a client sees is the SDK's, and the readiness + * scenario would pass exactly as it does for {@code NoOpProvider}. The tag gates the + * shutdown scenarios too — shutting down twice, initialising again after a shutdown, and + * shutting down promptly against a dead backend — and the second of those is one this + * provider could not pass honestly either: {@code shutdown()} terminates the executor the + * HTTP client runs on (OfrepProvider.java:91-108) and, with no {@code initialize()}, + * nothing ever recreates it. All of them are skipped rather than passed vacuously. *
  • {@link Capability#EVENTS} — the class declares {@code implements FeatureProvider}, * not {@code extends EventProvider} (OfrepProvider.java:19), so it has no {@code emit*} * method available and calls none. The whole file contains no reference to - * {@code ProviderEvent}. The {@code PROVIDER_READY} that a client does observe is - * synthesised by the SDK on successful initialisation and would appear for - * {@code NoOpProvider} just the same; it is not the provider participating in the event - * system, so declaring the capability would be claiming behaviour the provider does not - * have — and would silently assert an untestable {@code PROVIDER_ERROR}. + * {@code ProviderEvent}. Declaring the capability would be claiming behaviour the + * provider does not have — and would silently assert an untestable + * {@code PROVIDER_ERROR}. *
  • {@link Capability#STALE} — requires noticing that the backend went away between * evaluations. Nothing survives a call: {@code Resolver.resolve} builds its result purely * from the current response and, on {@code IOException}, returns @@ -97,29 +115,46 @@ public FeatureProvider createUnavailableProvider() { * {@code POST /ofrep/v1/evaluate/flags/{key}} (OfrepApi.java:27, 93-109). There is no * bulk endpoint, no ETag handling and no watch, so a change is never detected as an * event — merely reflected by the next evaluation. - *
  • {@link Capability#UNAVAILABLE_INIT} — {@code OfrepProvider} does not override - * {@code initialize(EvaluationContext)}, so the interface default runs and initialisation - * cannot fail. {@code constructProvider} only validates its arguments; it never touches - * the network (OfrepProvider.java:38-68). A provider pointed at a dead port therefore - * reaches {@code READY}, which is the opposite of what the scenario asserts. + *
  • {@link Capability#UNAVAILABLE_INIT} — with no {@code initialize()} of its own, + * initialisation cannot fail. A provider pointed at a dead port therefore reaches + * {@code READY}, which is the opposite of what the scenario asserts. *
* - *

{@link Capability#OBJECT} and {@link Capability#STRICT_NUMERIC_TYPING} are both declared, - * and the second is worth spelling out because the flagd provider cannot declare it. - * Deserialisation goes through a plain Jackson {@code ObjectMapper} into an untyped - * {@code Object value} (OfrepResponse.java:16, OfrepApi.java:109), which maps a JSON integer to - * {@link Integer} and a JSON fraction to {@link Double}. {@code handleResolved} then admits the - * value only on an exact {@code type.isInstance(responseValue)} check and otherwise returns - * {@code TYPE_MISMATCH} with the code default (Resolver.java:183-190). Nothing anywhere widens - * or narrows between the two numeric types, so {@code float-flag} (0.5) requested as an integer - * is rejected rather than truncated to {@code 0}. The same exact-instance check is what makes - * the {@code @object} mismatch matrix work; the structured happy path passes through - * {@code resolve(Object.class, ...)}, which every non-null value satisfies, and is converted - * with {@code Value.objectToValue} (Resolver.java:125-136). + *

{@link Capability#NUMERIC_COERCION} is withheld for a different reason: the + * provider keeps the two numeric types strictly apart, in both directions. Deserialisation goes + * through a plain Jackson {@code ObjectMapper} into an untyped {@code Object value} + * (OfrepResponse.java:16, OfrepApi.java:109), which maps a JSON integer to {@link Integer} and + * a JSON fraction to {@link Double}, and {@code handleResolved} then admits the value only on + * an exact {@code type.isInstance(responseValue)} check, otherwise returning + * {@code TYPE_MISMATCH} with the code default (Resolver.java:183-191). That satisfies the lossy + * half of the rule — {@code float-flag} (0.5) requested as an integer is rejected rather than + * truncated to {@code 0} — but the tag also requires the lossless half, and there the same + * check refuses: {@code integer-flag} (10) requested as a float arrives as an {@code Integer}, + * which {@code Double.class.isInstance} rejects, so "an integer requested as a float is widened + * without loss" cannot pass. Nothing in the provider widens or narrows a number. Strict typing + * in both directions is a choice the TCK lets a provider make — the SDK's own + * {@code InMemoryProvider} makes it — not a defect, so there is no {@code KnownDeviation} to go + * with it. This is read from the source rather than from a run; a run that shows the widening + * scenario passing means the deserialiser changed, and the declaration should follow it. + * + *

Two things this leaves in place. {@link Capability#OBJECT} is declared: the same + * exact-instance check is what makes the {@code @object} mismatch matrix work, and the + * structured happy path passes through {@code resolve(Object.class, ...)}, which every non-null + * value satisfies, and is converted with {@code Value.objectToValue} (Resolver.java:125-136). + * And {@link Capability#LARGE_INTEGERS} is absent without being named here: + * {@link Capability#declarableExcept} leaves out the not-applicable and reserved tags on its + * own, which is why it is used instead of {@code EnumSet.complementOf} — the complement would + * claim {@code @large-integers}, {@code @targeting} and {@code @caching} on the way past, and + * the suite refuses such a declaration at startup. */ @Override public Set capabilities() { - return EnumSet.complementOf(EnumSet.of( - Capability.EVENTS, Capability.STALE, Capability.CONFIGURATION_CHANGE, Capability.UNAVAILABLE_INIT)); + return Capability.declarableExcept( + Capability.LIFECYCLE, + Capability.EVENTS, + Capability.STALE, + Capability.CONFIGURATION_CHANGE, + Capability.UNAVAILABLE_INIT, + Capability.NUMERIC_COERCION); } } From 695e9bee062c1ceab51fb692f45df41322f29d47 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 12:35:46 +0200 Subject: [PATCH 03/22] docs(ofrep): note in the Compose header which canonical flags the testbed lacks The same note the flagd adoption carries, next to the image tag it is about: flagd-testbed v3.8.0 serves none of the six flags the bumped assets added, so the untagged scenarios that read them fail FLAG_NOT_FOUND until the testbed is updated, and the tag here is what to bump when it is. Signed-off-by: Simon Schrottner --- providers/ofrep/src/test/resources/tck/docker-compose.yaml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/providers/ofrep/src/test/resources/tck/docker-compose.yaml b/providers/ofrep/src/test/resources/tck/docker-compose.yaml index 93aa4cf800..11306f3840 100644 --- a/providers/ofrep/src/test/resources/tck/docker-compose.yaml +++ b/providers/ofrep/src/test/resources/tck/docker-compose.yaml @@ -4,7 +4,12 @@ # backend that speaks OFREP and exposes the control API will do. flagd serves the OFREP HTTP API # on 8016 alongside its own gRPC surfaces, and the testbed image already ships the "launchpad" # control API on 8080 — the same stack the flagd provider's TCK suite uses, seeded with the same -# canonical flag set. No new image, no new control API. +# flag set. No new image, no new control API. +# +# That flag set is not yet the whole canonical one: v3.8.0 has none of large-integer-flag, +# huge-integer-flag, integral-float-flag, false-flag, zero-flag or empty-string-flag, so the +# untagged scenarios that read them fail FLAG_NOT_FOUND until open-feature/flagd-testbed catches +# up. Bump the tag here once it does. # # Note there are no host port bindings. The TCK requires dynamically mapped ports and discovers # them after startup — a pinned host port would make the suite unrunnable in parallel and would From 6a99d55887061818ed2b1a0352da6f0fed1b4bf1 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 14:41:23 +0200 Subject: [PATCH 04/22] test(ofrep): follow the flag rename and settle @numeric-coercion by evidence The falsy-flag rename in the base removes three failures here without touching the provider: flagd-testbed already served boolean-zero-flag, integer-zero-flag and string-zero-flag with zero/non-zero variants, and spec ba002ce8 moved the canonical names onto the testbed's rather than the other way round. The suite goes from four untagged failures to one. The notes now say what is actually missing -- large-integer-flag, and only that, because huge-integer-flag sits behind @large-integers and integral-float-flag behind @numeric-coercion, neither of which this provider declares. @numeric-coercion stays withheld, now on stated evidence rather than a shorter argument. Go's OFREP provider declares it and this one does not, which looked like an unexamined declaration on one side; it is not. handleResolved admits a value only on an exact type.isInstance check (Resolver.java:183-191) with no integral check and no round trip anywhere in the path, so of the tag's three scenarios this provider passes one: the lossy case is right for the wrong reason, and both lossless cases fail, integer-flag requested as a float and integral-float-flag requested as an integer alike. Declaring it would turn two scenarios red -- three, counting that the testbed cannot serve integral-float-flag at all. Go coerces and this does not; the two declarations describe two implementations, not one protocol, which is possible precisely because OFREP is JSON and integer-ness is the provider's decision. No KnownDeviation accompanies it, and that is a decision rather than silence. Appendix F is explicit that this is the one capability the specification does not define, that its rule is borrowed from flagd's ADR, and that "a provider that behaves differently is not violating the specification" -- having retracted an earlier draft that called non-declaration an admission of a known bug. A deviation entry would assert a defect the spec says is not one: the opposite mistake from a vacuous declaration, in the same currency. What the entry does record is that the reasoning is source-derived, that no unit test pins the numeric pair, and what a run would have to show for the declaration to change. Signed-off-by: Simon Schrottner --- .../providers/ofrep/e2e/OfrepTckTest.java | 80 ++++++++++++++----- .../test/resources/tck/docker-compose.yaml | 7 +- 2 files changed, 64 insertions(+), 23 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index 6952c6ed4d..d4a9dc9b25 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -20,16 +20,23 @@ * testbed image and its launchpad control API unchanged — see * {@code src/test/resources/tck/docker-compose.yaml}. * - *

The testbed does not yet serve the whole canonical flag set. The suite's - * assets added six flags — {@code large-integer-flag}, {@code huge-integer-flag}, - * {@code integral-float-flag}, {@code false-flag}, {@code zero-flag} and - * {@code empty-string-flag} — and {@code flagd-testbed} v3.8.0 serves none of them (its - * {@code zero-flags.json} keys are {@code integer-zero-flag} and so on, not the canonical names). - * Until open-feature/flagd-testbed is updated, the four untagged scenarios that read them — the - * three falsy-value rows and the 32-bit precision scenario — fail with {@code FLAG_NOT_FOUND}. - * That is a gap in the stack, not in the provider, so it is recorded here rather than declared as - * a {@code KnownDeviation}: a deviation says the provider is wrong, and the provider was never - * given the flag to get wrong. + *

The testbed does not yet serve the whole canonical flag set. Three flags the + * suite's assets added are absent from {@code flagd-testbed} v3.8.0: {@code large-integer-flag}, + * {@code huge-integer-flag} and {@code integral-float-flag}. Only the first is reached — + * {@code huge-integer-flag} is asked for solely under {@code @large-integers}, which is not + * applicable in Java, and {@code integral-float-flag} solely under {@code @numeric-coercion}, which + * is withheld below — so exactly one untagged scenario, the 32-bit precision one, fails with + * {@code FLAG_NOT_FOUND} until open-feature/flagd-testbed#392 lands. + * + *

The three falsy flags used to fail the same way and no longer do. The testbed's + * {@code zero-flags.json} already served {@code boolean-zero-flag}, {@code integer-zero-flag} and + * {@code string-zero-flag} with {@code zero}/{@code non-zero} variants, while the canonical set + * called them {@code false-flag}, {@code zero-flag} and {@code empty-string-flag}; spec ba002ce8 + * renamed the canonical flags to the testbed's names rather than the other way round. + * + *

A missing flag is a gap in the stack, not in the provider, so it is recorded here rather than + * declared as a {@code KnownDeviation}: a deviation says the provider is wrong, and the provider was + * never given the flag to get wrong. */ public class OfrepTckTest extends ContainerizedProviderTckTest { @@ -126,16 +133,49 @@ public FeatureProvider createUnavailableProvider() { * (OfrepResponse.java:16, OfrepApi.java:109), which maps a JSON integer to {@link Integer} and * a JSON fraction to {@link Double}, and {@code handleResolved} then admits the value only on * an exact {@code type.isInstance(responseValue)} check, otherwise returning - * {@code TYPE_MISMATCH} with the code default (Resolver.java:183-191). That satisfies the lossy - * half of the rule — {@code float-flag} (0.5) requested as an integer is rejected rather than - * truncated to {@code 0} — but the tag also requires the lossless half, and there the same - * check refuses: {@code integer-flag} (10) requested as a float arrives as an {@code Integer}, - * which {@code Double.class.isInstance} rejects, so "an integer requested as a float is widened - * without loss" cannot pass. Nothing in the provider widens or narrows a number. Strict typing - * in both directions is a choice the TCK lets a provider make — the SDK's own - * {@code InMemoryProvider} makes it — not a defect, so there is no {@code KnownDeviation} to go - * with it. This is read from the source rather than from a run; a run that shows the widening - * scenario passing means the deserialiser changed, and the declaration should follow it. + * {@code TYPE_MISMATCH} with the code default (Resolver.java:183-191). There is no integral + * check and no round trip anywhere in that path — nothing in the provider widens or narrows a + * number. + * + *

The tag's rule is that lossless coercion must succeed and lossy coercion must return + * {@code TYPE_MISMATCH}, and the three scenarios carrying it test all three cases; a provider + * declaring the tag must pass all three. This provider passes one. The lossy case is right for + * the wrong reason — {@code float-flag} (0.5) requested as an integer is rejected rather than + * truncated to {@code 0}, because it is a {@link Double} and not because 0.5 is fractional — + * and both lossless cases fail on the same exact-instance check. {@code integer-flag} (10) + * requested as a float arrives as an {@code Integer}, which {@code Double.class.isInstance} + * rejects, so "an integer requested as a float is widened without loss" cannot pass; + * {@code integral-float-flag} (10.0) requested as an integer arrives as a {@link Double}, + * which {@code Integer.class.isInstance} rejects, so "an integral float requested as an + * integer is coerced without loss" cannot pass either. Declaring the tag would turn two of its + * three scenarios red. + * + *

No {@code KnownDeviation} accompanies it, and that is deliberate rather than silence. + * Appendix F is explicit that {@code @numeric-coercion} is the one capability the specification + * does not define: OpenFeature has a single {@code number} type, the rule this tag is tested + * against is borrowed from flagd's numeric-coercion ADR, and "a provider that behaves + * differently is not violating the specification". The appendix says so having retracted + * an earlier draft that called non-declaration "an admission of a known bug". Strict typing in + * both directions is therefore a legitimate choice — the SDK's own {@code InMemoryProvider} + * makes it — and a deviation entry would assert a defect the spec says is not one. That is the + * opposite mistake from a vacuous declaration, but a mistake in the same currency. + * + *

Go's OFREP provider declares the tag, and that is not an inconsistency to reconcile away: + * it coerces through an integral check and this one does not, so the two declarations describe + * two implementations rather than one protocol. OFREP being JSON is what makes the difference + * possible — one wire number type, so integer-ness is the provider's decision, not the + * payload's. Declaring the tag here to match Go would also run the + * {@code integral-float-flag} scenario, which the testbed cannot serve, so it would fail twice + * over: once for the provider and once for the stack. + * + *

All of that is read from the source, because a withheld tag means the scenarios are + * skipped and a run cannot confirm it; the three are reported as skipped with this reason on + * every run, which is the observable that the declaration is being honoured rather than the + * behaviour behind it. {@code OfrepProviderTest} asserts the exact-instance check itself, but + * only across Boolean and String (OfrepProviderTest.java:71, 338-339) — no unit test pins the + * numeric pair, which is why the reasoning above cites {@code handleResolved} directly. A run in + * which either lossless scenario passes means {@code handleResolved} or the deserialiser + * changed, and this declaration should follow it. * *

Two things this leaves in place. {@link Capability#OBJECT} is declared: the same * exact-instance check is what makes the {@code @object} mismatch matrix work, and the diff --git a/providers/ofrep/src/test/resources/tck/docker-compose.yaml b/providers/ofrep/src/test/resources/tck/docker-compose.yaml index 11306f3840..e238373882 100644 --- a/providers/ofrep/src/test/resources/tck/docker-compose.yaml +++ b/providers/ofrep/src/test/resources/tck/docker-compose.yaml @@ -7,9 +7,10 @@ # flag set. No new image, no new control API. # # That flag set is not yet the whole canonical one: v3.8.0 has none of large-integer-flag, -# huge-integer-flag, integral-float-flag, false-flag, zero-flag or empty-string-flag, so the -# untagged scenarios that read them fail FLAG_NOT_FOUND until open-feature/flagd-testbed catches -# up. Bump the tag here once it does. +# huge-integer-flag or integral-float-flag. Only large-integer-flag is reached by a scenario that +# runs here -- the other two sit behind @large-integers and @numeric-coercion, neither of which +# this provider declares -- so one untagged scenario fails FLAG_NOT_FOUND until +# open-feature/flagd-testbed#392 lands. Bump the tag then. # # Note there are no host port bindings. The TCK requires dynamically mapped ports and discovers # them after startup — a pinned host port would make the suite unrunnable in parallel and would From b54244651394fcfd9f877b7291f6a6aa6443153c Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 15:12:34 +0200 Subject: [PATCH 05/22] test(ofrep): withhold @reinitialization, which the provider cannot offer The base gates "A provider that was shut down can be initialized again" on @reinitialization, because Requirement 2.5.2 permits reuse after shutdown rather than requiring it. declarableExcept(...) hands the new tag out by default, so a provider that cannot be restarted has to say so or it publishes a claim nothing examined -- the one failure mode this declaration exists to prevent. OfrepProvider cannot be restarted. shutdown() terminates the executor the HTTP client runs on (OfrepProvider.java:90-108) and the class overrides no initialize(), so nothing recreates it. That is the choice 2.5.2 offers rather than a defect, and no KnownDeviation accompanies the omission. It changes no result on its own: the scenario carries @lifecycle too, which this provider already withholds because it has no initialisation to observe, so the skip was happening either way. The declaration is what stops being a half-truth. The @lifecycle bullet no longer counts re-initialisation among the scenarios that tag gates, since it does not any more. Signed-off-by: Simon Schrottner --- .../providers/ofrep/e2e/OfrepTckTest.java | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index d4a9dc9b25..20efecce08 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -88,7 +88,7 @@ public FeatureProvider createUnavailableProvider() { /** * {@inheritDoc} * - *

Five capabilities are withheld for the same root cause: {@code OfrepProvider} is a bare + *

Six capabilities are withheld for the same root cause: {@code OfrepProvider} is a bare * {@link dev.openfeature.sdk.FeatureProvider} (OfrepProvider.java:19) with no lifecycle of its * own. It holds no state, opens no stream, runs no poll loop and does not override * {@code initialize()} — every evaluation is a fresh, independent HTTP POST @@ -101,11 +101,18 @@ public FeatureProvider createUnavailableProvider() { * (OfrepProvider.java:38-68), and the interface default {@code initialize()} does * nothing, so the {@code PROVIDER_READY} a client sees is the SDK's, and the readiness * scenario would pass exactly as it does for {@code NoOpProvider}. The tag gates the - * shutdown scenarios too — shutting down twice, initialising again after a shutdown, and - * shutting down promptly against a dead backend — and the second of those is one this - * provider could not pass honestly either: {@code shutdown()} terminates the executor the - * HTTP client runs on (OfrepProvider.java:91-108) and, with no {@code initialize()}, - * nothing ever recreates it. All of them are skipped rather than passed vacuously. + * shutdown scenarios too — shutting down twice, and shutting down promptly against a dead + * backend — and they are skipped rather than passed vacuously. + *

  • {@link Capability#REINITIALIZATION} — omitted, and independently of the omission + * above. {@code shutdown()} terminates the executor the HTTP client runs on + * (OfrepProvider.java:90-108) and, with no {@code initialize()} of its own, nothing ever + * recreates it, so a shut-down {@code OfrepProvider} cannot be started again. Requirement + * 2.5.2 permits exactly that — a provider SHOULD revert to its uninitialized + * state and "some providers MAY allow reinitialization from this state" — so this + * is a choice the specification offers and not a defect to declare. It is named here + * rather than left to the {@code @lifecycle} skip because + * {@link Capability#declarableExcept} would otherwise have claimed it, and a claim nothing + * examined is exactly what the declaration exists to prevent. *
  • {@link Capability#EVENTS} — the class declares {@code implements FeatureProvider}, * not {@code extends EventProvider} (OfrepProvider.java:19), so it has no {@code emit*} * method available and calls none. The whole file contains no reference to @@ -191,6 +198,7 @@ public FeatureProvider createUnavailableProvider() { public Set capabilities() { return Capability.declarableExcept( Capability.LIFECYCLE, + Capability.REINITIALIZATION, Capability.EVENTS, Capability.STALE, Capability.CONFIGURATION_CHANGE, From b0bc748cebf371697b6427b175257991e59067c3 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 18:54:16 +0200 Subject: [PATCH 06/22] test(ofrep): withhold @large-integers explicitly The TCK no longer sets @large-integers apart as "not applicable in Java", so declarableExcept(...) no longer leaves it out on its own and this suite names it alongside the six capabilities it already withholds. The run is unchanged: the scenario was skipped before and is skipped now, because the SDK's integer accessor is a 32-bit Integer and 2^53 - 1 has no room in it. That is a fact about the SDK rather than about OFREP, which is why it is recorded once in Appendix F and needs no knownDeviations entry here -- unlike the numeric-coercion gap, which is the provider's own. Signed-off-by: Simon Schrottner --- .../providers/ofrep/e2e/OfrepTckTest.java | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index 20efecce08..4da2b2332d 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -188,11 +188,16 @@ public FeatureProvider createUnavailableProvider() { * exact-instance check is what makes the {@code @object} mismatch matrix work, and the * structured happy path passes through {@code resolve(Object.class, ...)}, which every non-null * value satisfies, and is converted with {@code Value.objectToValue} (Resolver.java:125-136). - * And {@link Capability#LARGE_INTEGERS} is absent without being named here: - * {@link Capability#declarableExcept} leaves out the not-applicable and reserved tags on its - * own, which is why it is used instead of {@code EnumSet.complementOf} — the complement would - * claim {@code @large-integers}, {@code @targeting} and {@code @caching} on the way past, and - * the suite refuses such a declaration at startup. + * And {@link Capability#LARGE_INTEGERS} is withheld, as every Java provider withholds it, for a + * reason that is not about OFREP at all: the tag asks for 2^53 − 1 and + * {@code Client.getIntegerDetails} is a 32-bit {@code Integer} with no room for it. The limit is + * the SDK's, it is recorded once in Appendix F rather than in each run, and the scenario is + * skipped for an undeclared capability like any other. It is named here for the same reason + * {@code REINITIALIZATION} is — {@link Capability#declarableExcept} would otherwise claim it. + * + *

    {@code declarableExcept} rather than {@code EnumSet.complementOf}, which would also claim + * {@code @targeting} and {@code @caching} on the way past; the suite refuses such a declaration + * at startup. */ @Override public Set capabilities() { @@ -203,6 +208,7 @@ public Set capabilities() { Capability.STALE, Capability.CONFIGURATION_CHANGE, Capability.UNAVAILABLE_INIT, - Capability.NUMERIC_COERCION); + Capability.NUMERIC_COERCION, + Capability.LARGE_INTEGERS); } } From d0f2b06f23982c4238a599b73ca20bae3ceaa126 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 08:38:03 +0200 Subject: [PATCH 07/22] docs(ofrep): say what the new tags gate here, and follow the testbed gap @variants and @targeting arrived with the base's submodule bump and declarableExcept picks both up, so the declaration grew by two capabilities while the javadoc that argues every withheld tag at length said nothing about either. One run against flagd-testbed v3.8.0 through the OFREP HTTP API: 38 pass, 12 skipped, 2 failed. @targeting is worth more here than its name suggests. The provider sends the evaluation context in the request body and the backend evaluates the rule, so targeting-key-flag's three scenarios are the only ones in the canonical set that would notice a context dropped on the way out -- every other flag resolves the same way with or without one. All three pass. @variants passes seven of its eight rows. The eighth asks for large-integer-flag's max-int32 and is answered with no variant, because v3.8.0 does not serve that flag at all -- the gap the Compose header already records for the untagged precision scenario, now reached twice rather than once, so "one untagged scenario" is a scenario short. Withholding the tag would hide both failures behind a claim about the provider that the run does not support. The complementOf paragraph called @targeting reserved alongside @caching. It is declarable now; the argument survives with one tag, because what makes the form of the call right is not how large the reserved set happens to be. Signed-off-by: Simon Schrottner --- .../providers/ofrep/e2e/OfrepTckTest.java | 16 ++++++++++++++-- .../src/test/resources/tck/docker-compose.yaml | 13 +++++++++---- 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index 4da2b2332d..9c8085249f 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -195,9 +195,21 @@ public FeatureProvider createUnavailableProvider() { * skipped for an undeclared capability like any other. It is named here for the same reason * {@code REINITIALIZATION} is — {@link Capability#declarableExcept} would otherwise claim it. * + *

    {@link Capability#VARIANTS} and {@link Capability#TARGETING} are declared, and unlike the + * withheld tags above both are confirmed by a run rather than read from the source. The OFREP + * response carries {@code variant} alongside {@code value} and {@code reason}, and the provider + * passes it through, so seven of the {@code @variants} outline's eight rows pass; the eighth asks + * for {@code large-integer-flag}'s {@code max-int32} and gets no variant because testbed v3.8.0 + * does not serve that flag — the gap already noted next to the image tag, not a second defect. + * {@code @targeting} matters more here than the capability's name suggests: the provider sends the + * evaluation context in the request body and the backend evaluates the rule, so the three + * {@code targeting-key-flag} scenarios are the only ones in the canonical set that would notice a + * context dropped on the way out. All three pass. + * *

    {@code declarableExcept} rather than {@code EnumSet.complementOf}, which would also claim - * {@code @targeting} and {@code @caching} on the way past; the suite refuses such a declaration - * at startup. + * {@code @caching} on the way past; the suite refuses such a declaration at startup. It claimed + * {@code @targeting} the same way until that tag gated something — the reserved set shrinks as + * the vocabulary fills up, which is an argument for the form of the call rather than against it. */ @Override public Set capabilities() { diff --git a/providers/ofrep/src/test/resources/tck/docker-compose.yaml b/providers/ofrep/src/test/resources/tck/docker-compose.yaml index e238373882..f55f4a5790 100644 --- a/providers/ofrep/src/test/resources/tck/docker-compose.yaml +++ b/providers/ofrep/src/test/resources/tck/docker-compose.yaml @@ -7,10 +7,15 @@ # flag set. No new image, no new control API. # # That flag set is not yet the whole canonical one: v3.8.0 has none of large-integer-flag, -# huge-integer-flag or integral-float-flag. Only large-integer-flag is reached by a scenario that -# runs here -- the other two sit behind @large-integers and @numeric-coercion, neither of which -# this provider declares -- so one untagged scenario fails FLAG_NOT_FOUND until -# open-feature/flagd-testbed#392 lands. Bump the tag then. +# huge-integer-flag or integral-float-flag. Only large-integer-flag is reached by scenarios that +# run here -- the other two sit behind @large-integers and @numeric-coercion, neither of which +# this provider declares -- and it is reached twice: the untagged precision scenario, which gets +# the code default instead of 2147483647, and the @variants row that asks for its "max-int32" +# variant and is answered with none. Both fail until open-feature/flagd-testbed#392 lands. Bump +# the tag then. +# +# targeting-key-flag is served, which is why @targeting needs no bump: its three scenarios pass +# on this image. # # Note there are no host port bindings. The TCK requires dynamically mapped ports and discovers # them after startup — a pinned host port would make the suite unrunnable in parallel and would From 6d296d6c81265ba0d2d58b6179741326959ed192 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 12:46:21 +0200 Subject: [PATCH 08/22] test(ofrep): withhold @disabled-flags, and say why it is a defect Measured, not assumed, and the measurement overturned the expectation it was made to confirm. Declared, all four rows of the new outline fail: 56 scenarios, 38 passing, 12 skipped, 6 failed, the four extra failures beyond the two testbed ones all reading "expected: null but was: FLAG_NOT_FOUND". The value assertion passes, because the provider returns the code default on an error -- right answer, wrong reason. The expectation was that this is architectural: the caller's default never leaves the process, so a backend cannot return it. That is not what the backend does. flagd's OFREP endpoint answers a disabled flag with 200, reason DISABLED and no value member, probed directly against the pinned v3.8.0 image. OFREP's evaluationSuccess requires only key and reason; value is not required, because one shape a success may take is codeDefaultFlag -- "This schema has no value property. The provider must use the code default value when processing this response." DISABLED is in the reason enum. The response is well-formed and says exactly what the scenario asserts. So this is a provider gap against a MUST in the protocol the provider implements, and it is wider than the rows that found it: every codeDefaultFlag response reaches the application as FLAG_NOT_FOUND, so an application checking the error code sees a failure on an evaluation that succeeded. handleResolved treats a null value as an absent flag and discards the reason it parsed a field earlier. The tag stays withheld, because it fails and a conformance run must not pass it. But it carries a KnownDeviation, which is the opposite call from @numeric-coercion next to it, and the difference is where the rule lives: numeric coercion is Appendix F borrowing flagd's ADR with no specification behind it, whereas codeDefaultFlag is normative OFREP. A bare omission would read as the same kind of choice, and this is not a choice. Untracked for now; delete both once handleResolved honours a value-less success. Nothing is owed upstream. The four disabled-* flags are served by the image already pinned, which the Compose header now records so the failures are not mistaken for another testbed gap. Signed-off-by: Simon Schrottner --- .../providers/ofrep/e2e/OfrepTckTest.java | 83 +++++++++++++++++++ .../test/resources/tck/docker-compose.yaml | 6 +- 2 files changed, 88 insertions(+), 1 deletion(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index 9c8085249f..0c26143b76 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -5,6 +5,7 @@ import dev.openfeature.contrib.tools.providertck.BackendEndpoint; import dev.openfeature.contrib.tools.providertck.Capability; import dev.openfeature.contrib.tools.providertck.ContainerizedProviderTckTest; +import dev.openfeature.contrib.tools.providertck.KnownDeviation; import dev.openfeature.sdk.FeatureProvider; import java.io.File; import java.time.Duration; @@ -206,6 +207,53 @@ public FeatureProvider createUnavailableProvider() { * {@code targeting-key-flag} scenarios are the only ones in the canonical set that would notice a * context dropped on the way out. All three pass. * + *

    {@link Capability#DISABLED_FLAGS} is withheld, and unlike every other withheld tag + * above it was measured. Declared, all four rows of its outline fail, and they fail on + * the error code rather than on the value: the run reported 56 scenarios, 38 passing, 12 skipped + * and 6 failed, the two extra failures beyond the testbed pair above being + * {@code expected: null but was: FLAG_NOT_FOUND} on each of the four rows. The value assertion + * passes, because the provider returns the caller's default on an error — right answer, wrong + * reason. + * + *

    What the backend actually answers is worth writing down, because the shape of the gap is not + * what one would guess. flagd's OFREP endpoint does not 404 a disabled flag: it answers + * {@code 200} with {@code {"key":"disabled-boolean-flag","reason":"DISABLED","metadata":{}}} and + * no {@code value} member, where an enabled flag comes back as + * {@code {"value":true,"key":"boolean-flag","reason":"STATIC","variant":"on","metadata":{}}} + * (probed directly against the pinned v3.8.0 image on port 8016). {@code handleResolved} reaches + * its {@code responseValue == null} branch and returns the code default with + * {@code FLAG_NOT_FOUND} and "No value returned for flag", discarding the {@code reason} it + * parsed one field earlier (Resolver.java:169-181, OfrepResponse.java:19). + * + *

    That response is well-formed OFREP, and it says what to do. The obvious + * reading of this failure — the caller's default never leaves the process, so a provider whose + * backend decides cannot hold the tag — does not survive the protocol. OFREP's + * {@code evaluationSuccess} requires only {@code key} and {@code reason}; {@code value} is + * not required, because one of the member schemas a success may be is + * {@code codeDefaultFlag}, described as "A flag evaluation that defers to the code default + * value ... This schema has no value property. The provider must use the code default value when + * processing this response." {@code DISABLED} is in the {@code reason} enum alongside + * {@code STATIC}. flagd is answering in exactly that shape, and the answer means what the + * scenario asserts. + * + *

    So this is a provider gap rather than an architectural limit, and it is wider than the four + * rows that found it: every {@code codeDefaultFlag} response is reported to the + * application as {@code FLAG_NOT_FOUND}, so an application checking the error code sees a failure + * on an evaluation that succeeded. The capability is gated because the answer can depend on + * architecture, and a provider that only ever received a value would have a real case — but this + * provider receives a response that is explicit about deferring, parses the {@code reason} that + * accompanies it, and then discards both. + * + *

    It is therefore recorded as a + * {@link dev.openfeature.contrib.tools.providertck.KnownDeviation} rather than left as a bare + * omission — see {@link #knownDeviations()}. That is the opposite call from + * {@code @numeric-coercion} above, and the difference is where the rule lives: numeric coercion + * is a rule Appendix F borrowed from flagd's ADR and that no specification states, whereas + * {@code codeDefaultFlag} is a {@code MUST} in the protocol this provider implements. Withholding + * the tag keeps the four rows honest — skipped, not passed — and the deviation is what says the + * omission is a bug rather than a choice. Delete both once {@code handleResolved} honours a + * value-less success. + * *

    {@code declarableExcept} rather than {@code EnumSet.complementOf}, which would also claim * {@code @caching} on the way past; the suite refuses such a declaration at startup. It claimed * {@code @targeting} the same way until that tag gated something — the reserved set shrinks as @@ -219,8 +267,43 @@ public Set capabilities() { Capability.EVENTS, Capability.STALE, Capability.CONFIGURATION_CHANGE, + Capability.DISABLED_FLAGS, Capability.UNAVAILABLE_INIT, Capability.NUMERIC_COERCION, Capability.LARGE_INTEGERS); } + + /** + * {@inheritDoc} + * + *

    One entry, for the withheld {@link Capability#DISABLED_FLAGS}, and it is the only omission + * in {@link #capabilities()} that is a defect rather than a fact about the provider's shape. + * Every other withheld tag describes something {@code OfrepProvider} has no machinery for — no + * initialisation, no events, no state between calls — or a rule no specification states. This one + * describes a response the provider receives, understands well enough to parse, and then answers + * wrongly. + * + *

    Untracked, because there is no issue to point at yet. Naming it anyway is the whole point of + * the mechanism: in the results a capability withheld by choice and one withheld because it is + * broken are the same absence, and only the provider author can say which happened. + * + *

    The summary names the response shape rather than the scenario, because the scenario is only + * where it was noticed. {@code codeDefaultFlag} is not specific to disabled flags — any backend + * deferring to the code default for any reason gets the same {@code FLAG_NOT_FOUND} — so a reader + * comparing providers needs the general statement, not the one outline that caught it. + */ + @Override + public List knownDeviations() { + return Collections.singletonList(KnownDeviation.untracked( + Capability.DISABLED_FLAGS, + "A value-less OFREP evaluation success is reported to the application as " + + "FLAG_NOT_FOUND. OFREP's evaluationSuccess requires only key and reason; a " + + "success matching codeDefaultFlag carries no value and means the provider " + + "MUST use the code default. handleResolved treats a null value as an " + + "absent flag instead (Resolver.java:174-181), discarding the reason it " + + "parsed, so the value returned is right and the error code is not. Found " + + "by the four @disabled-flags rows -- flagd answers a DISABLED flag with " + + "200, reason DISABLED and no value -- but it affects every codeDefaultFlag " + + "response, not only disabled flags.")); + } } diff --git a/providers/ofrep/src/test/resources/tck/docker-compose.yaml b/providers/ofrep/src/test/resources/tck/docker-compose.yaml index f55f4a5790..536d58ad5d 100644 --- a/providers/ofrep/src/test/resources/tck/docker-compose.yaml +++ b/providers/ofrep/src/test/resources/tck/docker-compose.yaml @@ -15,7 +15,11 @@ # the tag then. # # targeting-key-flag is served, which is why @targeting needs no bump: its three scenarios pass -# on this image. +# on this image. The four disabled-* flags behind @disabled-flags are served too, from this +# image's own flags/disabled-flags.json -- so the four failures that tag produced here are the +# provider's and not the stack's, and the tag is withheld in OfrepTckTest with the reason. flagd +# answers a disabled flag with 200, reason DISABLED and no value, which is what OFREP calls a +# codeDefaultFlag; nothing is missing from this image. # # Note there are no host port bindings. The TCK requires dynamically mapped ports and discovers # them after startup — a pinned host port would make the suite unrunnable in parallel and would From 0f5aaad612c9b4b53bc57058edd7adf919f67ce8 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 15:21:49 +0200 Subject: [PATCH 09/22] test(ofrep): follow the provider-tck -> tck rename Import-path and coordinate churn only: the artifact is dev.openfeature.contrib.tools:tck, the version range starts at 0.1.0, and the four imports come from dev.openfeature.contrib.tools.tck. No behavioural change, and no change to what is declared or withheld. Signed-off-by: Simon Schrottner --- providers/ofrep/pom.xml | 8 ++++---- .../contrib/providers/ofrep/e2e/OfrepTckTest.java | 10 +++++----- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/providers/ofrep/pom.xml b/providers/ofrep/pom.xml index 66064d472b..818d618c12 100644 --- a/providers/ofrep/pom.xml +++ b/providers/ofrep/pom.xml @@ -18,8 +18,8 @@ https://openfeature.dev - - [0.0.1,) + + [0.1.0,) @@ -94,8 +94,8 @@ --> dev.openfeature.contrib.tools - provider-tck - ${provider-tck.version} + tck + ${tck.version} test diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index 0c26143b76..da7abedf9c 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -2,10 +2,10 @@ import dev.openfeature.contrib.providers.ofrep.OfrepProvider; import dev.openfeature.contrib.providers.ofrep.OfrepProviderOptions; -import dev.openfeature.contrib.tools.providertck.BackendEndpoint; -import dev.openfeature.contrib.tools.providertck.Capability; -import dev.openfeature.contrib.tools.providertck.ContainerizedProviderTckTest; -import dev.openfeature.contrib.tools.providertck.KnownDeviation; +import dev.openfeature.contrib.tools.tck.BackendEndpoint; +import dev.openfeature.contrib.tools.tck.Capability; +import dev.openfeature.contrib.tools.tck.ContainerizedProviderTckTest; +import dev.openfeature.contrib.tools.tck.KnownDeviation; import dev.openfeature.sdk.FeatureProvider; import java.io.File; import java.time.Duration; @@ -245,7 +245,7 @@ public FeatureProvider createUnavailableProvider() { * accompanies it, and then discards both. * *

    It is therefore recorded as a - * {@link dev.openfeature.contrib.tools.providertck.KnownDeviation} rather than left as a bare + * {@link dev.openfeature.contrib.tools.tck.KnownDeviation} rather than left as a bare * omission — see {@link #knownDeviations()}. That is the opposite call from * {@code @numeric-coercion} above, and the difference is where the rule lives: numeric coercion * is a rule Appendix F borrowed from flagd's ADR and that no specification states, whereas From 127eba4566cd1493171f184c10a8274075585f09 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 17:43:47 +0200 Subject: [PATCH 10/22] build(ofrep): gate the TCK suite behind Docker, and bring own testcontainers Two fixes to this module's POM, both of which were latent. testExclusions was missing. The parent POM feeds ${testExclusions} to Surefire but defines no default, so a module that wants the Docker gate has to declare the property and a module that forgets one runs a Docker-dependent suite in every job. providers/flagd declares it; this module did not, so OfrepTckTest started a Compose stack during plain `mvn verify` and ended "Tests run: 56, Failures: 2" on the recorded testbed gaps. So the ofrep PR was red in CI while the flagd PR was green for the opposite reason - one ran a suite it should gate, the other gated a suite nobody ran. Adding the exclusion makes the policy uniform: `mvn -Pcodequality -pl providers/ofrep -am verify` is now BUILD SUCCESS with 18 tests and no Compose stack. Testcontainers is now declared here. The TCK moved it to provided/optional so that the majority of adopters, which have no backend and never load ContainerizedProviderTckTest, stop resolving it; a containerised adopter brings its own instead. A previous pass asserted that both adoptions in this repository already did - that was true of providers/flagd and false of this module, which was relying on the transitive edge. Stated plainly because the assertion was wrong, not because the fix is interesting. Verified with the suite actually run: 56 scenarios, 38 passed, 16 skipped for withheld capabilities, 2 failed on the flags flagd-testbed v3.8.0 does not serve - unchanged, so the dependency now resolves from here rather than through the TCK. Signed-off-by: Simon Schrottner --- providers/ofrep/pom.xml | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/providers/ofrep/pom.xml b/providers/ofrep/pom.xml index 818d618c12..74c3a6853b 100644 --- a/providers/ofrep/pom.xml +++ b/providers/ofrep/pom.xml @@ -20,6 +20,15 @@ [0.1.0,) + 2.0.4 + + **/e2e/*.java @@ -88,9 +97,9 @@ dev.openfeature.contrib.tools @@ -98,5 +107,19 @@ ${tck.version} test + + + + org.testcontainers + testcontainers + ${testcontainers.version} + test + From d3d2a7bc6487b09efe635550260175c30a4df285 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 17:43:47 +0200 Subject: [PATCH 11/22] docs(ofrep): say that the TCK suite is Docker-gated and hand-run The other half of adding testExclusions. An exclusion nobody writes down is indistinguishable from an oversight, and that is not a hypothetical here: this module's exclusion was missing precisely because nothing said the convention existed. So the README states the policy and its reasoning - Docker-gated, excluded from every job that exists, run by hand by a maintainer before merging a change to resolution or error behaviour, with the result quoted in the pull request. A scheduled or path-filtered workflow was considered and declined. It also gives the command and says which failures are expected, so a reader can tell a regression from the recorded testbed gaps, and notes that several capabilities are withheld because OFREP puts the decision on the server rather than in the provider - a fact about the protocol, not a defect. Signed-off-by: Simon Schrottner --- providers/ofrep/README.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/providers/ofrep/README.md b/providers/ofrep/README.md index 6a4688d055..089cf76e1a 100644 --- a/providers/ofrep/README.md +++ b/providers/ofrep/README.md @@ -72,3 +72,37 @@ Given below are the supported configurations: | proxySelector | ProxySelector | ProxySelector.getDefault() | The proxy selector used by HTTP Client. | executor | Executor | Thread Pool of size 5 | The executor used by HTTP Client. + +## Provider conformance (TCK) + +This provider adopts the [OpenFeature Provider TCK](../../tools/tck/README.md) as a single suite, +`OfrepTckTest`. Read that class before changing it: it records which capabilities are declared, +which are withheld and why — several are withheld because OFREP puts the decision on the server +rather than in the provider, which is a fact about the protocol and not a defect — and every known +deviation. + +Because OFREP is a protocol rather than a vendor, the backend under test is simply something that +speaks it. The suite reuses the unmodified `flagd-testbed` image and its launchpad control API. + +**The suite is Docker-gated and excluded from the default build**, via +`**/e2e/*.java` in this module's POM. That property is the +repository's convention for a Docker-dependent suite, fed to Surefire by the parent POM; the parent +defines no default, so each module that wants the gate declares it. This module did not, which meant +`mvn verify` started a Compose stack and the suite ran — and failed — in every job that touched +`providers/ofrep`. The exclusion is the fix, and this paragraph is the other half of it: an +exclusion nobody writes down is indistinguishable from an oversight. + +The consequence is that **no CI job runs the suite**, so a maintainer runs it by hand before merging +a change to the provider's resolution or error behaviour, and quotes the result in the pull request. +A scheduled or path-filtered workflow was considered and declined: a suite whose red is diagnosed by +whoever happens to read the notification is worse than one whose red is diagnosed by the person who +caused it. + +```bash +mvn -pl providers/ofrep -am -DtestExclusions= -Dtest=OfrepTckTest \ + -Dsurefire.failIfNoSpecifiedTests=false test +``` + +The suite is currently **expected to fail** on the failures enumerated in `OfrepTckTest`, which come +from flags the pinned testbed image does not serve (open-feature/flagd-testbed#392). Anything else +is a regression. From 43c0366b06b57ee4d1cf2590d215c5e3b524e94b Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 23:36:30 +0200 Subject: [PATCH 12/22] docs(ofrep): point at Appendix F for the CI-exclusion reasoning Appendix F now carries "Running the suite in CI", promoted there because the same reasoning restated in four adoption READMEs is where it drifted. So this section keeps the mechanism and the local record -- that this module had no testExclusions at all until it was added, which is the appendix's "exclusion nobody wrote down" -- and links to the appendix for the argument rather than paraphrasing it. Also states that no profile in this module touches the property, resolved rather than read: mvn -Pe2e -pl providers/ofrep help:evaluate -Dexpression=testExclusions -> **/e2e/*.java mvn -pl providers/ofrep help:evaluate -Dexpression=testExclusions -> **/e2e/*.java Unlike providers/flagd, which needs an e2e profile for its legacy suites and therefore narrows the pattern instead of clearing it. Signed-off-by: Simon Schrottner --- providers/ofrep/README.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/providers/ofrep/README.md b/providers/ofrep/README.md index 089cf76e1a..95450870f8 100644 --- a/providers/ofrep/README.md +++ b/providers/ofrep/README.md @@ -89,8 +89,20 @@ speaks it. The suite reuses the unmodified `flagd-testbed` image and its launchp repository's convention for a Docker-dependent suite, fed to Surefire by the parent POM; the parent defines no default, so each module that wants the gate declares it. This module did not, which meant `mvn verify` started a Compose stack and the suite ran — and failed — in every job that touched -`providers/ofrep`. The exclusion is the fix, and this paragraph is the other half of it: an -exclusion nobody writes down is indistinguishable from an oversight. +`providers/ofrep`. That is the second of the two mistakes +[Appendix F: Running the suite in CI](https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md#running-the-suite-in-ci) +names, and the appendix has the reasoning for the whole policy; the exclusion is the fix and this +paragraph is the other half of it. + +This module has no profile that touches the property, so both spellings resolve the same way — +checked rather than read, because that is the appendix's other warning: + +```bash +mvn -Pe2e -pl providers/ofrep help:evaluate -Dexpression=testExclusions -DforceStdout +``` + +The exclusion is Surefire's and not the compiler's, so the suite still builds against the harness in +every job. The consequence is that **no CI job runs the suite**, so a maintainer runs it by hand before merging a change to the provider's resolution or error behaviour, and quotes the result in the pull request. From 1ae903d2cd63fbd03432b8a55facb2cbbdcaa782 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 10:47:21 +0200 Subject: [PATCH 13/22] test(ofrep): record what @standard-reasons does and does not claim here This suite declares everything declarable except nine capabilities, so it picked @standard-reasons up by default the moment the TCK gained it. Measured before it was written down. Eight of reason.feature's nine scenarios run and pass: STATIC for the four rule-less flags, ERROR beside FLAG_NOT_FOUND and TYPE_MISMATCH, and -- because @targeting is declared here -- TARGETING_MATCH and DEFAULT either side of targeting-key-flag's rule. The ninth carries @disabled-flags as well and is skipped for that omission, which is the right outcome rather than a second report of the same gap: what this provider does wrong with a value-less success is already stated once, in the withheld capability and its KnownDeviation, and a reason it never reaches is not more evidence of it. So the tag means "the standard vocabulary, over the responses this provider actually completes", and a reader sees the withheld @disabled-flags beside it and can tell which scenario went unasked. A clean run is 65 scenarios, 46 passing, 17 skipped and 2 failing, up from 56, 38 and 16. The two failures are the same testbed gaps as before. Also records something this pass measured rather than introduced: the suite is intermittently flaky. About half of the runs carry one or two extra failures where an evaluation comes back as the code default, or as FLAG_NOT_FOUND where TYPE_MISMATCH was expected, or with reason ERROR where a resolution was expected. The victim moves between errors.feature, evaluation.feature and reason.feature, so it is not a property of any assertion. Eight runs were measured, five at this revision and three at ccdb8879, and the old pin produced a seven-failure run and a two-failure run from the same tree -- so this predates the reason scenarios and is not caused by them. That is the flagd-testbed readiness window of open-feature/flagd-testbed#394 reaching a provider that holds nothing between calls, so every evaluation races the stack afresh. Recorded in the class javadoc and the README with an explicit instruction not to cover it with a settle after control calls, because a suite that sleeps instead of holding the control API to its promise stops being able to detect when the promise breaks. Signed-off-by: Simon Schrottner --- providers/ofrep/README.md | 10 ++++- .../providers/ofrep/e2e/OfrepTckTest.java | 38 ++++++++++++++++++- 2 files changed, 45 insertions(+), 3 deletions(-) diff --git a/providers/ofrep/README.md b/providers/ofrep/README.md index 95450870f8..7ee7343609 100644 --- a/providers/ofrep/README.md +++ b/providers/ofrep/README.md @@ -116,5 +116,11 @@ mvn -pl providers/ofrep -am -DtestExclusions= -Dtest=OfrepTckTest \ ``` The suite is currently **expected to fail** on the failures enumerated in `OfrepTckTest`, which come -from flags the pinned testbed image does not serve (open-feature/flagd-testbed#392). Anything else -is a regression. +from flags the pinned testbed image does not serve (open-feature/flagd-testbed#392). A clean run is +65 scenarios, 46 passing, 17 skipped and 2 failing. + +It is also **intermittently flaky**, and `OfrepTckTest`'s class javadoc says why: about half the runs +carry one or two extra failures where an evaluation comes back as the code default or as +`FLAG_NOT_FOUND`, on a scenario that moves from run to run. That is the testbed readiness window of +open-feature/flagd-testbed#394, not a provider defect and not something to cover with a sleep. Repeat +the run before treating an extra failure as a regression; anything that reproduces is one. diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index da7abedf9c..2c66e1a54f 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -38,6 +38,25 @@ *

    A missing flag is a gap in the stack, not in the provider, so it is recorded here rather than * declared as a {@code KnownDeviation}: a deviation says the provider is wrong, and the provider was * never given the flag to get wrong. + * + *

    A clean run is 65 scenarios, 46 passing, 17 skipped and those 2 failing. + * + *

    The suite is intermittently flaky, and the flakiness is the backend's. Roughly + * half of the runs measured carry one or two additional failures on top of those two, and + * they all have the same shape: an evaluation that should have resolved comes back as the code + * default, or as {@code FLAG_NOT_FOUND} where {@code TYPE_MISMATCH} was expected, or with reason + * {@code ERROR} where a resolution was expected. Which scenario is hit moves from run to run — + * {@code errors.feature}, {@code evaluation.feature} and {@code reason.feature} have each been the + * victim — so it is not a property of any assertion. + * + *

    It is not new with the reason scenarios, and that was checked rather than assumed: five runs at + * this revision and three at {@code ccdb8879} before it, with the old pin producing a run of seven + * failures and a run of two from the same tree. It is the flagd-testbed readiness window that + * open-feature/flagd-testbed#394 exists to close — a control endpoint returning before the backend + * serves the new state — reaching a provider that holds nothing between calls, so every evaluation + * races the stack afresh. Do not add a settle after control calls to cover it: a + * suite that sleeps instead of holding the control API to its promise stops being able to detect + * when the promise breaks, which is the whole argument of that issue. */ public class OfrepTckTest extends ContainerizedProviderTckTest { @@ -207,9 +226,26 @@ public FeatureProvider createUnavailableProvider() { * {@code targeting-key-flag} scenarios are the only ones in the canonical set that would notice a * context dropped on the way out. All three pass. * + *

    {@link Capability#STANDARD_REASONS} is declared, and it arrived by the + * {@code declarableExcept} default rather than by a decision, so it was measured before being + * written down. Eight of {@code reason.feature}'s nine scenarios run and pass: {@code STATIC} for + * the four rule-less flags, {@code ERROR} beside {@code FLAG_NOT_FOUND} and + * {@code TYPE_MISMATCH}, and — because {@code @targeting} is declared here — {@code + * TARGETING_MATCH} and {@code DEFAULT} either side of {@code targeting-key-flag}'s rule. The + * ninth carries {@code @disabled-flags} as well and is skipped for that omission, which is the + * right outcome and not a second report of the same gap: what is wrong with this provider's + * handling of a disabled flag is already said once, below and in {@link #knownDeviations()}, and + * a reason it never reaches is not more evidence of it. + * + *

    Worth noting what the declaration does not claim. The reason for a disabled flag is + * the one standard reason this provider is not held to, so the tag here means "the standard + * vocabulary, over the responses this provider actually completes". A consumer reading the report + * sees the withheld {@code @disabled-flags} beside it and can tell which scenario went unasked. + * *

    {@link Capability#DISABLED_FLAGS} is withheld, and unlike every other withheld tag * above it was measured. Declared, all four rows of its outline fail, and they fail on - * the error code rather than on the value: the run reported 56 scenarios, 38 passing, 12 skipped + * the error code rather than on the value: the run — at spec revision {@code ccdb8879}, when the + * suite was 56 scenarios rather than 65 — reported 38 passing, 12 skipped * and 6 failed, the two extra failures beyond the testbed pair above being * {@code expected: null but was: FLAG_NOT_FOUND} on each of the four rows. The value assertion * passes, because the provider returns the caller's default on an error — right answer, wrong From 0632b79633da662a2776b755e0125367324f8259 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 15:53:49 +0200 Subject: [PATCH 14/22] test(ofrep): drop the one omission that was never about OFREP This suite's declarableExcept list is long, and every name in it is something this provider genuinely cannot do -- no lifecycle to observe, no events, no connection to lose, a backend that decides so the caller's default never leaves the process, a deserialiser that keeps the numeric types strictly apart. @large-integers was the odd one out: it said nothing about OFREP at all, only that Client.getIntegerDetails is a 32-bit Integer. The TCK refuses it centrally now, so it is gone from the list and from the paragraph that explained it. That matters more here than in the flagd suite, because here it was one name among nine and a reader had no way to tell which of the nine described the provider. Now all of them do. Measured on the pinned testbed image: 65 scenarios, 46 passed, 17 skipped, 2 failed -- a clean run, identical to the previous pass, since this changes the reason for a skip rather than the count. The stream now carries four "provider does not declare capability" reasons and one "the Java SDK cannot express capability LARGE_INTEGERS", which is the distinction a reader of the report needs. No KnownDeviation changes: the existing DISABLED_FLAGS entry is unaffected, and a capability no Java provider can be asked was never a deviation to record. Signed-off-by: Simon Schrottner --- .../providers/ofrep/e2e/OfrepTckTest.java | 23 ++++++++++--------- .../test/resources/tck/docker-compose.yaml | 5 ++-- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index 2c66e1a54f..183d57d252 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -24,9 +24,9 @@ *

    The testbed does not yet serve the whole canonical flag set. Three flags the * suite's assets added are absent from {@code flagd-testbed} v3.8.0: {@code large-integer-flag}, * {@code huge-integer-flag} and {@code integral-float-flag}. Only the first is reached — - * {@code huge-integer-flag} is asked for solely under {@code @large-integers}, which is not - * applicable in Java, and {@code integral-float-flag} solely under {@code @numeric-coercion}, which - * is withheld below — so exactly one untagged scenario, the 32-bit precision one, fails with + * {@code huge-integer-flag} is asked for solely under {@code @large-integers}, which no Java + * provider can be asked, and {@code integral-float-flag} solely under {@code @numeric-coercion}, + * which is withheld below — so exactly one untagged scenario, the 32-bit precision one, fails with * {@code FLAG_NOT_FOUND} until open-feature/flagd-testbed#392 lands. * *

    The three falsy flags used to fail the same way and no longer do. The testbed's @@ -208,12 +208,14 @@ public FeatureProvider createUnavailableProvider() { * exact-instance check is what makes the {@code @object} mismatch matrix work, and the * structured happy path passes through {@code resolve(Object.class, ...)}, which every non-null * value satisfies, and is converted with {@code Value.objectToValue} (Resolver.java:125-136). - * And {@link Capability#LARGE_INTEGERS} is withheld, as every Java provider withholds it, for a - * reason that is not about OFREP at all: the tag asks for 2^53 − 1 and - * {@code Client.getIntegerDetails} is a 32-bit {@code Integer} with no room for it. The limit is - * the SDK's, it is recorded once in Appendix F rather than in each run, and the scenario is - * skipped for an undeclared capability like any other. It is named here for the same reason - * {@code REINITIALIZATION} is — {@link Capability#declarableExcept} would otherwise claim it. + * And {@link Capability#LARGE_INTEGERS} is no longer in the list below, which is the one + * omission here that says nothing about OFREP. The tag asks for 2^53 − 1 and + * {@code Client.getIntegerDetails} is a 32-bit {@code Integer} with no room for it, so the limit + * is the SDK's; the TCK refuses the capability outright now rather than asking every Java + * adoption to remember, and {@link Capability#declarableExcept} no longer offers it. Its + * scenario is still skipped, with a reason naming the accessor rather than this provider — which + * matters more here than elsewhere, because every other name in that list is something + * this provider genuinely cannot do, and a reader should not have to guess which is which. * *

    {@link Capability#VARIANTS} and {@link Capability#TARGETING} are declared, and unlike the * withheld tags above both are confirmed by a run rather than read from the source. The OFREP @@ -305,8 +307,7 @@ public Set capabilities() { Capability.CONFIGURATION_CHANGE, Capability.DISABLED_FLAGS, Capability.UNAVAILABLE_INIT, - Capability.NUMERIC_COERCION, - Capability.LARGE_INTEGERS); + Capability.NUMERIC_COERCION); } /** diff --git a/providers/ofrep/src/test/resources/tck/docker-compose.yaml b/providers/ofrep/src/test/resources/tck/docker-compose.yaml index 536d58ad5d..d9725063f5 100644 --- a/providers/ofrep/src/test/resources/tck/docker-compose.yaml +++ b/providers/ofrep/src/test/resources/tck/docker-compose.yaml @@ -8,8 +8,9 @@ # # That flag set is not yet the whole canonical one: v3.8.0 has none of large-integer-flag, # huge-integer-flag or integral-float-flag. Only large-integer-flag is reached by scenarios that -# run here -- the other two sit behind @large-integers and @numeric-coercion, neither of which -# this provider declares -- and it is reached twice: the untagged precision scenario, which gets +# run here -- the other two sit behind @numeric-coercion, which this provider withholds, and +# @large-integers, which no Java provider can be asked because Client.getIntegerDetails is a +# 32-bit Integer -- and it is reached twice: the untagged precision scenario, which gets # the code default instead of 2147483647, and the @variants row that asks for its "max-int32" # variant and is answered with none. Both fail until open-feature/flagd-testbed#392 lands. Bump # the tag then. From ab99a8ec41e1bba7fadcc44419f7267654c4dc00 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 19:38:45 +0200 Subject: [PATCH 15/22] docs(ofrep): separate the reason a tag is withheld from the notes beside it Two capability omissions in this suite are argued in the javadoc, and Appendix F's declaring rules have since drawn lines through both arguments. @numeric-coercion is withheld because the provider does not coerce -- it type-checks in both directions -- and no requirement says it must, which is the question that comes first and settles this one. The paragraph that follows also observed that declaring the tag would run a scenario the pinned testbed cannot serve. That is now explicitly not a reason: a scenario failing for a missing fixture is not a provider defect, so it argues for nothing. It stays as a note about what such a run would look like, marked as one. The appendix's scenario-level rule does not reach this omission either, and the javadoc says why rather than leaving a reader to wonder: that rule is about whether a question is askable, and all three of these are -- what it does not decide is whether an answer is owed. @disabled-flags is the opposite case and the javadoc now says so plainly. The provider receives the codeDefaultFlag response, parses its reason and answers with the wrong error code, so it does attempt the behaviour, and the settled guidance prefers declaring the tag and letting the four rows fail with this same deviation beside them. That is a change of results rather than of prose and is not made here; recording it is what stops it being lost. Signed-off-by: Simon Schrottner --- .../providers/ofrep/e2e/OfrepTckTest.java | 28 +++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java index 183d57d252..b9b463aeda 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java @@ -192,8 +192,19 @@ public FeatureProvider createUnavailableProvider() { * two implementations rather than one protocol. OFREP being JSON is what makes the difference * possible — one wire number type, so integer-ness is the provider's decision, not the * payload's. Declaring the tag here to match Go would also run the - * {@code integral-float-flag} scenario, which the testbed cannot serve, so it would fail twice - * over: once for the provider and once for the stack. + * {@code integral-float-flag} scenario, which the testbed cannot serve — a note about what such + * a run would look like, and deliberately not a reason. Appendix F's declaring rules + * have since said why it cannot be one: a scenario that fails because the backend cannot serve + * its fixture is not a provider defect, so the gap is no argument for withholding anything. The + * reason is the paragraph above and only that paragraph. + * + *

    Nor does the appendix's scenario-level rule reach this omission, which is worth saying + * because it reads at first as though it should. That rule — once a provider is attempting a + * capability, declare it when at least one scenario gating it can be put to the provider — is + * about whether a question is askable, and all three of these are. What comes first is + * whether an answer is owed, and no requirement says this one is: the provider does not coerce, + * the specification permits that, and withholding is the honest report. Taking the second rule + * without the first would manufacture two failures out of a permitted choice. * *

    All of that is read from the source, because a withheld tag means the scenarios are * skipped and a run cannot confirm it; the three are reported as skipped with this reason on @@ -292,6 +303,19 @@ public FeatureProvider createUnavailableProvider() { * omission is a bug rather than a choice. Delete both once {@code handleResolved} honours a * value-less success. * + *

    This is the one declaration on this branch that the settled guidance would shape + * differently, and it is recorded here rather than quietly left. + * {@link dev.openfeature.contrib.tools.tck.KnownDeviation} prefers the declared-and-failing shape + * and confines the withheld-and-skipped one to a provider that cannot attempt the behaviour at + * all. This provider does attempt it: it receives the {@code codeDefaultFlag} response, parses + * the {@code reason} that accompanies it, and then answers with the wrong error code — measured, + * four rows, failing on the code and not on the value. By that reading the honest report is to + * declare {@code @disabled-flags}, let the four rows fail, and keep this same deviation beside + * them, exactly as the flagd adoption does for {@code @numeric-coercion}. The flip is a change of + * results rather than of prose, so it is not made in the documentation pass that noticed it; it + * costs four failures in place of four skips and nothing else, and the deviation's text needs no + * change when it happens. + * *

    {@code declarableExcept} rather than {@code EnumSet.complementOf}, which would also claim * {@code @caching} on the way past; the suite refuses such a declaration at startup. It claimed * {@code @targeting} the same way until that tag gated something — the reserved set shrinks as From e0ff64f054052a9caf87f306bb0d7fd81dcf4371 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 20:56:49 +0200 Subject: [PATCH 16/22] test(ofrep): run the conformance suite from a step of its own The same shape providers/flagd just gained, and for the same reason Appendix F gives: a conformance run carries failures by design wherever OfrepTckTest declares a knownDeviation, so a signal it shares with a suite that is expected green ends with somebody silencing the informative half. The `tck` profile clears this module's **/e2e/*.java exclusion and narrows Surefire's includes to **/e2e/*TckTest.java in the same breath, so `mvn -Ptck -pl providers/ofrep test` runs OfrepTckTest and nothing else - 65 scenarios, 46 passing, 17 skipped and 2 failing, which is the clean run the README already documents. This module had no profiles at all, so the README's claim that no profile touched testExclusions was true and now is not; it names the one that does. Nothing activates `tck` in CI. The README says not to add `-am` to that run, and why: it pulls tools/tck into the reactor and runs its 246 tests first, so a failure there comes out as a `-Ptck` failure. A one-off `install` is what `-am` was there for. Signed-off-by: Simon Schrottner --- providers/ofrep/README.md | 23 +++++++++++++++++---- providers/ofrep/pom.xml | 42 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 4 deletions(-) diff --git a/providers/ofrep/README.md b/providers/ofrep/README.md index 7ee7343609..5d3182b45d 100644 --- a/providers/ofrep/README.md +++ b/providers/ofrep/README.md @@ -94,8 +94,9 @@ defines no default, so each module that wants the gate declares it. This module names, and the appendix has the reasoning for the whole policy; the exclusion is the fix and this paragraph is the other half of it. -This module has no profile that touches the property, so both spellings resolve the same way — -checked rather than read, because that is the appendix's other warning: +Exactly one profile touches the property — `tck`, below — and `e2e`, which the repository's CI +activates on every push, does not exist in this module at all. Checked rather than read, because +that is the appendix's other warning: ```bash mvn -Pe2e -pl providers/ofrep help:evaluate -Dexpression=testExclusions -DforceStdout @@ -104,6 +105,14 @@ mvn -Pe2e -pl providers/ofrep help:evaluate -Dexpression=testExclusions -DforceS The exclusion is Surefire's and not the compiler's, so the suite still builds against the harness in every job. +**The conformance suite has a profile of its own, `tck`.** That is the separation the appendix asks +for, and the reason is what a red build *says* rather than how long it takes: a conformance run +carries failures by design, wherever `OfrepTckTest` declares a `knownDeviation`, so a signal shared +with a suite that is expected green ends with somebody silencing the informative half. The profile +clears the exclusion and narrows Surefire's includes to `**/e2e/*TckTest.java` in the same breath, +so it runs the conformance suite and nothing else — not the module's unit tests. Nothing activates +it in CI. + The consequence is that **no CI job runs the suite**, so a maintainer runs it by hand before merging a change to the provider's resolution or error behaviour, and quotes the result in the pull request. A scheduled or path-filtered workflow was considered and declined: a suite whose red is diagnosed by @@ -111,10 +120,16 @@ whoever happens to read the notification is worse than one whose red is diagnose caused it. ```bash -mvn -pl providers/ofrep -am -DtestExclusions= -Dtest=OfrepTckTest \ - -Dsurefire.failIfNoSpecifiedTests=false test +# once, if tools/tck is not in your local repository yet +mvn -pl tools/tck -am -DskipTests install + +mvn -Ptck -pl providers/ofrep test ``` +**Do not add `-am` to the run itself**: it pulls `tools/tck` into the reactor and runs its 246 tests +before the first scenario, so a failure there comes out as a `-Ptck` failure — the signal-mixing this +step exists to prevent, reintroduced by a flag. The separate `install` is what `-am` was there for. + The suite is currently **expected to fail** on the failures enumerated in `OfrepTckTest`, which come from flags the pinned testbed image does not serve (open-feature/flagd-testbed#392). A clean run is 65 scenarios, 46 passing, 17 skipped and 2 failing. diff --git a/providers/ofrep/pom.xml b/providers/ofrep/pom.xml index 74c3a6853b..01a0c548ef 100644 --- a/providers/ofrep/pom.xml +++ b/providers/ofrep/pom.xml @@ -122,4 +122,46 @@ test + + + + + tck + + + + + + + + org.apache.maven.plugins + maven-surefire-plugin + + + **/e2e/*TckTest.java + + + + + + + From f00973180766312847d871ece33c9ede5704caf7 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 22:26:21 +0200 Subject: [PATCH 17/22] test(ofrep): give the conformance adoption a package of its own The suite lived in an `e2e` package it was the only member of, and every selector reached it by filename: `**/e2e/*.java` to exclude it, `**/e2e/*TckTest.java` to run it. It now lives in `src/test/java/.../ofrep/tck/` and the selectors name that directory. The `e2e` package is gone -- there was never an end-to-end suite in this module for the adoption to sit beside, which is the clearest form of the argument: it was filed under a category it was not. The class drops what the directory now says. `OfrepTckTest` in package ...ofrep.tck said "tck" twice; it is `OfrepTest`. The `*Test` suffix stays, because Surefire's default includes need it -- that is a different thing from the selector being removed. The name a run is filed under is unchanged: configuration() strips the JUnit suffix either way, so it was and remains "ofrep". Both halves of the `tck` profile are still needed, for the reason they always were: the include alone leaves the exclusion in force and runs nothing, and dropping the exclusion alone runs this module's unit tests alongside the suite. testExclusions is still a Surefire and not a compiler exclusion, so the suite still compiles in the default build. The tally does not move: 65 scenarios, 46 passed, 17 skipped, 2 failed. Same scenarios, same results, a different directory. The profile comment documenting the run had kept `-am` on it, which the README already warns against; it says the command that works now. Signed-off-by: Simon Schrottner --- providers/ofrep/README.md | 22 +++++++----- providers/ofrep/pom.xml | 34 +++++++++++-------- .../OfrepTckTest.java => tck/OfrepTest.java} | 4 +-- .../test/resources/tck/docker-compose.yaml | 2 +- 4 files changed, 37 insertions(+), 25 deletions(-) rename providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/{e2e/OfrepTckTest.java => tck/OfrepTest.java} (99%) diff --git a/providers/ofrep/README.md b/providers/ofrep/README.md index 5d3182b45d..8a79acc374 100644 --- a/providers/ofrep/README.md +++ b/providers/ofrep/README.md @@ -76,7 +76,12 @@ Given below are the supported configurations: ## Provider conformance (TCK) This provider adopts the [OpenFeature Provider TCK](../../tools/tck/README.md) as a single suite, -`OfrepTckTest`. Read that class before changing it: it records which capabilities are declared, +`OfrepTest`, in a source directory of its own: `src/test/java/.../ofrep/tck/`. It is the module's +only Docker-dependent test package, and having it be a directory rather than a filename convention +is what lets every selector below name a place instead of a pattern. In a package called `tck` the +class needs no further label; the fully-qualified name still carries everything. + +Read that class before changing it: it records which capabilities are declared, which are withheld and why — several are withheld because OFREP puts the decision on the server rather than in the provider, which is a fact about the protocol and not a defect — and every known deviation. @@ -85,7 +90,7 @@ Because OFREP is a protocol rather than a vendor, the backend under test is simp speaks it. The suite reuses the unmodified `flagd-testbed` image and its launchpad control API. **The suite is Docker-gated and excluded from the default build**, via -`**/e2e/*.java` in this module's POM. That property is the +`**/tck/*.java` in this module's POM. That property is the repository's convention for a Docker-dependent suite, fed to Surefire by the parent POM; the parent defines no default, so each module that wants the gate declares it. This module did not, which meant `mvn verify` started a Compose stack and the suite ran — and failed — in every job that touched @@ -107,11 +112,12 @@ every job. **The conformance suite has a profile of its own, `tck`.** That is the separation the appendix asks for, and the reason is what a red build *says* rather than how long it takes: a conformance run -carries failures by design, wherever `OfrepTckTest` declares a `knownDeviation`, so a signal shared +carries failures by design, wherever `OfrepTest` declares a `knownDeviation`, so a signal shared with a suite that is expected green ends with somebody silencing the informative half. The profile -clears the exclusion and narrows Surefire's includes to `**/e2e/*TckTest.java` in the same breath, -so it runs the conformance suite and nothing else — not the module's unit tests. Nothing activates -it in CI. +drops the `tck` directory from the exclusion and narrows Surefire's includes to `**/tck/*.java` in +the same breath, so it runs the conformance suite and nothing else — not the module's unit tests. +Both halves are needed: the include alone leaves the exclusion in force and runs nothing, and +dropping the exclusion alone runs the unit tests too. Nothing activates it in CI. The consequence is that **no CI job runs the suite**, so a maintainer runs it by hand before merging a change to the provider's resolution or error behaviour, and quotes the result in the pull request. @@ -130,11 +136,11 @@ mvn -Ptck -pl providers/ofrep test before the first scenario, so a failure there comes out as a `-Ptck` failure — the signal-mixing this step exists to prevent, reintroduced by a flag. The separate `install` is what `-am` was there for. -The suite is currently **expected to fail** on the failures enumerated in `OfrepTckTest`, which come +The suite is currently **expected to fail** on the failures enumerated in `OfrepTest`, which come from flags the pinned testbed image does not serve (open-feature/flagd-testbed#392). A clean run is 65 scenarios, 46 passing, 17 skipped and 2 failing. -It is also **intermittently flaky**, and `OfrepTckTest`'s class javadoc says why: about half the runs +It is also **intermittently flaky**, and `OfrepTest`'s class javadoc says why: about half the runs carry one or two extra failures where an evaluation comes back as the code default or as `FLAG_NOT_FOUND`, on a scenario that moves from run to run. That is the testbed readiness window of open-feature/flagd-testbed#394, not a provider defect and not something to cover with a sleep. Repeat diff --git a/providers/ofrep/pom.xml b/providers/ofrep/pom.xml index 01a0c548ef..c946b93a6f 100644 --- a/providers/ofrep/pom.xml +++ b/providers/ofrep/pom.xml @@ -22,13 +22,14 @@ [0.1.0,) 2.0.4 - **/e2e/*.java + **/tck/*.java @@ -98,8 +99,9 @@ dev.openfeature.contrib.tools @@ -128,11 +130,15 @@ @@ -156,7 +162,7 @@ maven-surefire-plugin - **/e2e/*TckTest.java + **/tck/*.java diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java similarity index 99% rename from providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java rename to providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java index b9b463aeda..34c0d8a07d 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/e2e/OfrepTckTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java @@ -1,4 +1,4 @@ -package dev.openfeature.contrib.providers.ofrep.e2e; +package dev.openfeature.contrib.providers.ofrep.tck; import dev.openfeature.contrib.providers.ofrep.OfrepProvider; import dev.openfeature.contrib.providers.ofrep.OfrepProviderOptions; @@ -58,7 +58,7 @@ * suite that sleeps instead of holding the control API to its promise stops being able to detect * when the promise breaks, which is the whole argument of that issue. */ -public class OfrepTckTest extends ContainerizedProviderTckTest { +public class OfrepTest extends ContainerizedProviderTckTest { /** The container-internal port flagd serves the OFREP HTTP API on. */ private static final int OFREP_PORT = 8016; diff --git a/providers/ofrep/src/test/resources/tck/docker-compose.yaml b/providers/ofrep/src/test/resources/tck/docker-compose.yaml index d9725063f5..a7c2277837 100644 --- a/providers/ofrep/src/test/resources/tck/docker-compose.yaml +++ b/providers/ofrep/src/test/resources/tck/docker-compose.yaml @@ -18,7 +18,7 @@ # targeting-key-flag is served, which is why @targeting needs no bump: its three scenarios pass # on this image. The four disabled-* flags behind @disabled-flags are served too, from this # image's own flags/disabled-flags.json -- so the four failures that tag produced here are the -# provider's and not the stack's, and the tag is withheld in OfrepTckTest with the reason. flagd +# provider's and not the stack's, and the tag is withheld in OfrepTest with the reason. flagd # answers a disabled flag with 200, reason DISABLED and no value, which is what OFREP calls a # codeDefaultFlag; nothing is missing from this image. # From 8d2c5cefa64f9f10ce65c9ba8bd0ec98490fdeef Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 06:56:51 +0200 Subject: [PATCH 18/22] docs(ofrep): cut the conformance section to what is this adoption's The TCK section was 4.6 KB of a 7.1 KB provider README - larger than everything this module documents about itself. Most of it was the base README's or Appendix F's: why an adoption suite is excluded rather than gating, why it gets a step of its own, what a red conformance build says, why both halves of the tck profile are needed, and why the -am the command must not carry would mix signals. What is left answers the three questions an adoption README owes a reader. What this provider declares and why each absence is what it is - by pointing at OfrepTest, where the reasoning sits next to the declaration; the protocol-not-vendor note stays here because it is what makes an OFREP conformance run intelligible at all. What the tally is and which failures are expected - 65 scenarios, 46/17/2, plus the flagd-testbed#394 flake and how to tell it from a regression. And the command that runs it. 2.2 KB, from 4.6. The adoption now adds 36 lines to this README rather than 73. Signed-off-by: Simon Schrottner --- providers/ofrep/README.md | 81 +++++++++++---------------------------- 1 file changed, 22 insertions(+), 59 deletions(-) diff --git a/providers/ofrep/README.md b/providers/ofrep/README.md index 8a79acc374..6b6d4874d3 100644 --- a/providers/ofrep/README.md +++ b/providers/ofrep/README.md @@ -76,54 +76,12 @@ Given below are the supported configurations: ## Provider conformance (TCK) This provider adopts the [OpenFeature Provider TCK](../../tools/tck/README.md) as a single suite, -`OfrepTest`, in a source directory of its own: `src/test/java/.../ofrep/tck/`. It is the module's -only Docker-dependent test package, and having it be a directory rather than a filename convention -is what lets every selector below name a place instead of a pattern. In a package called `tck` the -class needs no further label; the fully-qualified name still carries everything. - -Read that class before changing it: it records which capabilities are declared, -which are withheld and why — several are withheld because OFREP puts the decision on the server -rather than in the provider, which is a fact about the protocol and not a defect — and every known -deviation. - -Because OFREP is a protocol rather than a vendor, the backend under test is simply something that -speaks it. The suite reuses the unmodified `flagd-testbed` image and its launchpad control API. - -**The suite is Docker-gated and excluded from the default build**, via -`**/tck/*.java` in this module's POM. That property is the -repository's convention for a Docker-dependent suite, fed to Surefire by the parent POM; the parent -defines no default, so each module that wants the gate declares it. This module did not, which meant -`mvn verify` started a Compose stack and the suite ran — and failed — in every job that touched -`providers/ofrep`. That is the second of the two mistakes -[Appendix F: Running the suite in CI](https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md#running-the-suite-in-ci) -names, and the appendix has the reasoning for the whole policy; the exclusion is the fix and this -paragraph is the other half of it. - -Exactly one profile touches the property — `tck`, below — and `e2e`, which the repository's CI -activates on every push, does not exist in this module at all. Checked rather than read, because -that is the appendix's other warning: - -```bash -mvn -Pe2e -pl providers/ofrep help:evaluate -Dexpression=testExclusions -DforceStdout -``` - -The exclusion is Surefire's and not the compiler's, so the suite still builds against the harness in -every job. - -**The conformance suite has a profile of its own, `tck`.** That is the separation the appendix asks -for, and the reason is what a red build *says* rather than how long it takes: a conformance run -carries failures by design, wherever `OfrepTest` declares a `knownDeviation`, so a signal shared -with a suite that is expected green ends with somebody silencing the informative half. The profile -drops the `tck` directory from the exclusion and narrows Surefire's includes to `**/tck/*.java` in -the same breath, so it runs the conformance suite and nothing else — not the module's unit tests. -Both halves are needed: the include alone leaves the exclusion in force and runs nothing, and -dropping the exclusion alone runs the unit tests too. Nothing activates it in CI. - -The consequence is that **no CI job runs the suite**, so a maintainer runs it by hand before merging -a change to the provider's resolution or error behaviour, and quotes the result in the pull request. -A scheduled or path-filtered workflow was considered and declined: a suite whose red is diagnosed by -whoever happens to read the notification is worse than one whose red is diagnosed by the person who -caused it. +`OfrepTest`, in `src/test/java/.../ofrep/tck/`. **Read that class before changing it.** It records +which capabilities are declared, which are withheld and why — several are withheld because OFREP puts +the decision on the server rather than in the provider, which is a fact about the protocol and not a +defect — and every known deviation. Because OFREP is a protocol rather than a vendor, the backend +under test is simply something that speaks it: the suite reuses the unmodified `flagd-testbed` image +and its launchpad control API. ```bash # once, if tools/tck is not in your local repository yet @@ -132,16 +90,21 @@ mvn -pl tools/tck -am -DskipTests install mvn -Ptck -pl providers/ofrep test ``` -**Do not add `-am` to the run itself**: it pulls `tools/tck` into the reactor and runs its 246 tests -before the first scenario, so a failure there comes out as a `-Ptck` failure — the signal-mixing this -step exists to prevent, reintroduced by a flag. The separate `install` is what `-am` was there for. +Do not add `-am` to the run itself; the TCK README says why. + +The suite is Docker-gated and excluded from the default build by +`**/tck/*.java` in this module's POM, and the `tck` profile above is +the only thing that undoes it — this module has no `e2e` profile, so the `-Pe2e` that CI activates on +every push changes nothing here. The exclusion was missing until recently, which meant `mvn verify` +started a Compose stack and ran the suite, red, in every job that touched `providers/ofrep`. -The suite is currently **expected to fail** on the failures enumerated in `OfrepTest`, which come -from flags the pinned testbed image does not serve (open-feature/flagd-testbed#392). A clean run is -65 scenarios, 46 passing, 17 skipped and 2 failing. +**No CI job runs it**, so a maintainer runs it by hand before merging a change to the provider's +resolution or error behaviour, and quotes the result in the pull request. -It is also **intermittently flaky**, and `OfrepTest`'s class javadoc says why: about half the runs -carry one or two extra failures where an evaluation comes back as the code default or as -`FLAG_NOT_FOUND`, on a scenario that moves from run to run. That is the testbed readiness window of -open-feature/flagd-testbed#394, not a provider defect and not something to cover with a sleep. Repeat -the run before treating an extra failure as a regression; anything that reproduces is one. +A clean run is **65 scenarios: 46 passing, 17 skipped, 2 failing**, the two failures being flags the +pinned testbed image does not serve (open-feature/flagd-testbed#392). It is also **intermittently +flaky** — about half the runs carry one or two extra failures where an evaluation comes back as the +code default or as `FLAG_NOT_FOUND`, on a scenario that moves from run to run. That is the testbed +readiness window of open-feature/flagd-testbed#394, not a provider defect and not something to cover +with a sleep; `OfrepTest`'s javadoc has the detail. Repeat the run before treating an extra failure as +a regression — anything that reproduces is one. From a84d34d6d8fcc9c60783cb0d6c38cd856320372b Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 08:36:20 +0200 Subject: [PATCH 19/22] test(ofrep): run against the shared Compose file, and cut what Appendix F owns The stack this suite needs is the one providers/flagd already brings up, so it names the shared tools/flagd-testbed/docker-compose.yaml instead of a near-identical copy of its own. Nothing about the run changes: the file exposes 8016 alongside flagd's ports and the TCK resolves only the ones this suite asks for. The class javadoc loses the testbed-gap paragraph, the falsy-flag rename history and the comparison with Go's OFREP provider; what the image does not serve is open-feature/flagd-testbed#392's. capabilities() loses the restatement of why @numeric-coercion is not a spec rule, which Capability.NUMERIC_COERCION says. Everything measured stays: the six-capability source-line evidence, the codeDefaultFlag protocol reading and the probed flagd responses behind it, the exact-instance analysis of both lossless directions, and the note that @disabled-flags is the one declaration the settled guidance would shape differently. Comments only. No behaviour change. Signed-off-by: Simon Schrottner --- .../providers/ofrep/tck/OfrepTest.java | 219 +++++++----------- .../test/resources/tck/docker-compose.yaml | 33 --- 2 files changed, 80 insertions(+), 172 deletions(-) delete mode 100644 providers/ofrep/src/test/resources/tck/docker-compose.yaml diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java index 34c0d8a07d..4170a7a821 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java @@ -18,45 +18,27 @@ * *

    OFREP is a protocol rather than a vendor, so the backend under test is simply something that * speaks it. flagd does, on port {@value #OFREP_PORT}, which means this suite reuses the flagd - * testbed image and its launchpad control API unchanged — see - * {@code src/test/resources/tck/docker-compose.yaml}. + * testbed image and its launchpad control API unchanged — the same stack the flagd adoption runs + * against, from the same Compose file. * - *

    The testbed does not yet serve the whole canonical flag set. Three flags the - * suite's assets added are absent from {@code flagd-testbed} v3.8.0: {@code large-integer-flag}, - * {@code huge-integer-flag} and {@code integral-float-flag}. Only the first is reached — - * {@code huge-integer-flag} is asked for solely under {@code @large-integers}, which no Java - * provider can be asked, and {@code integral-float-flag} solely under {@code @numeric-coercion}, - * which is withheld below — so exactly one untagged scenario, the 32-bit precision one, fails with - * {@code FLAG_NOT_FOUND} until open-feature/flagd-testbed#392 lands. - * - *

    The three falsy flags used to fail the same way and no longer do. The testbed's - * {@code zero-flags.json} already served {@code boolean-zero-flag}, {@code integer-zero-flag} and - * {@code string-zero-flag} with {@code zero}/{@code non-zero} variants, while the canonical set - * called them {@code false-flag}, {@code zero-flag} and {@code empty-string-flag}; spec ba002ce8 - * renamed the canonical flags to the testbed's names rather than the other way round. - * - *

    A missing flag is a gap in the stack, not in the provider, so it is recorded here rather than - * declared as a {@code KnownDeviation}: a deviation says the provider is wrong, and the provider was + *

    A clean run is 65 scenarios, 46 passing, 17 skipped and 2 failing. Both + * failures are the untagged 32-bit precision scenario and the {@code @variants} row beside it, on a + * flag the pinned testbed image does not serve — open-feature/flagd-testbed#392. That is a gap in + * the stack rather than in the provider, so neither is a {@code KnownDeviation}: the provider was * never given the flag to get wrong. * - *

    A clean run is 65 scenarios, 46 passing, 17 skipped and those 2 failing. - * *

    The suite is intermittently flaky, and the flakiness is the backend's. Roughly - * half of the runs measured carry one or two additional failures on top of those two, and - * they all have the same shape: an evaluation that should have resolved comes back as the code - * default, or as {@code FLAG_NOT_FOUND} where {@code TYPE_MISMATCH} was expected, or with reason - * {@code ERROR} where a resolution was expected. Which scenario is hit moves from run to run — + * half of the runs measured carry one or two additional failures, all of the same shape: an + * evaluation that should have resolved comes back as the code default, or as + * {@code FLAG_NOT_FOUND} where {@code TYPE_MISMATCH} was expected, or with reason {@code ERROR} + * where a resolution was expected. Which scenario is hit moves from run to run — * {@code errors.feature}, {@code evaluation.feature} and {@code reason.feature} have each been the - * victim — so it is not a property of any assertion. - * - *

    It is not new with the reason scenarios, and that was checked rather than assumed: five runs at - * this revision and three at {@code ccdb8879} before it, with the old pin producing a run of seven - * failures and a run of two from the same tree. It is the flagd-testbed readiness window that - * open-feature/flagd-testbed#394 exists to close — a control endpoint returning before the backend - * serves the new state — reaching a provider that holds nothing between calls, so every evaluation - * races the stack afresh. Do not add a settle after control calls to cover it: a - * suite that sleeps instead of holding the control API to its promise stops being able to detect - * when the promise breaks, which is the whole argument of that issue. + * victim — so it is not a property of any assertion. Checked rather than assumed: five runs at this + * revision and three at {@code ccdb8879} before it, the old pin producing a run of seven failures + * and a run of two from the same tree. It is the readiness window + * open-feature/flagd-testbed#394 exists to close, reaching a provider that holds nothing between + * calls, so every evaluation races the stack afresh. Do not add a settle after control + * calls to cover it — that issue, and Appendix F, say why. */ public class OfrepTest extends ContainerizedProviderTckTest { @@ -71,9 +53,16 @@ public class OfrepTest extends ContainerizedProviderTckTest { */ private static final int UNAVAILABLE_PORT = 9999; + /** + * {@inheritDoc} + * + *

    Outside this module on purpose, and not the idiomatic {@code src/test/resources} path: the + * flagd adoption runs against the same stack and names the same file, so there is one image tag + * for both rather than two that can drift. Module-relative, like any other value here. + */ @Override public File composeFile() { - return new File("src/test/resources/tck/docker-compose.yaml"); + return new File("../../tools/flagd-testbed/docker-compose.yaml"); } @Override @@ -126,13 +115,9 @@ public FeatureProvider createUnavailableProvider() { *

  • {@link Capability#REINITIALIZATION} — omitted, and independently of the omission * above. {@code shutdown()} terminates the executor the HTTP client runs on * (OfrepProvider.java:90-108) and, with no {@code initialize()} of its own, nothing ever - * recreates it, so a shut-down {@code OfrepProvider} cannot be started again. Requirement - * 2.5.2 permits exactly that — a provider SHOULD revert to its uninitialized - * state and "some providers MAY allow reinitialization from this state" — so this - * is a choice the specification offers and not a defect to declare. It is named here - * rather than left to the {@code @lifecycle} skip because - * {@link Capability#declarableExcept} would otherwise have claimed it, and a claim nothing - * examined is exactly what the declaration exists to prevent. + * recreates it, so a shut-down {@code OfrepProvider} cannot be started again — which + * Requirement 2.5.2 permits. Named here rather than left to the {@code @lifecycle} skip + * because {@link Capability#declarableExcept} would otherwise have claimed it. *
  • {@link Capability#EVENTS} — the class declares {@code implements FeatureProvider}, * not {@code extends EventProvider} (OfrepProvider.java:19), so it has no {@code emit*} * method available and calls none. The whole file contains no reference to @@ -164,10 +149,8 @@ public FeatureProvider createUnavailableProvider() { * check and no round trip anywhere in that path — nothing in the provider widens or narrows a * number. * - *

    The tag's rule is that lossless coercion must succeed and lossy coercion must return - * {@code TYPE_MISMATCH}, and the three scenarios carrying it test all three cases; a provider - * declaring the tag must pass all three. This provider passes one. The lossy case is right for - * the wrong reason — {@code float-flag} (0.5) requested as an integer is rejected rather than + *

    A provider declaring the tag must pass all three of its scenarios. This one passes one, and + * that one is right for the wrong reason — {@code float-flag} (0.5) requested as an integer is rejected rather than * truncated to {@code 0}, because it is a {@link Double} and not because 0.5 is fractional — * and both lossless cases fail on the same exact-instance check. {@code integer-flag} (10) * requested as a float arrives as an {@code Integer}, which {@code Double.class.isInstance} @@ -178,82 +161,50 @@ public FeatureProvider createUnavailableProvider() { * three scenarios red. * *

    No {@code KnownDeviation} accompanies it, and that is deliberate rather than silence. - * Appendix F is explicit that {@code @numeric-coercion} is the one capability the specification - * does not define: OpenFeature has a single {@code number} type, the rule this tag is tested - * against is borrowed from flagd's numeric-coercion ADR, and "a provider that behaves - * differently is not violating the specification". The appendix says so having retracted - * an earlier draft that called non-declaration "an admission of a known bug". Strict typing in - * both directions is therefore a legitimate choice — the SDK's own {@code InMemoryProvider} - * makes it — and a deviation entry would assert a defect the spec says is not one. That is the - * opposite mistake from a vacuous declaration, but a mistake in the same currency. - * - *

    Go's OFREP provider declares the tag, and that is not an inconsistency to reconcile away: - * it coerces through an integral check and this one does not, so the two declarations describe - * two implementations rather than one protocol. OFREP being JSON is what makes the difference - * possible — one wire number type, so integer-ness is the provider's decision, not the - * payload's. Declaring the tag here to match Go would also run the - * {@code integral-float-flag} scenario, which the testbed cannot serve — a note about what such - * a run would look like, and deliberately not a reason. Appendix F's declaring rules - * have since said why it cannot be one: a scenario that fails because the backend cannot serve - * its fixture is not a provider defect, so the gap is no argument for withholding anything. The - * reason is the paragraph above and only that paragraph. + * {@link Capability#NUMERIC_COERCION} records that the rule is borrowed rather than normative, + * so strict typing in both directions is a legitimate choice — the SDK's own + * {@code InMemoryProvider} makes it — and a deviation would assert a defect the specification + * says is not one. * - *

    Nor does the appendix's scenario-level rule reach this omission, which is worth saying - * because it reads at first as though it should. That rule — once a provider is attempting a - * capability, declare it when at least one scenario gating it can be put to the provider — is - * about whether a question is askable, and all three of these are. What comes first is - * whether an answer is owed, and no requirement says this one is: the provider does not coerce, - * the specification permits that, and withholding is the honest report. Taking the second rule - * without the first would manufacture two failures out of a permitted choice. + *

    Appendix F's scenario-level declaring rule does not reach this omission, which is worth + * saying because at first it reads as though it should. That rule is about whether a question is + * askable, and all three of these are; what comes first is whether an answer is owed, + * and none is. Taking the second rule without the first would manufacture two failures out of a + * permitted choice. * *

    All of that is read from the source, because a withheld tag means the scenarios are - * skipped and a run cannot confirm it; the three are reported as skipped with this reason on - * every run, which is the observable that the declaration is being honoured rather than the - * behaviour behind it. {@code OfrepProviderTest} asserts the exact-instance check itself, but - * only across Boolean and String (OfrepProviderTest.java:71, 338-339) — no unit test pins the - * numeric pair, which is why the reasoning above cites {@code handleResolved} directly. A run in - * which either lossless scenario passes means {@code handleResolved} or the deserialiser - * changed, and this declaration should follow it. + * skipped and a run cannot confirm it. {@code OfrepProviderTest} asserts the exact-instance + * check itself, but only across Boolean and String (OfrepProviderTest.java:71, 338-339) — no + * unit test pins the numeric pair, which is why the reasoning above cites {@code handleResolved} + * directly. A run in which either lossless scenario passes means {@code handleResolved} or the + * deserialiser changed, and this declaration should follow it. * - *

    Two things this leaves in place. {@link Capability#OBJECT} is declared: the same - * exact-instance check is what makes the {@code @object} mismatch matrix work, and the - * structured happy path passes through {@code resolve(Object.class, ...)}, which every non-null - * value satisfies, and is converted with {@code Value.objectToValue} (Resolver.java:125-136). - * And {@link Capability#LARGE_INTEGERS} is no longer in the list below, which is the one - * omission here that says nothing about OFREP. The tag asks for 2^53 − 1 and - * {@code Client.getIntegerDetails} is a 32-bit {@code Integer} with no room for it, so the limit - * is the SDK's; the TCK refuses the capability outright now rather than asking every Java - * adoption to remember, and {@link Capability#declarableExcept} no longer offers it. Its - * scenario is still skipped, with a reason naming the accessor rather than this provider — which - * matters more here than elsewhere, because every other name in that list is something - * this provider genuinely cannot do, and a reader should not have to guess which is which. + *

    {@link Capability#OBJECT} is declared: the same exact-instance check is what makes the + * {@code @object} mismatch matrix work, and the structured happy path passes through + * {@code resolve(Object.class, ...)}, which every non-null value satisfies, and is converted + * with {@code Value.objectToValue} (Resolver.java:125-136). {@link Capability#LARGE_INTEGERS} is + * absent from the list below for a reason that says nothing about OFREP — the TCK refuses it + * centrally — and that matters more here than elsewhere, because every other name in that list + * is something this provider genuinely cannot do. * *

    {@link Capability#VARIANTS} and {@link Capability#TARGETING} are declared, and unlike the * withheld tags above both are confirmed by a run rather than read from the source. The OFREP - * response carries {@code variant} alongside {@code value} and {@code reason}, and the provider - * passes it through, so seven of the {@code @variants} outline's eight rows pass; the eighth asks - * for {@code large-integer-flag}'s {@code max-int32} and gets no variant because testbed v3.8.0 - * does not serve that flag — the gap already noted next to the image tag, not a second defect. - * {@code @targeting} matters more here than the capability's name suggests: the provider sends the - * evaluation context in the request body and the backend evaluates the rule, so the three - * {@code targeting-key-flag} scenarios are the only ones in the canonical set that would notice a - * context dropped on the way out. All three pass. + * response carries {@code variant} alongside {@code value} and {@code reason} and the provider + * passes it through, so seven of the {@code @variants} outline's eight rows pass; the eighth is + * the testbed gap above, not a second defect. {@code @targeting} matters more here than its name + * suggests: the provider sends the evaluation context in the request body and the backend + * evaluates the rule, so the three {@code targeting-key-flag} scenarios are the only ones in the + * canonical set that would notice a context dropped on the way out. All three pass. * - *

    {@link Capability#STANDARD_REASONS} is declared, and it arrived by the - * {@code declarableExcept} default rather than by a decision, so it was measured before being - * written down. Eight of {@code reason.feature}'s nine scenarios run and pass: {@code STATIC} for - * the four rule-less flags, {@code ERROR} beside {@code FLAG_NOT_FOUND} and - * {@code TYPE_MISMATCH}, and — because {@code @targeting} is declared here — {@code - * TARGETING_MATCH} and {@code DEFAULT} either side of {@code targeting-key-flag}'s rule. The - * ninth carries {@code @disabled-flags} as well and is skipped for that omission, which is the - * right outcome and not a second report of the same gap: what is wrong with this provider's - * handling of a disabled flag is already said once, below and in {@link #knownDeviations()}, and - * a reason it never reaches is not more evidence of it. - * - *

    Worth noting what the declaration does not claim. The reason for a disabled flag is - * the one standard reason this provider is not held to, so the tag here means "the standard - * vocabulary, over the responses this provider actually completes". A consumer reading the report - * sees the withheld {@code @disabled-flags} beside it and can tell which scenario went unasked. + *

    {@link Capability#STANDARD_REASONS} arrived by the {@code declarableExcept} default rather + * than by a decision, so it was measured before being written down. Eight of + * {@code reason.feature}'s nine scenarios run and pass: {@code STATIC} for the four rule-less + * flags, {@code ERROR} beside {@code FLAG_NOT_FOUND} and {@code TYPE_MISMATCH}, and — because + * {@code @targeting} is declared here — {@code TARGETING_MATCH} and {@code DEFAULT} either side + * of {@code targeting-key-flag}'s rule. The ninth carries {@code @disabled-flags} and is skipped + * for that omission, so the tag here means "the standard vocabulary, over the responses this + * provider actually completes"; a consumer sees the withheld {@code @disabled-flags} beside it + * and can tell which scenario went unasked. * *

    {@link Capability#DISABLED_FLAGS} is withheld, and unlike every other withheld tag * above it was measured. Declared, all four rows of its outline fail, and they fail on @@ -297,29 +248,23 @@ public FeatureProvider createUnavailableProvider() { * {@link dev.openfeature.contrib.tools.tck.KnownDeviation} rather than left as a bare * omission — see {@link #knownDeviations()}. That is the opposite call from * {@code @numeric-coercion} above, and the difference is where the rule lives: numeric coercion - * is a rule Appendix F borrowed from flagd's ADR and that no specification states, whereas - * {@code codeDefaultFlag} is a {@code MUST} in the protocol this provider implements. Withholding - * the tag keeps the four rows honest — skipped, not passed — and the deviation is what says the - * omission is a bug rather than a choice. Delete both once {@code handleResolved} honours a - * value-less success. + * is borrowed from flagd's ADR and no specification states it, whereas {@code codeDefaultFlag} + * is a {@code MUST} in the protocol this provider implements. Delete both once + * {@code handleResolved} honours a value-less success. * *

    This is the one declaration on this branch that the settled guidance would shape - * differently, and it is recorded here rather than quietly left. - * {@link dev.openfeature.contrib.tools.tck.KnownDeviation} prefers the declared-and-failing shape - * and confines the withheld-and-skipped one to a provider that cannot attempt the behaviour at - * all. This provider does attempt it: it receives the {@code codeDefaultFlag} response, parses - * the {@code reason} that accompanies it, and then answers with the wrong error code — measured, - * four rows, failing on the code and not on the value. By that reading the honest report is to - * declare {@code @disabled-flags}, let the four rows fail, and keep this same deviation beside - * them, exactly as the flagd adoption does for {@code @numeric-coercion}. The flip is a change of - * results rather than of prose, so it is not made in the documentation pass that noticed it; it - * costs four failures in place of four skips and nothing else, and the deviation's text needs no - * change when it happens. + * differently, and it is recorded here rather than quietly left. The preferred shape is + * declared-and-failing, and this provider does attempt the behaviour: it receives the + * {@code codeDefaultFlag} response, parses the {@code reason} that accompanies it, and answers + * with the wrong error code — measured, four rows, failing on the code and not on the value. By + * that reading the honest report is to declare {@code @disabled-flags}, let the four rows fail, + * and keep this same deviation beside them. The flip is a change of results rather than of + * prose, so it is not made in the documentation pass that noticed it; it costs four failures in + * place of four skips and nothing else, and the deviation's text needs no change when it + * happens. * - *

    {@code declarableExcept} rather than {@code EnumSet.complementOf}, which would also claim - * {@code @caching} on the way past; the suite refuses such a declaration at startup. It claimed - * {@code @targeting} the same way until that tag gated something — the reserved set shrinks as - * the vocabulary fills up, which is an argument for the form of the call rather than against it. + *

    {@code declarableExcept} and not {@code EnumSet.complementOf}, which would claim + * {@code @caching} on the way past; the suite refuses such a declaration at startup. */ @Override public Set capabilities() { @@ -342,11 +287,7 @@ public Set capabilities() { * Every other withheld tag describes something {@code OfrepProvider} has no machinery for — no * initialisation, no events, no state between calls — or a rule no specification states. This one * describes a response the provider receives, understands well enough to parse, and then answers - * wrongly. - * - *

    Untracked, because there is no issue to point at yet. Naming it anyway is the whole point of - * the mechanism: in the results a capability withheld by choice and one withheld because it is - * broken are the same absence, and only the provider author can say which happened. + * wrongly. Untracked, because there is no issue to point at yet. * *

    The summary names the response shape rather than the scenario, because the scenario is only * where it was noticed. {@code codeDefaultFlag} is not specific to disabled flags — any backend diff --git a/providers/ofrep/src/test/resources/tck/docker-compose.yaml b/providers/ofrep/src/test/resources/tck/docker-compose.yaml deleted file mode 100644 index a7c2277837..0000000000 --- a/providers/ofrep/src/test/resources/tck/docker-compose.yaml +++ /dev/null @@ -1,33 +0,0 @@ -# Backend stack for the OpenFeature Provider TCK, wrapping the unmodified flagd testbed image. -# -# OFREP is a vendor-neutral protocol, so the TCK does not need an OFREP-specific testbed: any -# backend that speaks OFREP and exposes the control API will do. flagd serves the OFREP HTTP API -# on 8016 alongside its own gRPC surfaces, and the testbed image already ships the "launchpad" -# control API on 8080 — the same stack the flagd provider's TCK suite uses, seeded with the same -# flag set. No new image, no new control API. -# -# That flag set is not yet the whole canonical one: v3.8.0 has none of large-integer-flag, -# huge-integer-flag or integral-float-flag. Only large-integer-flag is reached by scenarios that -# run here -- the other two sit behind @numeric-coercion, which this provider withholds, and -# @large-integers, which no Java provider can be asked because Client.getIntegerDetails is a -# 32-bit Integer -- and it is reached twice: the untagged precision scenario, which gets -# the code default instead of 2147483647, and the @variants row that asks for its "max-int32" -# variant and is answered with none. Both fail until open-feature/flagd-testbed#392 lands. Bump -# the tag then. -# -# targeting-key-flag is served, which is why @targeting needs no bump: its three scenarios pass -# on this image. The four disabled-* flags behind @disabled-flags are served too, from this -# image's own flags/disabled-flags.json -- so the four failures that tag produced here are the -# provider's and not the stack's, and the tag is withheld in OfrepTest with the reason. flagd -# answers a disabled flag with 200, reason DISABLED and no value, which is what OFREP calls a -# codeDefaultFlag; nothing is missing from this image. -# -# Note there are no host port bindings. The TCK requires dynamically mapped ports and discovers -# them after startup — a pinned host port would make the suite unrunnable in parallel and would -# collide with a developer's local flagd. -services: - backend: - image: ghcr.io/open-feature/flagd-testbed:v3.8.0 - ports: - - 8016 # flagd OFREP evaluation (HTTP) - - 8080 # launchpad control API From b7673751918060559652428abf44e22504bcc01b Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 09:01:09 +0200 Subject: [PATCH 20/22] test(ofrep): say what the codeDefaultFlag reading establishes Capability.DISABLED_FLAGS now cites this class for the corrected gating question, so the paragraph reads as establishing the fact rather than rebutting a sentence no document makes any more. Every piece of evidence is unchanged. Comments only. Signed-off-by: Simon Schrottner --- .../openfeature/contrib/providers/ofrep/tck/OfrepTest.java | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java index 4170a7a821..684502c6b3 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java @@ -225,9 +225,10 @@ public FeatureProvider createUnavailableProvider() { * {@code FLAG_NOT_FOUND} and "No value returned for flag", discarding the {@code reason} it * parsed one field earlier (Resolver.java:169-181, OfrepResponse.java:19). * - *

    That response is well-formed OFREP, and it says what to do. The obvious - * reading of this failure — the caller's default never leaves the process, so a provider whose - * backend decides cannot hold the tag — does not survive the protocol. OFREP's + *

    That response is well-formed OFREP, and it says what to do. It is worth + * establishing, because the intuitive reading of the failure — the caller's default never leaves + * the process, so a provider whose backend decides cannot hold this tag — is wrong here, and + * {@link Capability#DISABLED_FLAGS} cites this class for why. OFREP's * {@code evaluationSuccess} requires only {@code key} and {@code reason}; {@code value} is * not required, because one of the member schemas a success may be is * {@code codeDefaultFlag}, described as "A flag evaluation that defers to the code default From 093abb88adff09df66d445b0c22dcb27fbca8841 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Tue, 15 Sep 2026 22:33:09 +0200 Subject: [PATCH 21/22] test(ofrep): declare @string-typing, confirmed by a run The "everything except" default already declares the new tag, so this records why that is right here rather than changing what is declared. handleResolved admits a value only on an exact type.isInstance check, so String.class.isInstance of a Boolean, an Integer or a Double is false and the provider answers TYPE_MISMATCH with the code default instead of the value's toString(). That is the same check the withheld @numeric-coercion reasoning cites, reached from the other side: strict typing loses the numeric tag and wins this one. Measured, not read: 65 scenarios with the skip count unchanged at 17, and none of the four @string-typing scenarios among the failures. The run carried three failures rather than the clean two -- a TYPE_MISMATCH answered as FLAG_NOT_FOUND, which passed on seven of surefire's eight reruns and is the testbed readiness flake of open-feature/flagd-testbed#394 this file already describes. The clean-run tally is unchanged. Signed-off-by: Simon Schrottner --- .../contrib/providers/ofrep/tck/OfrepTest.java | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java index 684502c6b3..2d9d2de96e 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java @@ -179,6 +179,19 @@ public FeatureProvider createUnavailableProvider() { * directly. A run in which either lossless scenario passes means {@code handleResolved} or the * deserialiser changed, and this declaration should follow it. * + *

    {@link Capability#STRING_TYPING} is declared, and all four of its scenarios + * pass. It gates what specification revision {@code d47a66eb} moved out of the + * mandatory matrix — {@code boolean-flag}, {@code integer-flag}, {@code float-flag} and + * {@code object-flag} asked through the String accessor — and the mechanism is the same + * exact-instance check as above, reached from the other side: {@code String.class.isInstance} of + * a {@link Boolean}, an {@link Integer} or a {@link Double} is false, so {@code handleResolved} + * returns {@code TYPE_MISMATCH} with the code default rather than the value's + * {@code toString()}. OFREP carries a JSON type per flag and Jackson preserves it, so this + * provider is on the typed side of the distinction the capability exists to draw. Confirmed by a + * run: skip count unchanged at 17, and none of the four among the failures. It is one of the + * cases {@code OfrepProviderTest} does pin directly, across Boolean and String + * (OfrepProviderTest.java:71, 338-339). + * *

    {@link Capability#OBJECT} is declared: the same exact-instance check is what makes the * {@code @object} mismatch matrix work, and the structured happy path passes through * {@code resolve(Object.class, ...)}, which every non-null value satisfies, and is converted From 887bfafaef06d53dd739d32d60845814a37a9281 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Wed, 16 Sep 2026 09:42:47 +0200 Subject: [PATCH 22/22] test(ofrep): declare @fully-typed-values too, confirmed by a run bda599f1 split @string-typing, holding float-flag and object-flag behind a new @fully-typed-values for backends that type a boolean and an integer but keep a float and a structure as text. OFREP is not one of them: the exact-instance check in handleResolved is indifferent to which type it is refusing, so one line of code answers all four questions, and both tags are declared. declarableExcept already picks the new tag up, so this is javadoc -- but the claim was measured, not inherited. After the re-pin: 65 scenarios, 45 passing, 3 failing, 17 skipped, with no FULLY_TYPED_VALUES entry among the skip reasons, and the newly standalone "A float flag is not returned as its string representation" executed and passing alongside the structured one. The skip composition is unchanged: LIFECYCLE 6, DISABLED_FLAGS 5, NUMERIC_COERCION 3, EVENTS 2, LARGE_INTEGERS 1. The clean-run tally in the class comment and the README stays at 46 passing and 2 failing. This run carried one extra failure, Example #1.1 resolving "on" as null, which is the shape and the magnitude the class comment already records for open-feature/flagd-testbed#394. The run before it was worse and is not reported as a regression either: the launchpad control API refused the first POST /start outright and all 65 scenarios errored, which cleared completely on rerun. Signed-off-by: Simon Schrottner --- .../providers/ofrep/tck/OfrepTest.java | 30 +++++++++++-------- 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java index 2d9d2de96e..f452b2eb23 100644 --- a/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java +++ b/providers/ofrep/src/test/java/dev/openfeature/contrib/providers/ofrep/tck/OfrepTest.java @@ -179,18 +179,24 @@ public FeatureProvider createUnavailableProvider() { * directly. A run in which either lossless scenario passes means {@code handleResolved} or the * deserialiser changed, and this declaration should follow it. * - *

    {@link Capability#STRING_TYPING} is declared, and all four of its scenarios - * pass. It gates what specification revision {@code d47a66eb} moved out of the - * mandatory matrix — {@code boolean-flag}, {@code integer-flag}, {@code float-flag} and - * {@code object-flag} asked through the String accessor — and the mechanism is the same - * exact-instance check as above, reached from the other side: {@code String.class.isInstance} of - * a {@link Boolean}, an {@link Integer} or a {@link Double} is false, so {@code handleResolved} - * returns {@code TYPE_MISMATCH} with the code default rather than the value's - * {@code toString()}. OFREP carries a JSON type per flag and Jackson preserves it, so this - * provider is on the typed side of the distinction the capability exists to draw. Confirmed by a - * run: skip count unchanged at 17, and none of the four among the failures. It is one of the - * cases {@code OfrepProviderTest} does pin directly, across Boolean and String - * (OfrepProviderTest.java:71, 338-339). + *

    {@link Capability#STRING_TYPING} and {@link Capability#FULLY_TYPED_VALUES} are both + * declared, and all four of their scenarios pass. Together they gate what specification + * revision {@code d47a66eb} moved out of the mandatory matrix — {@code boolean-flag}, + * {@code integer-flag}, {@code float-flag} and {@code object-flag} asked through the String + * accessor — and the mechanism is the same exact-instance check as above, reached from the other + * side: {@code String.class.isInstance} of a {@link Boolean}, an {@link Integer} or a + * {@link Double} is false, so {@code handleResolved} returns {@code TYPE_MISMATCH} with the code + * default rather than the value's {@code toString()}. OFREP carries a JSON type per flag and + * Jackson preserves it, so this provider is on the typed side of the distinction the capabilities + * exist to draw. + * + *

    Both tags, because the check is indifferent to which type it is refusing. {@code bda599f1} + * split {@code @fully-typed-values} off for backends that type a boolean and an integer but keep + * a float and a structure as text; OFREP is not one of them, and the same line of code answers + * all four. Confirmed by a run after the re-pin: skip count unchanged at 17, with no + * {@code FULLY_TYPED_VALUES} entry among the skip reasons, and none of the four among the + * failures. It is one of the cases {@code OfrepProviderTest} does pin directly, across Boolean + * and String (OfrepProviderTest.java:71, 338-339). * *

    {@link Capability#OBJECT} is declared: the same exact-instance check is what makes the * {@code @object} mismatch matrix work, and the structured happy path passes through