From 7a9354f26f0fecb72d9a00acc4f1be0cd1bb5a50 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 18:00:23 +0200 Subject: [PATCH 01/22] feat(provider-tck): emit a machine-readable conformance report Set PROVIDER_TCK_REPORT_DIR (or -Dprovider.tck.report.dir) and each suite writes two files: an envelope conforming to the report schema in Appendix F, and the run's results as a Cucumber Messages stream. An environment variable rather than a method on ProviderTckHarness, so emitting a report is a property of the run and not of the code: CI sets it, a developer running the suite locally does not, and no adopter changes a line to publish one. Unset means no report, which is not an error. Several suites in one JVM each write their own pair, so two resolver modes do not collide. The results are not a format this project defines. The .ndjson is produced by Cucumber's own io.cucumber.core.plugin.MessageFormatter -- the same class the built-in message: plugin instantiates -- so the bytes are what --plugin message:... would have written. It already carries everything a per-scenario report would have had to invent: every scenario's outcome, its tags including any on an individual Examples block, an exact Scenario Outline row identity in pickle.astNodeIds, and the source of every feature that ran. An earlier version of this reverse-engineered that last fact by re-parsing the feature source and matching line numbers; the stream states it outright, which is the argument for a standard format over one we maintain. A plugin of our own rather than the built-in one only because a @ConfigurationParameter value is a compile-time constant, so the built-in plugin's path cannot be derived from the directory the run asked for. The envelope carries the four things no results format can state: what the provider calls itself, the SDK version actually on the classpath (read at runtime, because the TCK depends on a version range and never pins one), which TCK implementation and open-feature/spec revision asked the questions, and the capability declaration. The declaration is an input to reading the results rather than a summary of them: the stream says a scenario was skipped, and only the declaration says whether that is because the provider declines the capability it needed. Nothing here widens the adopter-facing API. capabilities(), knownDeviations(), configuration() and BackendControl.controlApi() are all on the base, and this branch only reads them -- so adopting the TCK and emitting a report are the same declaration, and a provider that never emits one is not asked for less. What the report is for: this suite promises that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed, and a promise is not a check. Every scenario appears exactly once, whatever happened to it -- a report that quietly omitted the scenarios it did not run would satisfy every other rule and still mislead. ConformanceReportPluginTest drives a fixture suite through the real Cucumber engine and asserts both properties over the emitted stream. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 122 ++++ tools/tck/pom.xml | 66 ++ .../contrib/tools/tck/ConformanceReport.java | 244 +++++++ .../tools/tck/ConformanceReportPlugin.java | 302 +++++++++ .../contrib/tools/tck/ProviderTck.java | 7 +- .../contrib/tools/tck/ReportNames.java | 42 ++ .../contrib/tools/tck/TckBuildInfo.java | 124 ++++ .../contrib/tools/tck/TckRunMetadata.java | 101 +++ .../contrib/tools/tck/TckRuntime.java | 50 ++ .../tools/tck/steps/ProviderSteps.java | 12 +- .../tools/tck/provider-tck-build.properties | 7 + .../tck/ConformanceReportPluginTest.java | 622 ++++++++++++++++++ .../tck/selftest/ReportSelfTestSteps.java | 75 +++ .../resources/report-selftest/report.feature | 37 ++ 14 files changed, 1802 insertions(+), 9 deletions(-) create mode 100644 tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java create mode 100644 tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java create mode 100644 tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckBuildInfo.java create mode 100644 tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java create mode 100644 tools/tck/src/main/resources-filtered/dev/openfeature/contrib/tools/tck/provider-tck-build.properties create mode 100644 tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java create mode 100644 tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/selftest/ReportSelfTestSteps.java create mode 100644 tools/tck/src/test/resources/report-selftest/report.feature diff --git a/tools/tck/README.md b/tools/tck/README.md index 242d8009b..9fed8fc01 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -345,6 +345,128 @@ the exclusion **and** narrows Surefire's includes to it: **Both halves are needed.** Dropping alone runs the module's unit tests alongside the suites; narrowing alone leaves the exclusion in force and runs nothing. +## Conformance reports + +Set `PROVIDER_TCK_REPORT_DIR` and each suite writes two files: an envelope conforming to the +[report schema][report-schema] in the specification, and the run's results as a +[Cucumber Messages][messages] stream. + +```console +$ PROVIDER_TCK_REPORT_DIR=./reports mvn test -Dtest='Flagd*TckTest' +$ ls reports/ +flagd-in-process.json flagd-in-process.ndjson flagd-rpc.json flagd-rpc.ndjson +``` + +`-Dprovider.tck.report.dir=...` does the same thing and is often easier to pass through Maven. The +environment variable is the portable spelling — every language's TCK reads it, so one cross-language +CI job can set one thing. + +It is an environment variable rather than a method on `ProviderTckHarness` so that emitting a report +is a property of the run and not of the code: CI sets it, a developer running the suite locally does +not, and no adopter changes a line to publish one. Unset means no report, which is not an error. +Several suites in one JVM each write their own pair, so flagd's two resolvers do not collide. + +### The results are not a format this project defines + +The `.ndjson` is a Cucumber Messages stream, produced by Cucumber's own `MessageFormatter` — the +same class the built-in `message:` plugin instantiates, so the bytes are what +`--plugin message:...` would have written. It already carries everything a per-scenario report would +have had to invent: the outcome of every scenario, its tags including any set on an individual +`Examples` block, an exact Scenario Outline row identity, and the source of every feature that ran. + +The plugin exists rather than the built-in one because a `@ConfigurationParameter` value is a +compile-time constant, so the built-in plugin's path cannot be derived from the directory the run +asked for — and flagd's two suites would write to the same file. + +Reading it needs no special tooling, but it does need one thing understood: **a scenario's outcome +is the most severe result among its steps**, hooks included. `testCaseFinished` carries no status of +its own. That is what makes a capability-gated skip truthful, because the aborted `@Before` hook +contributes a `SKIPPED` result that outranks every step it stopped from running. + +```console +$ jq -c 'select(.testStepFinished) | .testStepFinished + | {c: .testCaseStartedId, s: .testStepResult.status}' reports/flagd-rpc.ndjson \ + | jq -s 'group_by(.c) | map({s: (map(.s) | if any(. == "FAILED") then "FAILED" + elif any(. == "SKIPPED") then "SKIPPED" + else "PASSED" end)}) + | group_by(.s) | map({(.[0].s): length}) | add' +{ + "PASSED": 28, + "SKIPPED": 1 +} +``` + +The [`cucumber-query`](https://github.com/cucumber/messages/tree/main/java) helpers do this properly +and in several languages; the above is only to show that the fact is in the file. + +### What identifies a scenario + +`pickle.astNodeIds`. For a scenario compiled from a Scenario Outline it is +`[scenario id, table row id]`, and the second entry resolves in the `gherkinDocument` message to the +`Examples` row the scenario was built from. Feature and name are not enough — the type-mismatch +matrix in `errors.feature` is eleven rows sharing one name — and this is exact rather than derived: + +```console +$ jq -c 'select(.pickle) | .pickle + | select(.name == "Requesting the wrong type returns the code default") + | {id, row: .astNodeIds[1]}' reports/flagd-rpc.ndjson | head -3 +{"id":"6c8debd2-...","row":"ab8b4a4b-..."} +{"id":"a7c76b0a-...","row":"63d6d6c8-..."} +{"id":"fecd333d-...","row":"bbd7f5ee-..."} +``` + +An earlier version of this module reverse-engineered the same fact by re-parsing the feature source +and matching a pickle's reported line number against the Examples tables. The stream states it +outright, which is the whole argument for a standard format over one we maintain. + +### What the envelope is for + +A Messages stream cannot say what it was a test *of*. The envelope carries the four things no +standard results format identifies: + +- **`provider`** — what the provider calls itself through its own metadata, not the suite name. The + suite name is chosen to read well in a failure message (`flagd-rpc`), which makes it the + *configuration*, and it is reported as such. One provider with two materially different modes + produces two reports that are not interchangeable. Derived from the suite class name + (`FlagdInProcessTckTest` → `flagd-in-process`); override `ProviderTckHarness.configuration()`. +- **`sdk`** — read from the classpath rather than declared, because the TCK depends on an SDK version + *range* so that adopting it can never force an upgrade. What a consumer actually ran against is + only knowable at runtime. +- **`tck`** — which implementation asked the questions, and `specRevision`, the open-feature/spec + commit the packaged artifacts came from. Baked into the JAR at build time from this module's POM: + the artifacts travel in the JAR, the repository they came from does not. The executed Gherkin no + longer rests on that pin alone — the stream carries the `source` of every feature, so it can be + diffed against the revision — but the pin is what identifies the two artifacts the stream does not + carry, `flags/canonical-flags.json` and `openapi/control-api.yaml`. +- **`declaration`** — the capability set the provider claims. This is an **input** to reading the + results, not a summary of them, which is why it cannot be derived from the stream. The stream says + a scenario was skipped; only the declaration says whether that is because the provider declines the + capability it needed. Given the declaration and a scenario's tags — both present — the reason for + each skip follows, so it does not have to be transported per scenario. + +`knownDeviations` is the one thing neither the stream nor the declaration can express: whether a +withheld capability is a limitation or a bug. See +[Saying that a withheld capability is a defect](#saying-that-a-withheld-capability-is-a-defect). + +`results.digest` covers the `.ndjson`, so a consumer that fetched the two separately can tell that +what it has is what the envelope describes. + +### What the report is for + +This suite promises that a scenario skipped for an undeclared capability is reported as skipped with +the reason and *never* as passed — and a promise is not a check. The stream records every scenario +individually, so a consumer can verify the rule instead of trusting a runner's headline number. Go's +runner counts capability-gated skips in its **passed** tally, which is exactly the failure mode this +makes impossible to hide. + +Every scenario appears exactly once, whatever happened to it. A report that quietly omitted the +scenarios it did not run would satisfy every rule above and still mislead, because a reader would +have no way to know how many questions went unasked. `ConformanceReportPluginTest` runs a fixture +suite through the real Cucumber engine and asserts both properties over the emitted stream. + +[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json +[messages]: https://github.com/cucumber/messages + ## Extending it A provider with features of its own — flagd's `fractional` targeting, a vendor's proprietary mode — diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index 99e5987db..385f954cb 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -25,6 +25,32 @@ --> false + + 0bedacc22489697ccfbe1f5f4c5ae7649a5be457 3.27.7 4.3.0 2.22.1 @@ -109,6 +135,29 @@ cucumber-junit-platform-engine + + + io.cucumber + cucumber-core + + + + io.cucumber + messages + + + io.cucumber @@ -217,6 +266,23 @@ + + + + src/main/resources + false + + + src/main/resources-filtered + true + + + + + + io.cucumber + gherkin + compile + + + + + org.junit.jupiter + junit-jupiter-api + compile + + io.cucumber diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java new file mode 100644 index 000000000..58e63d5e1 --- /dev/null +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java @@ -0,0 +1,292 @@ +package dev.openfeature.contrib.tools.providertck; + +import io.cucumber.junit.platform.engine.Constants; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.TreeSet; +import java.util.concurrent.ConcurrentHashMap; +import org.junit.jupiter.api.Assumptions; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.BeforeEachCallback; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.extension.ExtensionContext; +import org.junit.platform.engine.TestSource; +import org.junit.platform.engine.support.descriptor.ClasspathResourceSource; +import org.junit.platform.launcher.TestIdentifier; +import org.junit.platform.launcher.TestPlan; + +/** + * Fails a suite that is set up to run less than the whole canonical scenario set. + * + *

Everything else in this module makes the canonical set easy to extend safely. Nothing makes it + * hard to shrink, and shrinking it is the failure that matters: a run that asks twenty-seven of the + * twenty-nine questions and reports success is indistinguishable, in every artifact it produces, from + * one that asked all twenty-nine. The known ways to get there are a feature file dropped into + * {@code gherkin/}, a {@code cucumber.filter.tags} or {@code cucumber.filter.name} expression, and + * selectors or glue overridden in a consuming module's {@code junit-platform.properties}. + * + *

Selected by {@link ProviderTckTest} as an ordinary JUnit Jupiter test, so a reduced set + * fails the build the way any other failing test does. A {@link + * org.junit.platform.launcher.TestExecutionListener} cannot do that job: the JUnit Platform catches + * and logs whatever a listener throws, which is exactly the silent pass being guarded against. + * + *

Two kinds of evidence, both available before a single scenario has run: + * + *

    + *
  • the discovered test plan, captured by {@link TckSuiteListener}. Which + * scenarios a suite selected is settled at discovery, so a canonical file that was shadowed, + * replaced or added to is visible there. + *
  • the effective filter configuration, read through this test's own + * {@link ExtensionContext}. Cucumber applies {@code cucumber.filter.tags} as a skip at + * execution rather than as a discovery filter — a filtered scenario is in the plan and never + * runs — so the plan cannot show it and the configured expression has to be read directly. The + * Jupiter engine inside the suite resolves configuration from the same sources the Cucumber + * engine does, whether the value came from a system property, a + * {@code junit-platform.properties} or an annotation on the suite. + *
+ * + *

Checking the setup rather than counting afterwards is not a compromise made for convenience. A + * count is only complete once the last scenario has finished, and the only hook that runs there is a + * listener, which cannot fail anything. Both kinds of evidence are settled before the first scenario + * runs, so the check itself takes no measurable time and needs no Compose stack — though where its + * result appears in a run depends on the order the JUnit Platform happens to execute the two engines + * in, which is not specified. + * + *

Extension scenarios are ignored entirely. The check is defined over {@code gherkin/}, so an + * adopter's {@code extensions/} scenarios can neither stand in for a canonical scenario nor look + * like a spurious one. + * + *

Separable from the rest of the extension work by design: it guards a bypass rather than enabling + * anything, and dropping it leaves the extension point unaffected. + */ +@ExtendWith(CanonicalScenarioGuard.CaptureConfiguration.class) +public final class CanonicalScenarioGuard { + + /** + * System property that downgrades this check to a skip. + * + *

An escape hatch is necessary rather than a weakness. Running one scenario with + * {@code -Dcucumber.filter.tags=@events} is routine while debugging a provider, and a check that + * made that impossible would be switched off permanently instead of temporarily. It produces a + * skip rather than a pass, so the run says out loud that its canonical set was not verified. + */ + public static final String PARTIAL_PROPERTY = "provider.tck.partial"; + + /** Environment variable equivalent of {@link #PARTIAL_PROPERTY}. */ + public static final String PARTIAL_ENV = "PROVIDER_TCK_PARTIAL"; + + /** Cucumber configuration keys that stop a discovered scenario from running. */ + private static final String[] FILTER_KEYS = { + Constants.FILTER_TAGS_PROPERTY_NAME, Constants.FILTER_NAME_PROPERTY_NAME + }; + + /** + * What each TCK suite in a discovered plan selected under {@code gherkin/}. + * + *

Keyed by suite class name and accumulated rather than replaced. A suite discovers its + * children through a launcher of its own, so this is called more than once per build with plans + * of differing scope, and a plan containing no TCK suite says nothing about the ones already + * recorded. + */ + private static final Map> DISCOVERED = new ConcurrentHashMap<>(); + + private static final String CANONICAL_PREFIX = ProviderTck.FEATURES + "/"; + + /** + * Records what the suites in a discovered plan will run. + * + * @param plan the plan about to be executed + */ + static void observe(TestPlan plan) { + for (TestIdentifier root : plan.getRoots()) { + collectSuites(plan, root); + } + } + + /** Forgets what has been observed, so a test can drive the guard over a plan of its own. */ + static void forget() { + DISCOVERED.clear(); + } + + /** + * Returns what each suite in the observed plans selected under {@code gherkin/}. + * + * @return canonical scenarios by suite class name + */ + static Map> discovered() { + return Collections.unmodifiableMap(new LinkedHashMap<>(DISCOVERED)); + } + + /** Asserts that this run will execute the whole canonical scenario set. */ + @Test + @DisplayName("the canonical scenario set was not reduced") + void theCanonicalScenarioSetWasNotReduced() { + if (partialRunAllowed()) { + Assumptions.abort("Skipped: " + PARTIAL_PROPERTY + " is set, so the canonical scenario set was not " + + "verified. This run is not a conformance run."); + } + + String problems = check(CanonicalScenarios.shipped(), discovered(), CaptureConfiguration.filters()); + if (problems != null) { + throw new AssertionError(problems); + } + } + + /** + * Reports every way this run falls short of the canonical set. + * + * @param canonical the canonical scenario set, as the TCK ships it + * @param discovered what each suite selected under {@code gherkin/} + * @param filters configured Cucumber filters, by configuration key + * @return a report of every problem found, or {@code null} when there is none + */ + static String check( + Set canonical, + Map> discovered, + Map filters) { + List problems = new ArrayList<>(); + + for (Map.Entry filter : filters.entrySet()) { + problems.add(filter.getKey() + " is set to '" + filter.getValue() + + "'. Cucumber applies it by skipping scenarios that would otherwise have run, so a " + + "conformance run cannot be filtered. Decline capabilities your provider does not have " + + "through capabilities() instead — those scenarios are reported as skipped with a reason, " + + "which a filtered one is not. To filter anyway while debugging, set -D" + PARTIAL_PROPERTY + + "=true and accept that the run is not a conformance run."); + } + + if (discovered.isEmpty()) { + problems.add("No TCK suite was found in the JUnit test plan, so it cannot be shown that the canonical " + + "scenarios will run. This normally means TckSuiteListener was not auto-registered — the same " + + "condition that makes harness discovery fall back to ServiceLoader. Enable JUnit Platform " + + "listener auto-registration, or set -D" + PARTIAL_PROPERTY + "=true to accept a run whose " + + "canonical set is unverified."); + } + + for (Map.Entry> suite : discovered.entrySet()) { + Set missing = new TreeSet<>(canonical); + missing.removeAll(suite.getValue()); + if (!missing.isEmpty()) { + problems.add(suite.getKey() + " left out " + missing.size() + " of " + canonical.size() + + " canonical scenarios: " + missing + + ". A conformance run executes the canonical set in full. Look for a selector or glue " + + "override in junit-platform.properties, a cucumber.features property, or a feature file " + + "of your own in " + CANONICAL_PREFIX + " shadowing a canonical one. Scenarios your " + + "provider cannot support are declined through capabilities(), which reports them as " + + "skipped rather than removing them."); + } + + Set unexpected = new TreeSet<>(suite.getValue()); + unexpected.removeAll(canonical); + if (!unexpected.isEmpty()) { + problems.add(suite.getKey() + " selected " + unexpected.size() + " scenario(s) under " + + CANONICAL_PREFIX + " that this TCK does not ship: " + unexpected + + ". That directory is the canonical set, and adding to it changes what conformance means. " + + "Put your own scenarios in " + ProviderTck.EXTENSIONS + "/ instead, where they run in the " + + "same suite and the same backend lifecycle."); + } + } + + return problems.isEmpty() ? null : String.join(System.lineSeparator() + System.lineSeparator(), problems); + } + + /** + * Returns the Cucumber filters this run is configured with. + * + * @param context the executing test's context, which resolves configuration the way the Cucumber + * engine beside it does + * @return configured filters by configuration key, empty when the run is unfiltered + */ + static Map filtersIn(ExtensionContext context) { + Map configured = new LinkedHashMap<>(); + for (String key : FILTER_KEYS) { + context.getConfigurationParameter(key) + .map(String::trim) + .filter(value -> !value.isEmpty()) + .ifPresent(value -> configured.put(key, value)); + } + return configured; + } + + /** Returns whether the run has declared itself partial. */ + private static boolean partialRunAllowed() { + return isTrue(System.getProperty(PARTIAL_PROPERTY)) || isTrue(System.getenv(PARTIAL_ENV)); + } + + private static boolean isTrue(String value) { + return value != null && ("".equals(value.trim()) || Boolean.parseBoolean(value.trim())); + } + + private static void collectSuites(TestPlan plan, TestIdentifier identifier) { + Optional> suite = TckSuiteListener.harnessClassOf(identifier); + if (suite.isPresent()) { + Set canonical = new LinkedHashSet<>(); + collectCanonical(plan, identifier, canonical); + DISCOVERED.put(suite.get().getName(), Collections.unmodifiableSet(canonical)); + return; + } + for (TestIdentifier child : plan.getChildren(identifier)) { + collectSuites(plan, child); + } + } + + private static void collectCanonical(TestPlan plan, TestIdentifier identifier, Set into) { + if (identifier.isTest()) { + canonicalRefOf(identifier).ifPresent(into::add); + } + for (TestIdentifier child : plan.getChildren(identifier)) { + collectCanonical(plan, child, into); + } + } + + /** + * Returns the canonical scenario a test identifier stands for, if it is one. + * + *

A Cucumber scenario discovered from a classpath resource carries a + * {@link ClasspathResourceSource} naming the feature resource and the scenario's line — for a + * Scenario Outline, the line of the {@code Examples} row it was compiled from, which is what makes + * each row count separately. + */ + private static Optional canonicalRefOf(TestIdentifier identifier) { + Optional source = identifier.getSource(); + if (!source.isPresent() || !(source.get() instanceof ClasspathResourceSource)) { + return Optional.empty(); + } + ClasspathResourceSource resource = (ClasspathResourceSource) source.get(); + String name = resource.getClasspathResourceName(); + if (!name.startsWith(CANONICAL_PREFIX)) { + return Optional.empty(); + } + return resource.getPosition() + .map(position -> new CanonicalScenarios.Ref(name, position.getLine(), identifier.getDisplayName())); + } + + /** + * Hands the guard the configuration of the engine it is running in. + * + *

Jupiter resolves configuration parameters for an extension, not for a test method, so the + * context is captured here rather than injected. Inside a suite it carries the suite's own + * {@code @ConfigurationParameter} values as well as the system properties and + * {@code junit-platform.properties} the Cucumber engine beside it reads. + */ + static final class CaptureConfiguration implements BeforeEachCallback { + + private static volatile Map filters = Collections.emptyMap(); + + @Override + public void beforeEach(ExtensionContext context) { + filters = Collections.unmodifiableMap(filtersIn(context)); + } + + static Map filters() { + return filters; + } + } +} diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarios.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarios.java new file mode 100644 index 000000000..8bb56b60d --- /dev/null +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarios.java @@ -0,0 +1,260 @@ +package dev.openfeature.contrib.tools.providertck; + +import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; +import io.cucumber.gherkin.GherkinParser; +import io.cucumber.messages.types.Envelope; +import io.cucumber.messages.types.Examples; +import io.cucumber.messages.types.Feature; +import io.cucumber.messages.types.FeatureChild; +import io.cucumber.messages.types.GherkinDocument; +import io.cucumber.messages.types.Rule; +import io.cucumber.messages.types.RuleChild; +import io.cucumber.messages.types.Scenario; +import io.cucumber.messages.types.TableRow; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.UncheckedIOException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.file.DirectoryStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.security.CodeSource; +import java.util.Collections; +import java.util.Enumeration; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; +import java.util.jar.JarEntry; +import java.util.jar.JarFile; +import java.util.stream.Stream; + +/** + * The canonical scenario set, as this artifact ships it. + * + *

Read from this artifact's own code source — the JAR or {@code target/classes} that + * {@link ProviderTck} was loaded from — rather than through the classloader. That is the point of the + * class: a feature file placed in {@code gherkin/} on another classpath root shadows the canonical + * one of the same name, and a check that read the shadowed copy back through + * {@code getResource("gherkin/errors.feature")} would be checking the replacement against itself. + * The code source is the only view of {@code gherkin/} that an adopter's classpath cannot alter. + * + *

A scenario is identified by its resource name and its line, which is what a Cucumber + * {@code ClasspathResourceSource} in the JUnit test plan carries. For a Scenario Outline that line is + * the {@code Examples} row, so each row counts individually — which is necessary, because eleven rows + * of the type-mismatch matrix share one name. + * + *

What this does not establish is that the canonical files contain what they + * should. A replacement file that happened to place its scenarios on the same lines would satisfy the + * comparison. Content is covered elsewhere and better: the results stream carries the {@code source} + * of every feature that executed, and the report's {@code tck.specRevision} says which revision it + * should match. + */ +final class CanonicalScenarios { + + private static final String FEATURE_SUFFIX = ".feature"; + + private static volatile Set shipped; + + private CanonicalScenarios() {} + + /** + * Returns every scenario the canonical feature files in this artifact compile to. + * + * @return the canonical scenario set, never empty + * @throws IllegalStateException if this artifact's own feature files cannot be read + */ + static Set shipped() { + Set cached = shipped; + if (cached == null) { + synchronized (CanonicalScenarios.class) { + if (shipped == null) { + shipped = Collections.unmodifiableSet(read()); + } + cached = shipped; + } + } + return cached; + } + + /** Reads and compiles the canonical feature files out of the code source this class came from. */ + @SuppressFBWarnings( + value = "PATH_TRAVERSAL_IN", + justification = "The path is this class's own code source, reported by the JVM. Nothing outside the " + + "running artifact can influence it, and reading it from anywhere else would defeat the point " + + "of the class") + private static Set read() { + CodeSource codeSource = ProviderTck.class.getProtectionDomain().getCodeSource(); + URL location = codeSource == null ? null : codeSource.getLocation(); + if (location == null) { + throw new IllegalStateException("The provider-tck code source is not visible to this JVM, so the " + + "canonical scenario set cannot be established. Set -D" + CanonicalScenarioGuard.PARTIAL_PROPERTY + + "=true to run without the canonical-set check, understanding that the run is then not a " + + "conformance run."); + } + + Path path; + try { + path = Paths.get(location.toURI()); + } catch (URISyntaxException | IllegalArgumentException e) { + throw new IllegalStateException( + "The provider-tck code source " + location + " is not a file, so the " + + "canonical scenario set cannot be established.", + e); + } + + Set refs = new LinkedHashSet<>(); + try { + if (Files.isDirectory(path)) { + fromDirectory(path, refs); + } else { + fromJar(path, refs); + } + } catch (IOException e) { + throw new UncheckedIOException("Could not read the canonical feature files from " + path, e); + } + + if (refs.isEmpty()) { + throw new IllegalStateException("No canonical scenarios found in " + ProviderTck.FEATURES + "/ of the " + + "provider-tck artifact at " + path + ". The artifact is not intact."); + } + return refs; + } + + private static void fromDirectory(Path root, Set refs) throws IOException { + Path features = root.resolve(ProviderTck.FEATURES); + if (!Files.isDirectory(features)) { + return; + } + try (DirectoryStream entries = Files.newDirectoryStream(features, "*" + FEATURE_SUFFIX)) { + for (Path entry : entries) { + String name = ProviderTck.FEATURES + "/" + entry.getFileName(); + collect(name, Files.readAllBytes(entry), refs); + } + } + } + + private static void fromJar(Path jar, Set refs) throws IOException { + try (JarFile file = new JarFile(jar.toFile())) { + Enumeration entries = file.entries(); + while (entries.hasMoreElements()) { + JarEntry entry = entries.nextElement(); + String name = entry.getName(); + if (!entry.isDirectory() + && name.startsWith(ProviderTck.FEATURES + "/") + && name.endsWith(FEATURE_SUFFIX)) { + try (InputStream in = file.getInputStream(entry)) { + collect(name, readAll(in), refs); + } + } + } + } + } + + private static byte[] readAll(InputStream in) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + byte[] buffer = new byte[8192]; + int read; + while ((read = in.read(buffer)) != -1) { + out.write(buffer, 0, read); + } + return out.toByteArray(); + } + + /** + * Compiles one feature file to the scenarios Cucumber would run from it. + * + *

The Gherkin document is walked rather than the pickles it produces, because a pickle carries + * no line number — and the line is what the JUnit test plan identifies a scenario by. The walk is + * the same rule Gherkin's own compiler applies: a Scenario Outline yields one scenario per + * {@code Examples} body row, anything else yields one at its own line. + */ + private static void collect(String resource, byte[] content, Set refs) { + GherkinParser parser = GherkinParser.builder() + .includeSource(false) + .includePickles(false) + .includeGherkinDocument(true) + .build(); + + try (Stream envelopes = parser.parse(resource, content)) { + envelopes.forEach(envelope -> envelope.getGherkinDocument() + .flatMap(GherkinDocument::getFeature) + .ifPresent(feature -> collectFeature(resource, feature, refs))); + } + } + + private static void collectFeature(String resource, Feature feature, Set refs) { + for (FeatureChild child : feature.getChildren()) { + child.getScenario().ifPresent(scenario -> collectScenario(resource, scenario, refs)); + child.getRule().ifPresent(rule -> collectRule(resource, rule, refs)); + } + } + + private static void collectRule(String resource, Rule rule, Set refs) { + for (RuleChild child : rule.getChildren()) { + child.getScenario().ifPresent(scenario -> collectScenario(resource, scenario, refs)); + } + } + + private static void collectScenario(String resource, Scenario scenario, Set refs) { + List examples = scenario.getExamples(); + if (examples.isEmpty()) { + refs.add(new Ref(resource, scenario.getLocation().getLine(), scenario.getName())); + return; + } + for (Examples block : examples) { + for (TableRow row : block.getTableBody()) { + refs.add(new Ref(resource, row.getLocation().getLine(), scenario.getName())); + } + } + } + + /** + * One scenario, identified the way the JUnit test plan identifies it. + * + *

The name is carried for the failure message only; two scenarios are the same scenario when + * they are at the same line of the same classpath resource. + */ + static final class Ref implements Comparable { + + private final String resource; + private final long line; + private final String name; + + Ref(String resource, long line, String name) { + this.resource = resource; + this.line = line; + this.name = name; + } + + @Override + public boolean equals(Object other) { + if (this == other) { + return true; + } + if (!(other instanceof Ref)) { + return false; + } + Ref that = (Ref) other; + return line == that.line && resource.equals(that.resource); + } + + @Override + public int hashCode() { + return 31 * resource.hashCode() + Long.hashCode(line); + } + + @Override + public int compareTo(Ref other) { + int byResource = resource.compareTo(other.resource); + return byResource != 0 ? byResource : Long.compare(line, other.line); + } + + @Override + public String toString() { + return resource + ":" + line + (name == null || name.isEmpty() ? "" : " (" + name + ")"); + } + } +} diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ProviderTckTest.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ProviderTckTest.java index 7773f7e52..72aac4101 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ProviderTckTest.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ProviderTckTest.java @@ -3,6 +3,7 @@ import io.cucumber.junit.platform.engine.Constants; import org.junit.platform.suite.api.ConfigurationParameter; import org.junit.platform.suite.api.IncludeEngines; +import org.junit.platform.suite.api.SelectClasses; import org.junit.platform.suite.api.SelectClasspathResource; import org.junit.platform.suite.api.Suite; @@ -50,6 +51,13 @@ * this JAR holding nothing but a README, because {@link SelectClasspathResource} on a resource that * exists on no classpath root is a discovery error rather than an empty selection. * + *

One JUnit Jupiter test runs alongside the scenarios: {@link CanonicalScenarioGuard}, which + * fails a suite whose canonical set has been reduced — by a tag filter, a selector override, or a + * feature file shadowing a canonical one. It is why {@code junit-jupiter} is in the engine list. It + * inspects the discovered test plan and the run's filter configuration, both of which are settled + * before the first scenario, so it costs nothing and does not depend on the order the engines happen + * to run in. + * *

Every value these annotations carry is named in {@link ProviderTck}. An adopter who does write a * {@code @ConfigurationParameter} of their own composes from those constants — * {@code ProviderTck.ALL_GLUE + ",com.vendor.steps"} — rather than restating this configuration as a @@ -58,11 +66,14 @@ * @see ProviderTckHarness * @see ContainerizedProviderTckTest * @see ProviderTck + * @see ConformanceReportPlugin + * @see CanonicalScenarioGuard */ @Suite -@IncludeEngines("cucumber") +@IncludeEngines({"cucumber", "junit-jupiter"}) @SelectClasspathResource(ProviderTck.FEATURES) @SelectClasspathResource(ProviderTck.EXTENSIONS) +@SelectClasses(CanonicalScenarioGuard.class) @ConfigurationParameter(key = Constants.PLUGIN_PROPERTY_NAME, value = ProviderTck.PLUGINS) @ConfigurationParameter( key = Constants.PARALLEL_EXECUTION_ENABLED_PROPERTY_NAME, diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckSuiteListener.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckSuiteListener.java index 8116deff0..30809f0fe 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckSuiteListener.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckSuiteListener.java @@ -6,6 +6,7 @@ import org.junit.platform.engine.support.descriptor.ClassSource; import org.junit.platform.launcher.TestExecutionListener; import org.junit.platform.launcher.TestIdentifier; +import org.junit.platform.launcher.TestPlan; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -35,6 +36,21 @@ public class TckSuiteListener implements TestExecutionListener { private static volatile Class current; + /** + * Records what each TCK suite in the plan is about to run, for {@link CanonicalScenarioGuard}. + * + *

Read from the plan before any of it executes, which is as early as the selected scenarios + * can be known: selectors and glue are resolved by then, so a canonical file that was shadowed, + * replaced or added to is already visible. The guard needs nothing from the run itself and no + * backend. + * + * @param testPlan the plan about to be executed + */ + @Override + public void testPlanExecutionStarted(TestPlan testPlan) { + CanonicalScenarioGuard.observe(testPlan); + } + @Override public void executionStarted(TestIdentifier testIdentifier) { harnessClassOf(testIdentifier).ifPresent(suite -> { diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java new file mode 100644 index 000000000..fff4721e5 --- /dev/null +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java @@ -0,0 +1,229 @@ +package dev.openfeature.contrib.tools.providertck; + +import static org.assertj.core.api.Assertions.assertThat; + +import io.cucumber.junit.platform.engine.Constants; +import java.util.Collections; +import java.util.Iterator; +import java.util.LinkedHashSet; +import java.util.Map; +import java.util.Set; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.platform.engine.discovery.DiscoverySelectors; +import org.junit.platform.launcher.EngineFilter; +import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder; +import org.junit.platform.launcher.core.LauncherFactory; +import org.junit.platform.launcher.listeners.SummaryGeneratingListener; +import org.junit.platform.launcher.listeners.TestExecutionSummary; + +/** + * A run that quietly asks fewer questions must not be able to report success. + * + *

Two levels here, deliberately. The comparison itself is checked directly, because a failure + * message is the whole product of a guard and it has to name the scenario that went missing. The + * wiring is checked by running the guard through the JUnit Platform under a real filter + * configuration, because "the guard sees what the Cucumber engine beside it sees" is a claim about + * the platform rather than about this code. + * + *

No scenario is executed anywhere in this file. The guard works from the discovered plan and the + * run's configuration, which is what lets it be checked — and, in a real build, fail — without a + * backend. + */ +class CanonicalScenarioGuardTest { + + private static final Map UNFILTERED = Collections.emptyMap(); + + @BeforeEach + void forgetPreviousPlans() { + CanonicalScenarioGuard.forget(); + } + + @Test + @DisplayName("the canonical set is read from this artifact, outline row by outline row") + void theCanonicalSetIsReadFromTheArtifact() { + Set canonical = CanonicalScenarios.shipped(); + + assertThat(canonical).isNotEmpty(); + assertThat(canonical) + .as("every canonical scenario comes from a packaged feature file") + .allSatisfy(ref -> assertThat(ref.toString()).startsWith(ProviderTck.FEATURES + "/")); + + // The eleven rows of the type-mismatch matrix share one name, so a set that counted scenarios + // by name would see one of them. Each Examples row is its own line and its own entry. + assertThat(canonical) + .filteredOn(ref -> ref.toString().contains("Requesting the wrong type returns the code default")) + .hasSize(11); + } + + @Test + @DisplayName("a suite that selects the whole canonical set, unfiltered, passes") + void anIntactSuitePasses() { + CanonicalScenarioGuard.observe(discoverFixtureSuite()); + + assertThat(CanonicalScenarioGuard.check( + CanonicalScenarios.shipped(), CanonicalScenarioGuard.discovered(), UNFILTERED)) + .isNull(); + } + + @Test + @DisplayName("extension scenarios neither count towards the canonical set nor disturb it") + void extensionScenariosAreIgnored() { + CanonicalScenarioGuard.observe(discoverFixtureSuite()); + + Set observed = + CanonicalScenarioGuard.discovered().get(TckSuiteFixture.class.getName()); + + // The fixture suite does select an extension feature — ExtensionPointTest checks that it does + // — and none of it reaches the guard. + assertThat(observed) + .as("the guard is defined over %s/ alone", ProviderTck.FEATURES) + .isEqualTo(CanonicalScenarios.shipped()) + .allSatisfy(ref -> assertThat(ref.toString()).doesNotContain(ProviderTck.EXTENSIONS + "/")); + } + + @Test + @DisplayName("a scenario filter fails the run, whatever it would have excluded") + void aScenarioFilterFails() { + CanonicalScenarioGuard.observe(discoverFixtureSuite()); + + String problems = CanonicalScenarioGuard.check( + CanonicalScenarios.shipped(), + CanonicalScenarioGuard.discovered(), + Collections.singletonMap(Constants.FILTER_TAGS_PROPERTY_NAME, "not @events")); + + // Cucumber applies a tag filter by skipping scenarios during execution, so the plan still + // contains them and only the configured expression shows what will be left out. The guard + // therefore rejects the filter itself rather than trying to predict what it matches. + assertThat(problems) + .isNotNull() + .contains(Constants.FILTER_TAGS_PROPERTY_NAME) + .contains("not @events") + .as("and it points at the mechanism that legitimately narrows a run") + .contains("capabilities()"); + } + + @Test + @DisplayName("the guard reads the filter the Cucumber engine beside it would read") + void theGuardReadsTheRunsFilterConfiguration() { + CanonicalScenarioGuard.observe(discoverFixtureSuite()); + + TestExecutionSummary filtered = runGuard(Constants.FILTER_TAGS_PROPERTY_NAME, "@events"); + + assertThat(filtered.getTestsFailedCount()) + .as("a filtered run fails at the guard, before a Compose stack is worth starting") + .isEqualTo(1); + assertThat(filtered.getFailures().get(0).getException()) + .hasMessageContaining(Constants.FILTER_TAGS_PROPERTY_NAME); + } + + @Test + @DisplayName("an unfiltered run of an intact suite passes the guard as executed") + void anIntactRunPassesTheGuard() { + CanonicalScenarioGuard.observe(discoverFixtureSuite()); + + assertThat(runGuard(null, null).getTestsSucceededCount()).isEqualTo(1); + } + + @Test + @DisplayName("a run that declares itself partial is skipped rather than passed") + void aPartialRunIsSkipped() { + CanonicalScenarioGuard.observe(discoverFixtureSuite()); + + String previous = System.getProperty(CanonicalScenarioGuard.PARTIAL_PROPERTY); + System.setProperty(CanonicalScenarioGuard.PARTIAL_PROPERTY, "true"); + try { + TestExecutionSummary summary = runGuard(Constants.FILTER_TAGS_PROPERTY_NAME, "@events"); + + // Aborted rather than skipped: the test ran and declined to conclude, which is what an + // assumption produces and what Surefire and the JUnit reports show as skipped. + assertThat(summary.getTestsAbortedCount()) + .as("a skip says the canonical set was not verified; a pass would claim it was") + .isEqualTo(1); + assertThat(summary.getTestsSucceededCount()).isZero(); + assertThat(summary.getTestsFailedCount()).isZero(); + } finally { + restore(CanonicalScenarioGuard.PARTIAL_PROPERTY, previous); + } + } + + @Test + @DisplayName("a feature file added to the canonical directory fails the run") + void anAddedCanonicalFeatureFails() { + Set withExtra = new LinkedHashSet<>(CanonicalScenarios.shipped()); + withExtra.add(new CanonicalScenarios.Ref(ProviderTck.FEATURES + "/vendor.feature", 7, "A vendor scenario")); + + String problems = CanonicalScenarioGuard.check( + CanonicalScenarios.shipped(), + Collections.singletonMap(TckSuiteFixture.class.getName(), withExtra), + UNFILTERED); + + assertThat(problems) + .isNotNull() + .contains(ProviderTck.FEATURES + "/vendor.feature") + .as("and it says where the scenario should have gone") + .contains(ProviderTck.EXTENSIONS + "/"); + } + + @Test + @DisplayName("a canonical file replaced by another of the same name fails the run") + void aShadowedCanonicalFileFails() { + // What the separate extension directory exists to prevent, expressed as what the guard sees + // if it happens anyway: the replacement compiles to different scenarios, so entries of the + // canonical set go missing. + Set shadowed = new LinkedHashSet<>(CanonicalScenarios.shipped()); + Iterator entries = shadowed.iterator(); + CanonicalScenarios.Ref removed = entries.next(); + entries.remove(); + + String problems = CanonicalScenarioGuard.check( + CanonicalScenarios.shipped(), + Collections.singletonMap(TckSuiteFixture.class.getName(), shadowed), + UNFILTERED); + + assertThat(problems).isNotNull().contains(removed.toString()).contains("shadowing"); + } + + @Test + @DisplayName("a plan with no TCK suite in it is a failure, not a pass") + void anUnobservedRunFails() { + Map> nothing = Collections.emptyMap(); + + assertThat(CanonicalScenarioGuard.check(CanonicalScenarios.shipped(), nothing, UNFILTERED)) + .as("an unverifiable conformance run is not a conformance run") + .isNotNull() + .contains("TckSuiteListener") + .contains(CanonicalScenarioGuard.PARTIAL_PROPERTY); + } + + /** Discovers the real suite configuration, without executing anything. */ + private static org.junit.platform.launcher.TestPlan discoverFixtureSuite() { + return LauncherFactory.create() + .discover(LauncherDiscoveryRequestBuilder.request() + .selectors(DiscoverySelectors.selectClass(TckSuiteFixture.class)) + .build()); + } + + /** Runs the guard itself through the JUnit Platform, optionally under a Cucumber filter. */ + private static TestExecutionSummary runGuard(String key, String value) { + LauncherDiscoveryRequestBuilder request = LauncherDiscoveryRequestBuilder.request() + .selectors(DiscoverySelectors.selectClass(CanonicalScenarioGuard.class)) + .filters(EngineFilter.includeEngines("junit-jupiter")); + if (key != null) { + request.configurationParameter(key, value); + } + + SummaryGeneratingListener summary = new SummaryGeneratingListener(); + LauncherFactory.create().execute(request.build(), summary); + return summary.getSummary(); + } + + private static void restore(String key, String previous) { + if (previous == null) { + System.clearProperty(key); + } else { + System.setProperty(key, previous); + } + } +} From 327efb2bf6a594225960b56ecca6eb99fa6a52aa Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 12:32:43 +0200 Subject: [PATCH 03/22] chore(provider-tck): follow the spec submodule bump in the reported revision The base moved the spec submodule twice: to ba002ce8 when it renamed the falsy canonical flags, and to fc99d5ac when it gated the reinitialisation scenario on @reinitialization. A report's tck.specRevision has to name the revision that actually produced its scenarios, so the pin follows -- left behind, every report from this branch would cite a revision predating both the flag names and the tag it evaluated. This is the invariant the property's comment states: provider-tck.spec.revision equals `git -C spec rev-parse HEAD`. Checked by eye again; worth automating, since this is the third bump it has had to follow. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index bfd17ab7e..070cc8ae4 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -50,7 +50,7 @@ pin already moves only by deliberate commit. Update both together; a mismatch means a report names a revision that did not produce its scenarios. --> - 0bedacc22489697ccfbe1f5f4c5ae7649a5be457 + fc99d5ace4da472a5fea0595fa4db8034bbbc769 3.27.7 4.3.0 2.22.1 From c5fa63ca309c8d6d9d26f0fcc1d07503f15d1b56 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Fri, 11 Sep 2026 18:52:10 +0200 Subject: [PATCH 04/22] fix(provider-tck): follow the capability model in the declaration test @large-integers is no longer set apart as "not applicable in Java": it is an ordinary declarable capability that a Java provider withholds, because the reason it cannot hold -- a 32-bit integer accessor -- is a property of the SDK and is recorded in Appendix F rather than in every report. So the maximal declaration this test asks for now includes it, and the assertion follows. The test's own point is unchanged and still holds: a reserved tag, which no scenario gates, cannot reach the declaration however the set was built. The emitted envelope is unaffected. declaration carries declared and nothing else, which is what the schema at spec 7f03f672 permits -- it sets additionalProperties: false, so a notApplicable member would now be rejected rather than merely unused. Signed-off-by: Simon Schrottner --- .../contrib/tools/tck/ConformanceReportPluginTest.java | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index f2fada79f..97e391b74 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -301,7 +301,13 @@ void aReservedCapabilityCannotReachTheDeclaration() { Capability.CONFIGURATION_CHANGE.tag(), Capability.OBJECT.tag(), Capability.UNAVAILABLE_INIT.tag(), - Capability.NUMERIC_COERCION.tag()); + Capability.NUMERIC_COERCION.tag(), + // @large-integers gates a scenario and is an ordinary declarable capability. + // No Java provider holds it, because the SDK's integer accessor is 32 bits, + // but that is a fact about the SDK recorded in Appendix F rather than a + // second kind of declaration, so the maximal claim includes it and a real + // harness withholds it. + Capability.LARGE_INTEGERS.tag()); } @Test From 2f12079acaf2de6e334de3eaa518f0731c7b65d0 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 08:30:18 +0200 Subject: [PATCH 05/22] chore(provider-tck): follow the spec submodule bump in the reported revision The base moved the pin to 26362f85, where @variants gates the variant assertions and @targeting stops being reserved. A report's tck.specRevision has to name the revision that actually produced its scenarios, so the property follows: left at fc99d5ac, every report from this branch would cite a revision whose evaluation.feature had twelve scenario instances rather than twenty-four, no @variants in its vocabulary, and no scenario passing an evaluation context. This is the invariant the property's comment states: provider-tck.spec.revision equals `git -C spec rev-parse HEAD`. Fourth bump it has had to follow by hand, and the argument for automating it has not got weaker. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index 070cc8ae4..58aea5a59 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -50,7 +50,7 @@ pin already moves only by deliberate commit. Update both together; a mismatch means a report names a revision that did not produce its scenarios. --> - fc99d5ace4da472a5fea0595fa4db8034bbbc769 + 26362f85b7fcd59b35b969e6feebee80e206b24f 3.27.7 4.3.0 2.22.1 From 837c89c7854cafc737e99aab6bc697b78a5e0937 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 08:30:18 +0200 Subject: [PATCH 06/22] test(provider-tck): follow the capability change in the declaration test @variants is new and @targeting is no longer reserved, and the maximal-declaration test asserts the declared list exactly, so both join it in enum order. The reserved case changes with it: "everything except X" spelt EnumSet.complementOf now sweeps up one reserved tag rather than two, so the overclaim fixture asks for @caching, which is the tag the refusal is about. Asking for @targeting would have tested nothing -- it is declarable now, so the declaration would have been accepted and the assertion would have failed for the wrong reason. The canonical count in the guard's javadoc and in the README follows the assets: fifty-two scenario instances. Prose only -- the guard counts nothing itself, it compares the discovered plan against CanonicalScenarios.shipped() and prints canonical.size() in the failure -- but a number that disagrees with the suite is exactly what makes a reader think the check is a count taken afterwards. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 4 ++-- .../tools/tck/CanonicalScenarioGuard.java | 6 ++--- .../tck/ConformanceReportPluginTest.java | 23 ++++++++++++------- 3 files changed, 20 insertions(+), 13 deletions(-) diff --git a/tools/tck/README.md b/tools/tck/README.md index a4890cb52..5a5580639 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -348,8 +348,8 @@ narrowing alone leaves the exclusion in force and runs nothing. ## The canonical set cannot be reduced Extending the suite is safe by convention. Shrinking it is what a conformance suite has to prevent, -because a run that asks twenty-seven of the twenty-nine questions and reports success is -indistinguishable, in every artifact it produces, from one that asked all twenty-nine. +because a run that asks fifty of the fifty-two questions and reports success is indistinguishable, in +every artifact it produces, from one that asked all fifty-two. `CanonicalScenarioGuard` is an ordinary JUnit test that the suite selects, and it fails the build if this run is set up to execute less than the canonical set: diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java index 58e63d5e1..b295591ae 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java @@ -26,9 +26,9 @@ * Fails a suite that is set up to run less than the whole canonical scenario set. * *

Everything else in this module makes the canonical set easy to extend safely. Nothing makes it - * hard to shrink, and shrinking it is the failure that matters: a run that asks twenty-seven of the - * twenty-nine questions and reports success is indistinguishable, in every artifact it produces, from - * one that asked all twenty-nine. The known ways to get there are a feature file dropped into + * hard to shrink, and shrinking it is the failure that matters: a run that asks fifty of the fifty-two + * questions and reports success is indistinguishable, in every artifact it produces, from one that + * asked all fifty-two. The known ways to get there are a feature file dropped into * {@code gherkin/}, a {@code cucumber.filter.tags} or {@code cucumber.filter.name} expression, and * selectors or glue overridden in a consuming module's {@code junit-platform.properties}. * diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index 97e391b74..a09ac9a3a 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -279,7 +279,7 @@ void theDeclarationListsWhatIsClaimed() { void aReservedCapabilityCannotReachTheDeclaration() { // The provider that claims the most is the case that used to break the rule: "everything" // spelt EnumSet.allOf, or "everything except X" spelt EnumSet.complementOf, collected the - // reserved tags along the way and published a claim about two capabilities no scenario + // reserved tags along the way and published a claim about capabilities no scenario // examines. So this asks the maximal declaration for its report. JsonNode declared = MAPPER.valueToTree(report(Capability.declarable())) .get("declaration") @@ -290,7 +290,7 @@ void aReservedCapabilityCannotReachTheDeclaration() { assertThat(tags) .as("a reserved tag gates nothing, so declaring it is a claim nothing can contradict") - .doesNotContain(Capability.TARGETING.tag(), Capability.CACHING.tag()); + .doesNotContain(Capability.CACHING.tag()); assertThat(tags) .as("and every capability some scenario does gate is still there") .containsExactly( @@ -300,6 +300,10 @@ void aReservedCapabilityCannotReachTheDeclaration() { Capability.STALE.tag(), Capability.CONFIGURATION_CHANGE.tag(), Capability.OBJECT.tag(), + // @variants gates the one outline that asserts a variant. 2.2.4 is a SHOULD + // and types.md types the field optional, so a backend with no variant + // concept withholds it and those rows are skipped with that reason. + Capability.VARIANTS.tag(), Capability.UNAVAILABLE_INIT.tag(), Capability.NUMERIC_COERCION.tag(), // @large-integers gates a scenario and is an ordinary declarable capability. @@ -307,7 +311,10 @@ void aReservedCapabilityCannotReachTheDeclaration() { // but that is a fact about the SDK recorded in Appendix F rather than a // second kind of declaration, so the maximal claim includes it and a real // harness withholds it. - Capability.LARGE_INTEGERS.tag()); + Capability.LARGE_INTEGERS.tag(), + // @targeting was reserved until targeting-key-flag's three scenarios + // arrived. It gates something now, so the maximal claim includes it. + Capability.TARGETING.tag()); } @Test @@ -316,11 +323,11 @@ void namingAReservedCapabilityFailsTheRun() { // Chosen over a warning: the declaration is the one part of the report no result can check, // and a report is read long after the log it would have been warned in has gone. Nothing is // lost by refusing, because no scenario carries the tag. - Set overclaimed = EnumSet.of(Capability.OBJECT, Capability.TARGETING); + Set overclaimed = EnumSet.of(Capability.OBJECT, Capability.CACHING); assertThatThrownBy(() -> metadata(overclaimed)) .isInstanceOf(IllegalArgumentException.class) - .hasMessageContaining(Capability.TARGETING.tag()) + .hasMessageContaining(Capability.CACHING.tag()) .hasMessageContaining("declarableExcept"); } @@ -328,13 +335,13 @@ void namingAReservedCapabilityFailsTheRun() { @DisplayName("\"everything except X\" means everything declarable except X") void declarableExceptYieldsOnlyDeclarableCapabilities() { assertThat(Capability.declarableExcept(Capability.STALE)) - .doesNotContain(Capability.STALE, Capability.TARGETING, Capability.CACHING) - .contains(Capability.OBJECT, Capability.NUMERIC_COERCION); + .doesNotContain(Capability.STALE, Capability.CACHING) + .contains(Capability.OBJECT, Capability.NUMERIC_COERCION, Capability.TARGETING); // The counterpart it replaces, and why it had to be replaced. assertThat(EnumSet.complementOf(EnumSet.of(Capability.STALE))) .as("complementOf is the complement of the enum, not of the declarable vocabulary") - .contains(Capability.TARGETING, Capability.CACHING); + .contains(Capability.CACHING); } @Test From 7f662037c8eeffbc35750026ea2df64c959ece43 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 12:28:18 +0200 Subject: [PATCH 07/22] chore(provider-tck): follow the disabled-flag capability in the report The reported spec revision has to move with the submodule pin, or a conformance report names a revision that did not produce its scenarios. 009afe06 is where the @disabled-flags outline and its four flags come from. The maximal-declaration test gains @disabled-flags in vocabulary order. It asserts the exact declared list a provider claiming everything declarable publishes, so a new capability that no scenario carried would be caught there; one that does gate scenarios has to be added, and the comment says why this one is gated at all. CanonicalScenarioGuard's javadoc and the README's section on it both counted the canonical set out loud. Fifty-two becomes fifty-six. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 4 ++-- tools/tck/pom.xml | 2 +- .../contrib/tools/tck/CanonicalScenarioGuard.java | 6 +++--- .../contrib/tools/tck/ConformanceReportPluginTest.java | 6 ++++++ 4 files changed, 12 insertions(+), 6 deletions(-) diff --git a/tools/tck/README.md b/tools/tck/README.md index 5a5580639..1ec985a08 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -348,8 +348,8 @@ narrowing alone leaves the exclusion in force and runs nothing. ## The canonical set cannot be reduced Extending the suite is safe by convention. Shrinking it is what a conformance suite has to prevent, -because a run that asks fifty of the fifty-two questions and reports success is indistinguishable, in -every artifact it produces, from one that asked all fifty-two. +because a run that asks fifty-four of the fifty-six questions and reports success is +indistinguishable, in every artifact it produces, from one that asked all fifty-six. `CanonicalScenarioGuard` is an ordinary JUnit test that the suite selects, and it fails the build if this run is set up to execute less than the canonical set: diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index 58aea5a59..bdc1db62c 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -50,7 +50,7 @@ pin already moves only by deliberate commit. Update both together; a mismatch means a report names a revision that did not produce its scenarios. --> - 26362f85b7fcd59b35b969e6feebee80e206b24f + 009afe0617947121dcbebe4b66e0cc0c5cc4ada8 3.27.7 4.3.0 2.22.1 diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java index b295591ae..bd75410aa 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java @@ -26,9 +26,9 @@ * Fails a suite that is set up to run less than the whole canonical scenario set. * *

Everything else in this module makes the canonical set easy to extend safely. Nothing makes it - * hard to shrink, and shrinking it is the failure that matters: a run that asks fifty of the fifty-two - * questions and reports success is indistinguishable, in every artifact it produces, from one that - * asked all fifty-two. The known ways to get there are a feature file dropped into + * hard to shrink, and shrinking it is the failure that matters: a run that asks fifty-four of the + * fifty-six questions and reports success is indistinguishable, in every artifact it produces, from + * one that asked all fifty-six. The known ways to get there are a feature file dropped into * {@code gherkin/}, a {@code cucumber.filter.tags} or {@code cucumber.filter.name} expression, and * selectors or glue overridden in a consuming module's {@code junit-platform.properties}. * diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index a09ac9a3a..a4ec816bb 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -304,6 +304,12 @@ void aReservedCapabilityCannotReachTheDeclaration() { // and types.md types the field optional, so a backend with no variant // concept withholds it and those rows are skipped with that reason. Capability.VARIANTS.tag(), + // @disabled-flags gates the four-row outline that asks what a flag disabled + // in the management system resolves to. Gated because the answer follows + // from where the substitution happens rather than from provider quality: a + // provider that evaluates locally can return the caller's default, one whose + // backend decides never sent it. The maximal claim includes it. + Capability.DISABLED_FLAGS.tag(), Capability.UNAVAILABLE_INIT.tag(), Capability.NUMERIC_COERCION.tag(), // @large-integers gates a scenario and is an ordinary declarable capability. From 65c5cba47107ecdbc0ff2ad829add2a8c8f1bcf6 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 14:23:06 +0200 Subject: [PATCH 08/22] refactor(tck): follow the rename into the reporting machinery Path and package churn from the base's provider-tck -> tck rename, which this branch's own files had to be carried through: the package declarations of the six classes added here, the plugin's fully-qualified name in ProviderTck.PLUGINS, and the filtered build-info resource, which becomes tck-build.properties fed by a tck.spec.revision property. Two values are more than churn and were checked rather than swept: - TckBuildInfo.IMPLEMENTATION, which a conformance report carries as `tck.implementation`, becomes "java-sdk-contrib/tools/tck". It names the module a run came from, so it has to name the module that exists. - the plugin's log and failure prefixes become "tck []". They are what a build log shows when a report cannot be written. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 4 ++-- .../tools/tck/CanonicalScenarioGuard.java | 2 +- .../contrib/tools/tck/CanonicalScenarios.java | 8 ++++---- .../contrib/tools/tck/ConformanceReport.java | 2 +- .../tools/tck/ConformanceReportPlugin.java | 17 ++++++----------- .../contrib/tools/tck/ProviderTck.java | 2 +- .../contrib/tools/tck/TckBuildInfo.java | 8 ++++---- .../contrib/tools/tck/TckRunMetadata.java | 2 +- ...ck-build.properties => tck-build.properties} | 4 ++-- .../tools/tck/CanonicalScenarioGuardTest.java | 2 +- .../tools/tck/ConformanceReportPluginTest.java | 6 +++--- .../tools/tck/selftest/ReportSelfTestSteps.java | 6 +++--- 12 files changed, 29 insertions(+), 34 deletions(-) rename tools/tck/src/main/resources-filtered/dev/openfeature/contrib/tools/tck/{provider-tck-build.properties => tck-build.properties} (85%) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index bdc1db62c..74530c4d5 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -31,7 +31,7 @@ A conformance report has to say which questions were asked, not only what the answers were, so it records the open-feature/spec commit the packaged Gherkin, flag set and control API came from. It is filtered into - src/main/resources-filtered/.../provider-tck-build.properties and packaged in the JAR, + src/main/resources-filtered/.../tck-build.properties and packaged in the JAR, because the repository it came from is not. The executed Gherkin no longer rests on this pin alone: the results stream carries the @@ -50,7 +50,7 @@ pin already moves only by deliberate commit. Update both together; a mismatch means a report names a revision that did not produce its scenarios. --> - 009afe0617947121dcbebe4b66e0cc0c5cc4ada8 + 009afe0617947121dcbebe4b66e0cc0c5cc4ada8 3.27.7 4.3.0 2.22.1 diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java index bd75410aa..d231b60ac 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java @@ -1,4 +1,4 @@ -package dev.openfeature.contrib.tools.providertck; +package dev.openfeature.contrib.tools.tck; import io.cucumber.junit.platform.engine.Constants; import java.util.ArrayList; diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarios.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarios.java index 8bb56b60d..d417f045f 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarios.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarios.java @@ -1,4 +1,4 @@ -package dev.openfeature.contrib.tools.providertck; +package dev.openfeature.contrib.tools.tck; import edu.umd.cs.findbugs.annotations.SuppressFBWarnings; import io.cucumber.gherkin.GherkinParser; @@ -89,7 +89,7 @@ private static Set read() { CodeSource codeSource = ProviderTck.class.getProtectionDomain().getCodeSource(); URL location = codeSource == null ? null : codeSource.getLocation(); if (location == null) { - throw new IllegalStateException("The provider-tck code source is not visible to this JVM, so the " + throw new IllegalStateException("The tck code source is not visible to this JVM, so the " + "canonical scenario set cannot be established. Set -D" + CanonicalScenarioGuard.PARTIAL_PROPERTY + "=true to run without the canonical-set check, understanding that the run is then not a " + "conformance run."); @@ -100,7 +100,7 @@ private static Set read() { path = Paths.get(location.toURI()); } catch (URISyntaxException | IllegalArgumentException e) { throw new IllegalStateException( - "The provider-tck code source " + location + " is not a file, so the " + "The tck code source " + location + " is not a file, so the " + "canonical scenario set cannot be established.", e); } @@ -118,7 +118,7 @@ private static Set read() { if (refs.isEmpty()) { throw new IllegalStateException("No canonical scenarios found in " + ProviderTck.FEATURES + "/ of the " - + "provider-tck artifact at " + path + ". The artifact is not intact."); + + "tck artifact at " + path + ". The artifact is not intact."); } return refs; } diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java index c3d2aafd3..30704ecae 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java @@ -1,4 +1,4 @@ -package dev.openfeature.contrib.tools.providertck; +package dev.openfeature.contrib.tools.tck; import com.fasterxml.jackson.annotation.JsonInclude; import java.util.List; diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java index 0c991286d..4ccbe65d9 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java @@ -1,4 +1,4 @@ -package dev.openfeature.contrib.tools.providertck; +package dev.openfeature.contrib.tools.tck; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.JsonNode; @@ -215,7 +215,7 @@ private static String protocolVersionOf(byte[] results) { } catch (JsonProcessingException e) { // A stream whose first line will not parse is a bug worth surfacing, but not here: // omitting an optional field is better than failing a run that otherwise succeeded. - log.warn("provider-tck: could not read the Messages protocol version from the stream", e); + log.warn("tck: could not read the Messages protocol version from the stream", e); return null; } } @@ -244,7 +244,7 @@ private void write(String dir, byte[] results) { directory = Paths.get(dir); } catch (InvalidPathException e) { throw new IllegalStateException( - "provider-tck [" + configuration + "]: " + REPORT_DIR_ENV + " is not a usable path: " + dir, e); + "tck [" + configuration + "]: " + REPORT_DIR_ENV + " is not a usable path: " + dir, e); } String base = ReportNames.baseNameOf(configuration); @@ -259,8 +259,7 @@ private void write(String dir, byte[] results) { .writeValueAsString(build(run, location, digestOf(results), protocolVersionOf(results))) + "\n"; } catch (JsonProcessingException e) { - throw new IllegalStateException( - "provider-tck [" + configuration + "]: could not encode the conformance report", e); + throw new IllegalStateException("tck [" + configuration + "]: could not encode the conformance report", e); } // A failure to write is raised rather than logged and swallowed. CI that asked for a report @@ -272,14 +271,10 @@ private void write(String dir, byte[] results) { Files.write(envelopePath, json.getBytes(StandardCharsets.UTF_8)); } catch (IOException e) { throw new UncheckedIOException( - "provider-tck [" + configuration + "]: could not write the conformance report to " + directory, e); + "tck [" + configuration + "]: could not write the conformance report to " + directory, e); } - log.info( - "provider-tck [{}]: conformance report written to {}, results to {}", - configuration, - envelopePath, - resultsPath); + log.info("tck [{}]: conformance report written to {}, results to {}", configuration, envelopePath, resultsPath); } /** Digests the results stream in the {@code sha256:} form the schema asks for. */ diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ProviderTck.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ProviderTck.java index 2a186c24c..b3325c3a3 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ProviderTck.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ProviderTck.java @@ -85,7 +85,7 @@ public final class ProviderTck { *

Spelled out rather than derived from {@code ConformanceReportPlugin.class.getName()}, * which is not a compile-time constant and so cannot appear in an annotation value. */ - public static final String PLUGINS = "summary,dev.openfeature.contrib.tools.providertck.ConformanceReportPlugin"; + public static final String PLUGINS = "summary,dev.openfeature.contrib.tools.tck.ConformanceReportPlugin"; /** * Whether scenarios may run in parallel: never. diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckBuildInfo.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckBuildInfo.java index 4b69b494b..7346db619 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckBuildInfo.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckBuildInfo.java @@ -1,4 +1,4 @@ -package dev.openfeature.contrib.tools.providertck; +package dev.openfeature.contrib.tools.tck; import dev.openfeature.sdk.OpenFeatureAPI; import java.io.IOException; @@ -23,7 +23,7 @@ final class TckBuildInfo { /** Which TCK implementation this is, in the form the report schema asks for. */ - static final String IMPLEMENTATION = "java-sdk-contrib/tools/provider-tck"; + static final String IMPLEMENTATION = "java-sdk-contrib/tools/tck"; /** Maven coordinates of the SDK, reported without a version. */ static final String SDK_NAME = "dev.openfeature:sdk"; @@ -34,7 +34,7 @@ final class TckBuildInfo { private static final Logger log = LoggerFactory.getLogger(TckBuildInfo.class); /** Generated by the build from the module POM; see {@code src/main/resources-filtered}. */ - private static final String BUILD_PROPERTIES = "provider-tck-build.properties"; + private static final String BUILD_PROPERTIES = "tck-build.properties"; /** Written into every JAR by maven-archiver, and the most direct statement of what resolved. */ private static final String SDK_POM_PROPERTIES = "META-INF/maven/dev.openfeature/sdk/pom.properties"; @@ -96,7 +96,7 @@ private static Properties loadBuildProperties() { log.warn( "{} is missing from the TCK JAR, so a conformance report cannot identify the build " + "that produced it. This means the resource filtering configured in the " - + "provider-tck POM did not run.", + + "tck POM did not run.", BUILD_PROPERTIES); return new Properties(); } diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java index 0f967e69d..5748df839 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java @@ -1,4 +1,4 @@ -package dev.openfeature.contrib.tools.providertck; +package dev.openfeature.contrib.tools.tck; import java.util.ArrayList; import java.util.Collections; diff --git a/tools/tck/src/main/resources-filtered/dev/openfeature/contrib/tools/tck/provider-tck-build.properties b/tools/tck/src/main/resources-filtered/dev/openfeature/contrib/tools/tck/tck-build.properties similarity index 85% rename from tools/tck/src/main/resources-filtered/dev/openfeature/contrib/tools/tck/provider-tck-build.properties rename to tools/tck/src/main/resources-filtered/dev/openfeature/contrib/tools/tck/tck-build.properties index 72b965a64..eabdcecc1 100644 --- a/tools/tck/src/main/resources-filtered/dev/openfeature/contrib/tools/tck/provider-tck-build.properties +++ b/tools/tck/src/main/resources-filtered/dev/openfeature/contrib/tools/tck/tck-build.properties @@ -1,7 +1,7 @@ # Generated at build time by Maven resource filtering. Do not edit the copy in target/. # # Identifies the TCK build and the conformance artifacts it carries, for the -# tck section of a conformance report. The values come from the provider-tck +# tck section of a conformance report. The values come from the tck # POM; see the comment there for how the spec revision is maintained. tck.version=${project.version} -spec.revision=${provider-tck.spec.revision} +spec.revision=${tck.spec.revision} diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java index fff4721e5..e9d5f4297 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java @@ -1,4 +1,4 @@ -package dev.openfeature.contrib.tools.providertck; +package dev.openfeature.contrib.tools.tck; import static org.assertj.core.api.Assertions.assertThat; diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index a4ec816bb..81c2d3ad9 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -1,11 +1,11 @@ -package dev.openfeature.contrib.tools.providertck; +package dev.openfeature.contrib.tools.tck; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; -import dev.openfeature.contrib.tools.providertck.selftest.ReportSelfTestSteps; +import dev.openfeature.contrib.tools.tck.selftest.ReportSelfTestSteps; import io.cucumber.junit.platform.engine.Constants; import io.cucumber.plugin.event.EventHandler; import io.cucumber.plugin.event.EventPublisher; @@ -250,7 +250,7 @@ void theEnvelopeCarriesWhatTheSchemaRequires() { assertThat(envelope.get("provider").get("configuration").asText()).isEqualTo(CONFIGURATION); assertThat(envelope.get("sdk").get("name").asText()).isEqualTo("dev.openfeature:sdk"); assertThat(envelope.get("sdk").get("version").asText()).isNotEmpty(); - assertThat(envelope.get("tck").get("implementation").asText()).isEqualTo("java-sdk-contrib/tools/provider-tck"); + assertThat(envelope.get("tck").get("implementation").asText()).isEqualTo("java-sdk-contrib/tools/tck"); assertThat(envelope.get("tck").get("specRevision").asText()).hasSizeGreaterThanOrEqualTo(7); assertThat(envelope.get("backend").get("controlApi").asText()).isEqualTo("http"); diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/selftest/ReportSelfTestSteps.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/selftest/ReportSelfTestSteps.java index 544c8839d..67fe60086 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/selftest/ReportSelfTestSteps.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/selftest/ReportSelfTestSteps.java @@ -1,7 +1,7 @@ -package dev.openfeature.contrib.tools.providertck.selftest; +package dev.openfeature.contrib.tools.tck.selftest; -import dev.openfeature.contrib.tools.providertck.Capability; -import dev.openfeature.contrib.tools.providertck.CapabilityGate; +import dev.openfeature.contrib.tools.tck.Capability; +import dev.openfeature.contrib.tools.tck.CapabilityGate; import io.cucumber.java.Before; import io.cucumber.java.Scenario; import io.cucumber.java.en.Given; From b849a19b19f900b1f73536d3154adc8202ec47d5 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 16:33:29 +0200 Subject: [PATCH 09/22] refactor(tck): always write the backend block, and type controlApi The report schema now has backend in its top-level required array and controlApi in backend's, so the conditional emission goes away. The old backend description said "omitted for a provider with no backend", which contradicted the controlApi enum whose in-process member exists for exactly that provider - the one value most worth knowing could never legally appear. With the member required and closed on the base branch, backendOf() has nothing left to decide: it builds the block unconditionally and the empty-value fallback stops existing. TckRunMetadata.controlApi() returns ControlApi rather than Optional for the same reason. Re-pins tck.spec.revision to 93eb1a58, matching the submodule the base branch now carries. The two have to move together or a report names a revision that did not produce its scenarios. No report test had to change in substance: the one assertion on the field already expected "http", which ControlApi.HTTP serialises to. The fixture's metadata now passes the enum, and the envelope test additionally asserts that backend is present rather than present-if-set. Note for the record: this branch carries no copy of conformance-report.schema.json and no JSON-schema validator, so it cannot validate an emitted envelope against the schema. The assertions are field-by- field against the schema read by hand. Claiming otherwise would be the kind of unverified assertion this effort keeps tripping over. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 8 +++++++- tools/tck/pom.xml | 2 +- .../contrib/tools/tck/ConformanceReport.java | 19 ++++++++++++++++--- .../tools/tck/ConformanceReportPlugin.java | 13 ++++++++----- .../contrib/tools/tck/TckRunMetadata.java | 15 ++++++++++----- .../tck/ConformanceReportPluginTest.java | 5 ++++- 6 files changed, 46 insertions(+), 16 deletions(-) diff --git a/tools/tck/README.md b/tools/tck/README.md index 1ec985a08..7d4dd8a80 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -453,7 +453,7 @@ outright, which is the whole argument for a standard format over one we maintain ### What the envelope is for -A Messages stream cannot say what it was a test *of*. The envelope carries the four things no +A Messages stream cannot say what it was a test *of*. The envelope carries the five things no standard results format identifies: - **`provider`** — what the provider calls itself through its own metadata, not the suite name. The @@ -470,6 +470,12 @@ standard results format identifies: longer rests on that pin alone — the stream carries the `source` of every feature, so it can be diffed against the revision — but the pin is what identifies the two artifacts the stream does not carry, `flags/canonical-flags.json` and `openapi/control-api.yaml`. +- **`backend`** — what the provider was pointed at, and `controlApi`, which of the two control + contracts drove it. Always present, both of them: the same scenarios passing over the HTTP control + API and passing through in-process manipulation of a provider that *does* have a backend are not + the same claim, and this is the only field that separates them, so an omission would be an + unfalsifiable claim rather than no claim at all. See [How the backend was + driven](#how-the-backend-was-driven). - **`declaration`** — the capability set the provider claims. This is an **input** to reading the results, not a summary of them, which is why it cannot be derived from the stream. The stream says a scenario was skipped; only the declaration says whether that is because the provider declines the diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index 74530c4d5..e5358764f 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -50,7 +50,7 @@ pin already moves only by deliberate commit. Update both together; a mismatch means a report names a revision that did not produce its scenarios. --> - 009afe0617947121dcbebe4b66e0cc0c5cc4ada8 + 93eb1a58d2d2ec015acb298c914c5822e7a38dd2 3.27.7 4.3.0 2.22.1 diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java index 30704ecae..3eedf2231 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java @@ -47,7 +47,14 @@ public final class ConformanceReport { /** What asked the questions, and which questions. */ public final Tck tck; - /** What the provider was pointed at. */ + /** + * What the provider was pointed at. + * + *

Always present: {@code backend} is in the schema's top-level {@code required} array. An + * earlier revision described it as "omitted for a provider with no backend", which contradicted + * the {@code controlApi} enum whose {@code in-process} member exists for exactly that provider — + * so the one value most worth knowing could never legally appear. + */ public final Backend backend; /** The capability set this provider claims. */ @@ -164,10 +171,16 @@ public static final class Backend { * *

{@code in-process} is the narrow allowance made for providers with no backend; a report * claiming it for a provider that has one should be treated with suspicion. + * + *

Always written, because there is no third case an absent value would cover: every + * {@link BackendControl} is either driving a real backend over the control API or + * manipulating an in-process one, so an omission would be an unfalsifiable claim rather than + * no claim at all. {@link ControlApi} is closed for the same reason the schema's enum is, and + * serialises to the two values it names. */ - public final String controlApi; + public final ControlApi controlApi; - Backend(String description, String controlApi) { + Backend(String description, ControlApi controlApi) { this.description = description; this.controlApi = controlApi; } diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java index 4ccbe65d9..31ddfd7db 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java @@ -183,12 +183,15 @@ private static List declaredTags(Set declared) { return Collections.unmodifiableList(tags); } + /** + * Assembles the {@code backend} block, which is always emitted. + * + *

The schema requires it at the top level, and {@code controlApi} within it, so there is no + * conditional emission left here: a control that could not say how it drove the backend would + * have failed to compile. + */ private static ConformanceReport.Backend backendOf(TckRunMetadata run) { - String description = run.backendDescription().orElse(null); - String controlApi = run.controlApi().orElse(null); - return description == null && controlApi == null - ? null - : new ConformanceReport.Backend(description, controlApi); + return new ConformanceReport.Backend(run.backendDescription().orElse(null), run.controlApi()); } /** diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java index 5748df839..1095f2658 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/TckRunMetadata.java @@ -28,7 +28,7 @@ final class TckRunMetadata { private final String configuration; private final Set capabilities; private final String backendDescription; - private final String controlApi; + private final ControlApi controlApi; private final List knownDeviations; private volatile String providerName; @@ -37,7 +37,7 @@ final class TckRunMetadata { String configuration, Set capabilities, String backendDescription, - String controlApi, + ControlApi controlApi, List knownDeviations) { Capability.requireDeclarable(capabilities); this.configuration = configuration; @@ -64,9 +64,14 @@ Optional backendDescription() { return Optional.ofNullable(backendDescription); } - /** Returns how the backend was driven, or empty when there is no control API. */ - Optional controlApi() { - return Optional.ofNullable(controlApi); + /** + * Returns how the backend was driven. + * + *

Never absent. {@link BackendControl#controlApi()} is required and closed, so there is + * nothing here to be optional about — see {@link ControlApi}. + */ + ControlApi controlApi() { + return controlApi; } /** Returns the deviations the provider author acknowledged, empty when there are none. */ diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index 81c2d3ad9..e321abadd 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -252,6 +252,9 @@ void theEnvelopeCarriesWhatTheSchemaRequires() { assertThat(envelope.get("sdk").get("version").asText()).isNotEmpty(); assertThat(envelope.get("tck").get("implementation").asText()).isEqualTo("java-sdk-contrib/tools/tck"); assertThat(envelope.get("tck").get("specRevision").asText()).hasSizeGreaterThanOrEqualTo(7); + // backend is in the schema's top-level required array and controlApi in backend's, so both + // are asserted as present rather than as present-if-set. + assertThat(envelope.has("backend")).isTrue(); assertThat(envelope.get("backend").get("controlApi").asText()).isEqualTo("http"); // The schema sets additionalProperties: false throughout, so anything the results payload @@ -447,7 +450,7 @@ private static TckRunMetadata metadata(Set declared) { CONFIGURATION, declared, "a test double", - "http", + ControlApi.HTTP, Collections.singletonList(KnownDeviation.untracked( Capability.NUMERIC_COERCION, "the fixture provider narrows a float to an integer"))); metadata.recordProviderName("My Provider"); From ecb75c89d46278fefb181f417a2f563c4d5908d1 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sat, 12 Sep 2026 23:33:52 +0200 Subject: [PATCH 10/22] refactor(tck)!: TCK knobs lose the PROVIDER_ prefix PROVIDER_TCK_REPORT_DIR becomes TCK_REPORT_DIR and PROVIDER_TCK_PARTIAL becomes TCK_PARTIAL, with the Maven system properties following: provider.tck.report.dir becomes tck.report.dir and provider.tck.partial becomes tck.partial. The package is called tck, not provider-tck, so PROVIDER_ names the thing after what it happens to test today. The report-directory variable in particular is read by all four languages' suites, so one cross-language CI job sets one variable and the name has to agree; renaming it in three languages and not the fourth is worse than either consistent answer. Nothing is published and nobody has scripted against either spelling, so it is free now and expensive later. PROVIDER_TCK_PARTIAL was not in the agreed list, and is renamed anyway: a half-renamed set of knobs is unguessable, which is the same reason the agreed list covered two variables rather than one. Flagged for the other languages that carry it. Two tests now pin the four spellings as literals rather than through the constants, because what matters is the name an adopter or a CI job types, and nothing about a self-consistent rename would fail a compiler. Also re-pins the spec submodule and tck.spec.revision to ccdb8879 together, so a report cannot name a revision that did not produce its scenarios. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 8 ++++---- tools/tck/pom.xml | 2 +- .../contrib/tools/tck/CanonicalScenarioGuard.java | 4 ++-- .../contrib/tools/tck/ConformanceReportPlugin.java | 4 ++-- .../contrib/tools/tck/CanonicalScenarioGuardTest.java | 10 ++++++++++ .../tools/tck/ConformanceReportPluginTest.java | 11 +++++++++++ 6 files changed, 30 insertions(+), 9 deletions(-) diff --git a/tools/tck/README.md b/tools/tck/README.md index 7d4dd8a80..582a4f283 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -369,7 +369,7 @@ defined over `gherkin/` alone. Narrowing a run legitimately is what `capabilities()` is for — those scenarios are reported as skipped with a reason, which a filtered scenario is not. To filter anyway while debugging, set -`-Dprovider.tck.partial=true` (or `PROVIDER_TCK_PARTIAL`). The guard then reports itself as +`-Dtck.partial=true` (or `TCK_PARTIAL`). The guard then reports itself as **skipped** rather than passed, so the run states that its canonical set was not verified. What the guard does not establish is that the canonical files contain what they should — a @@ -379,17 +379,17 @@ elsewhere: the results stream carries the `source` of every feature that execute ## Conformance reports -Set `PROVIDER_TCK_REPORT_DIR` and each suite writes two files: an envelope conforming to the +Set `TCK_REPORT_DIR` and each suite writes two files: an envelope conforming to the [report schema][report-schema] in the specification, and the run's results as a [Cucumber Messages][messages] stream. ```console -$ PROVIDER_TCK_REPORT_DIR=./reports mvn test -Dtest='Flagd*TckTest' +$ TCK_REPORT_DIR=./reports mvn test -Dtest='Flagd*TckTest' $ ls reports/ flagd-in-process.json flagd-in-process.ndjson flagd-rpc.json flagd-rpc.ndjson ``` -`-Dprovider.tck.report.dir=...` does the same thing and is often easier to pass through Maven. The +`-Dtck.report.dir=...` does the same thing and is often easier to pass through Maven. The environment variable is the portable spelling — every language's TCK reads it, so one cross-language CI job can set one thing. diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index e5358764f..bc04f4cf4 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -50,7 +50,7 @@ pin already moves only by deliberate commit. Update both together; a mismatch means a report names a revision that did not produce its scenarios. --> - 93eb1a58d2d2ec015acb298c914c5822e7a38dd2 + ccdb88790bb4f4beaef14a182d0c2592feab34b2 3.27.7 4.3.0 2.22.1 diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java index d231b60ac..494028e4e 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuard.java @@ -77,10 +77,10 @@ public final class CanonicalScenarioGuard { * made that impossible would be switched off permanently instead of temporarily. It produces a * skip rather than a pass, so the run says out loud that its canonical set was not verified. */ - public static final String PARTIAL_PROPERTY = "provider.tck.partial"; + public static final String PARTIAL_PROPERTY = "tck.partial"; /** Environment variable equivalent of {@link #PARTIAL_PROPERTY}. */ - public static final String PARTIAL_ENV = "PROVIDER_TCK_PARTIAL"; + public static final String PARTIAL_ENV = "TCK_PARTIAL"; /** Cucumber configuration keys that stop a discovered scenario from running. */ private static final String[] FILTER_KEYS = { diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java index 31ddfd7db..b669b5122 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReportPlugin.java @@ -65,7 +65,7 @@ public final class ConformanceReportPlugin implements ConcurrentEventListener { /** Environment variable naming the directory reports are written to. */ - public static final String REPORT_DIR_ENV = "PROVIDER_TCK_REPORT_DIR"; + public static final String REPORT_DIR_ENV = "TCK_REPORT_DIR"; /** * System property naming the directory reports are written to, taking precedence over the @@ -75,7 +75,7 @@ public final class ConformanceReportPlugin implements ConcurrentEventListener { * Gradle invocation is usually parameterised. The environment variable is the portable spelling * and is what every other language's TCK reads, so a cross-language CI job can set one thing. */ - public static final String REPORT_DIR_PROPERTY = "provider.tck.report.dir"; + public static final String REPORT_DIR_PROPERTY = "tck.report.dir"; /** Extension of the envelope, which is what a consumer reads first. */ static final String ENVELOPE_EXTENSION = ".json"; diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java index e9d5f4297..a53a0b033 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java @@ -197,6 +197,16 @@ void anUnobservedRunFails() { .contains(CanonicalScenarioGuard.PARTIAL_PROPERTY); } + @Test + @DisplayName("the partial-run escape hatch carries no PROVIDER_ prefix either") + void thePartialKnobIsSpelledWithoutThePrefix() { + // Literals on purpose: every TCK knob an adopter or a CI job sets is spelled TCK_*, and one + // of them keeping the old PROVIDER_TCK_* prefix is the half-rename that makes the set + // unguessable. + assertThat(CanonicalScenarioGuard.PARTIAL_ENV).isEqualTo("TCK_PARTIAL"); + assertThat(CanonicalScenarioGuard.PARTIAL_PROPERTY).isEqualTo("tck.partial"); + } + /** Discovers the real suite configuration, without executing anything. */ private static org.junit.platform.launcher.TestPlan discoverFixtureSuite() { return LauncherFactory.create() diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index e321abadd..e0ca8c633 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -389,6 +389,17 @@ void nothingIsWrittenByDefault(@TempDir Path dir) throws IOException { } } + @Test + @DisplayName("the report-directory knob is spelled the way the other three languages spell it") + void theReportDirectoryKnobIsSpelledPortably() { + // Asserted as literals rather than through the constants, because the point is the name a + // cross-language CI job sets, not that the code is self-consistent. Go, Python and + // JavaScript read TCK_REPORT_DIR; one job exporting one variable has to drive all four, so + // a rename here is a breaking change to something no compiler checks. + assertThat(ConformanceReportPlugin.REPORT_DIR_ENV).isEqualTo("TCK_REPORT_DIR"); + assertThat(ConformanceReportPlugin.REPORT_DIR_PROPERTY).isEqualTo("tck.report.dir"); + } + @Test @DisplayName("both files are named after the configuration, safely") void theFilesAreNamedAfterTheConfiguration() { From 7fecbc435b750232fd882e2d6c7aa9e00886d754 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 09:59:06 +0200 Subject: [PATCH 11/22] feat(tck): report the revision that gated the reasons tck.spec.revision follows the submodule to c342461a, so a report names the revision that actually produced its scenarios rather than the one before the reasons moved into reason.feature. Checked with help:evaluate against the module POM and against `git -C spec rev-parse HEAD`, which is the pair the property's comment says must agree. The one pinned expectation that had to move is the maximal declaration in ConformanceReportPluginTest: it lists every declarable tag in order, so @standard-reasons had to be added with the note saying why an opt-in claim is still an ordinary declarable capability. That test is the reason the list is pinned at all -- "everything" once meant EnumSet.allOf and published claims about capabilities no scenario examines. Nothing else on this branch pins a count. CanonicalScenarios reads the packaged gherkin directory out of the artifact's own code source, so the canonical set moved from 57 to 66 scenarios per suite with no edit, and CanonicalScenarioGuard compared the new set against the new plan. That is the under-collection guard doing the job it exists for: had the sixth feature file not been collected, the guard would have failed rather than the run going quietly green on a smaller question set. 272 tests, 43 skipped, up from 245 and 37. The 32-test gap to the base branch is unchanged. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 2 +- .../contrib/tools/tck/ConformanceReportPluginTest.java | 8 +++++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index bc04f4cf4..7400ec161 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -50,7 +50,7 @@ pin already moves only by deliberate commit. Update both together; a mismatch means a report names a revision that did not produce its scenarios. --> - ccdb88790bb4f4beaef14a182d0c2592feab34b2 + c342461aa95df9e3b46320dbae65e88e5e8b815a 3.27.7 4.3.0 2.22.1 diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index e0ca8c633..646528b18 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -323,7 +323,13 @@ void aReservedCapabilityCannotReachTheDeclaration() { Capability.LARGE_INTEGERS.tag(), // @targeting was reserved until targeting-key-flag's three scenarios // arrived. It gates something now, so the maximal claim includes it. - Capability.TARGETING.tag()); + Capability.TARGETING.tag(), + // @standard-reasons gates reason.feature in its entirety. It is a claim + // rather than an exemption -- 2.2.5 lets a provider report "some other + // string", so a provider that does not declare it is not thereby deficient + // -- but it is an ordinary declarable capability all the same, and the + // maximal claim includes it. + Capability.STANDARD_REASONS.tag()); } @Test From 85fe8d7128312ec36e4e36bed0ab723029a25bf3 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 15:21:09 +0200 Subject: [PATCH 12/22] feat(tck): close the loop from the gitlink to the revision a report names Three things follow from the base branch's two changes. tck.spec.revision moves to 89b1519a, and stops being maintained on trust. ConformanceReportPluginTest now asserts that what TckBuildInfo reads back equals CanonicalAssetDigestTest.PINNED_REVISION, which that test has in turn checked against the packaged assets by digest. The POM comment used to end "update both together; a mismatch means a report names a revision that did not produce its scenarios", and nothing enforced it. Now the chain from the submodule gitlink to the revision a published report claims is checked end to end, and a re-pin that forgets this line fails the build. The maximal declaration loses @large-integers. That test exists because "everything" once meant EnumSet.allOf and published claims about capabilities nothing examined; the maximal claim has to be the maximal claim a provider written against THIS SDK can make, and no Java provider can be asked for 2^53 - 1. And the report self-test fixture gains a scenario carrying @large-integers, because the report is where the distinction has to survive. The envelope's declaration explains every other skip: a reader takes the scenario's tags, checks them against the declared set, and the reason follows. It does not explain this one -- the capability is absent from every Java declaration, and absent for a reason that says nothing about the provider, so a reader inferring "the provider declined" would be reading a decision into something no Java provider was ever offered. The gate's reason is carried on the hook result that produced the skip, and the test now asserts that the two skips in the same stream cannot be confused: one says "does not declare capability STALE", the other says "the Java SDK cannot express capability LARGE_INTEGERS" and "not the provider under test declining", and neither contains the other's wording. The README's declaration bullet says so rather than continuing to claim the declaration accounts for every skip. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 9 ++ tools/tck/pom.xml | 13 ++- .../tck/ConformanceReportPluginTest.java | 89 +++++++++++++++---- .../resources/report-selftest/report.feature | 9 ++ 4 files changed, 99 insertions(+), 21 deletions(-) diff --git a/tools/tck/README.md b/tools/tck/README.md index 582a4f283..6e056aeeb 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -482,6 +482,15 @@ standard results format identifies: capability it needed. Given the declaration and a scenario's tags — both present — the reason for each skip follows, so it does not have to be transported per scenario. + **One skip does not follow from it, and that one is why the reason travels anyway.** A capability + this SDK cannot express is absent from every Java declaration, and absent for a reason that says + nothing about the provider — `@large-integers` is not there because `Client.getIntegerDetails` is + 32 bits, not because anyone declined it. A reader who inferred *"the provider declined"* from that + absence would be reading a decision into something no Java provider was ever offered. The gate's + reason is carried on the hook result that produced the skip, and its wording is deliberately unlike + an undeclared capability's, so the two are distinguishable in the results themselves. See + [Declaring capabilities](#declaring-capabilities). + `knownDeviations` is the one thing neither the stream nor the declaration can express: whether a withheld capability is a limitation or a bug. See [Saying that a withheld capability is a defect](#saying-that-a-withheld-capability-is-a-defect). diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index 7400ec161..d87e68378 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -47,10 +47,17 @@ A property rather than a value read from git during the build because Maven has no way to capture a command's output into a property without another plugin, and the submodule - pin already moves only by deliberate commit. Update both together; a mismatch means a - report names a revision that did not produce its scenarios. + pin already moves only by deliberate commit. + + It is no longer only a convention that they agree: this value reaches the classpath + through tck-build.properties, and ConformanceReportPluginTest asserts that what + TckBuildInfo reads back equals CanonicalAssetDigestTest.PINNED_REVISION, which that + test has in turn checked against the packaged assets by digest. So the chain from the + gitlink to the revision a report publishes is closed, and a re-pin that forgets this + line fails the build rather than producing a report that names a revision which did not + produce its scenarios. --> - c342461aa95df9e3b46320dbae65e88e5e8b815a + 89b1519a08d81c46ba47fc2a54c44d40fdee845d 3.27.7 4.3.0 2.22.1 diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index 646528b18..0017e9906 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -55,7 +55,11 @@ */ class ConformanceReportPluginTest { - /** The fixture's {@code @stale} tag is deliberately absent, so two scenarios must be skipped. */ + /** + * The fixture's {@code @stale} tag is deliberately absent, so two scenarios must be skipped — and + * a third that no declaration could have covered, because {@code @large-integers} is + * {@linkplain Capability#inexpressible() inexpressible} in Java. + */ private static final Set DECLARED = EnumSet.of(Capability.OBJECT, Capability.EVENTS); private static final String CONFIGURATION = "my-provider-rpc"; @@ -101,10 +105,10 @@ static void runTheFixtureSuite() throws IOException { @Test @DisplayName("every scenario appears exactly once, whatever happened to it") void everyScenarioAppearsExactlyOnce() { - // Seven pickles: four plain scenarios and three outline rows. A report that quietly omitted + // Eight pickles: five plain scenarios and three outline rows. A report that quietly omitted // the ones it did not run would satisfy every other rule here and still mislead, because a // reader would have no way to know how many questions went unasked. - assertThat(results.pickles).hasSize(7); + assertThat(results.pickles).hasSize(8); assertThat(results.pickleOfTestCase.values()) .as("one test case per pickle, and no pickle executed twice") @@ -123,7 +127,7 @@ void everyScenarioAppearsExactlyOnce() { @DisplayName("the outcome counts are what the fixture describes") void theOutcomeCountsAreWhatTheFixtureDescribes() { assertThat(results.outcomeCounts()) - .containsExactlyInAnyOrderEntriesOf(counts("PASSED", 4L, "FAILED", 1L, "SKIPPED", 2L)); + .containsExactlyInAnyOrderEntriesOf(counts("PASSED", 4L, "FAILED", 1L, "SKIPPED", 3L)); } @Test @@ -137,8 +141,9 @@ void aGatedScenarioIsSkipped() { } assertThat(gated) - .as("the premise: the fixture has a gated plain scenario and a gated outline row") - .hasSize(2); + .as("the premise: the fixture has a gated plain scenario, a gated outline row and an " + + "inexpressible one, which no declaration could have contained") + .hasSize(3); for (String pickleId : gated) { assertThat(results.outcomeOfPickle(pickleId)) @@ -152,13 +157,37 @@ void aGatedScenarioIsSkipped() { } @Test - @DisplayName("the gate's own reason survives into the stream") + @DisplayName("the gate's own reason survives into the stream, and the two kinds of skip stay apart") void theGateReasonSurvives() { - // The declaration in the envelope is enough to work out *why* a scenario was skipped, but - // the reason the gate gave is carried too, on the hook result that produced the skip. - assertThat(results.messagesOfSkippedSteps()).isNotEmpty().allSatisfy(message -> assertThat(message) - .contains("does not declare capability") - .contains("STALE")); + // The declaration in the envelope is enough to work out *why* most scenarios were skipped, + // but the reason the gate gave is carried too, on the hook result that produced the skip. + List messages = results.messagesOfSkippedSteps(); + assertThat(messages).hasSize(3); + + // And for one of them the declaration is NOT enough, which is why this reason has to travel. + // @large-integers is absent from every Java declaration and absent for a reason that says + // nothing about the provider: the SDK's integer accessor is 32 bits. A reader who inferred + // "the provider declined" from the absence would be reading a decision into something no + // Java provider was ever offered. So the message names the SDK, and says it is not the + // provider declining, in words that could not be confused with the other two. + List unaskable = new ArrayList<>(); + messages.forEach(message -> { + if (message.contains("LARGE_INTEGERS")) { + unaskable.add(message); + } + }); + assertThat(unaskable).hasSize(1); + assertThat(unaskable.get(0)) + .contains("the Java SDK cannot express capability") + .contains("not the provider under test declining") + .doesNotContain("does not declare capability"); + + assertThat(messages) + .filteredOn(message -> !message.contains("LARGE_INTEGERS")) + .hasSize(2) + .allSatisfy(message -> assertThat(message) + .contains("does not declare capability") + .contains("STALE")); } @Test @@ -252,6 +281,9 @@ void theEnvelopeCarriesWhatTheSchemaRequires() { assertThat(envelope.get("sdk").get("version").asText()).isNotEmpty(); assertThat(envelope.get("tck").get("implementation").asText()).isEqualTo("java-sdk-contrib/tools/tck"); assertThat(envelope.get("tck").get("specRevision").asText()).hasSizeGreaterThanOrEqualTo(7); + assertThat(envelope.get("tck").get("specRevision").asText()) + .as("and it is the revision the packaged assets actually came from; see the test below") + .isEqualTo(CanonicalAssetDigestTest.PINNED_REVISION); // backend is in the schema's top-level required array and controlApi in backend's, so both // are asserted as present rather than as present-if-set. assertThat(envelope.has("backend")).isTrue(); @@ -315,12 +347,11 @@ void aReservedCapabilityCannotReachTheDeclaration() { Capability.DISABLED_FLAGS.tag(), Capability.UNAVAILABLE_INIT.tag(), Capability.NUMERIC_COERCION.tag(), - // @large-integers gates a scenario and is an ordinary declarable capability. - // No Java provider holds it, because the SDK's integer accessor is 32 bits, - // but that is a fact about the SDK recorded in Appendix F rather than a - // second kind of declaration, so the maximal claim includes it and a real - // harness withholds it. - Capability.LARGE_INTEGERS.tag(), + // @large-integers is absent, and for a different reason from @caching's. + // Scenarios do carry it -- Go and JavaScript run them -- but the Java SDK's + // integer accessor is 32 bits, so no Java provider can be asked for 2^53 - 1 + // and none may claim it. The maximal claim is the maximal claim a provider + // written against THIS SDK can make, which is what a report has to carry. // @targeting was reserved until targeting-key-flag's three scenarios // arrived. It gates something now, so the maximal claim includes it. Capability.TARGETING.tag(), @@ -332,6 +363,28 @@ void aReservedCapabilityCannotReachTheDeclaration() { Capability.STANDARD_REASONS.tag()); } + @Test + @DisplayName("the revision a report publishes is the one the packaged assets came from") + void theReportedRevisionIsTheRevisionTheAssetsCameFrom() { + // A report's whole claim to be readable later is that it says which questions were asked. + // The revision it names is a POM property, maintained by hand beside a submodule gitlink + // that moves by a different command -- so until now "keep them in step" was a comment in + // the POM, and a re-pin that updated one and not the other would have published a revision + // that did not produce the scenarios in the same file. + // + // CanonicalAssetDigestTest has already established that the packaged assets are + // PINNED_REVISION's, by digest. This is the last link: what the build filtered into the JAR + // and TckBuildInfo reads back is that same revision. + assertThat(TckBuildInfo.specRevision()) + .as("tck.spec.revision in tools/tck/pom.xml must equal " + + "CanonicalAssetDigestTest.PINNED_REVISION -- a re-pin moves the submodule " + + "gitlink, that constant and this property together") + .isEqualTo(CanonicalAssetDigestTest.PINNED_REVISION); + assertThat(TckBuildInfo.specRevision()) + .as("and the properties resource must actually have been filtered, or both are 'unknown'") + .isNotEqualTo(TckBuildInfo.UNKNOWN); + } + @Test @DisplayName("naming a reserved capability fails the run rather than being dropped quietly") void namingAReservedCapabilityFailsTheRun() { diff --git a/tools/tck/src/test/resources/report-selftest/report.feature b/tools/tck/src/test/resources/report-selftest/report.feature index 89678df9d..99d5caaa0 100644 --- a/tools/tck/src/test/resources/report-selftest/report.feature +++ b/tools/tck/src/test/resources/report-selftest/report.feature @@ -5,6 +5,11 @@ # preserve are properties of that shape: a capability tag on the feature, one on a scenario, one on # a single Examples block, and an outline whose rows all share a name. Running real Cucumber over # this is the only way to check what a capability-gated abort actually becomes in the results. +# +# It also carries both kinds of skip, because they are the two a reader has to be able to tell +# apart: a capability this provider did not declare, and a capability no provider in this language +# can be asked. Both end as SKIPPED, and only the reason carried on the hook result distinguishes +# them. @events Feature: Report self-test @@ -20,6 +25,10 @@ Feature: Report self-test Scenario: A scenario needing an undeclared capability Given a step that passes + @large-integers + Scenario: A scenario needing a capability this SDK cannot express + Given a step that passes + Scenario: A scenario that fails Given a step that fails From 644291d580cc56de9069cae4e53b83beb5723df5 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 19:01:29 +0200 Subject: [PATCH 13/22] chore(tck): follow the re-pin to aa2ad24f in the revision a report names tck.spec.revision is the second half of the pin: it is what a conformance report publishes as the source of its scenarios, and ConformanceReportPluginTest asserts it equals CanonicalAssetDigestTest.PINNED_REVISION, which the digest has in turn checked against the packaged assets. So this line moves in the same pass as the gitlink or the build fails. Prose-only upstream, assets byte-identical, and the numbers say so: 279 tests, 0 failures, 43 skipped, unchanged from the old pin. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index d87e68378..61d6faa1b 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -57,7 +57,7 @@ line fails the build rather than producing a report that names a revision which did not produce its scenarios. --> - 89b1519a08d81c46ba47fc2a54c44d40fdee845d + aa2ad24f5a14ae2b5756df0b6d23f493f39507e6 3.27.7 4.3.0 2.22.1 From aa76e4a6a4820d9661a4f01fa810cf55e27b3e33 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 19:59:30 +0200 Subject: [PATCH 14/22] test(tck): stop the report fixture depicting the shape the guidance discourages The report self-test's metadata fixture carried an untracked deviation against a withheld @numeric-coercion, summarised "the fixture provider narrows a float to an integer". That is withhold-plus-deviate: a provider that narrows is attempting the coercion, so the honest report declares the capability and lets the scenario fail. A fixture is read as an example whether or not it is meant as one, and this was the fifth place in these branches where that example appeared. The deviation now names @configuration-change with a summary describing a provider that never subscribes and so can never report a change -- withheld because the behaviour cannot be attempted at all, which is the shape a withheld-and-skipped deviation is for. The assertion follows the tag; nothing else about the fixture moves. 279 tests, 0 failures, 43 skipped, unchanged. Signed-off-by: Simon Schrottner --- .../contrib/tools/tck/ConformanceReportPluginTest.java | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index 0017e9906..03f7989fd 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -418,7 +418,7 @@ void aDefectIsReportedAsADeviation() { JsonNode deviations = envelope.get("knownDeviations"); assertThat(deviations).hasSize(1); - assertThat(deviations.get(0).get("capability").asText()).isEqualTo(Capability.NUMERIC_COERCION.tag()); + assertThat(deviations.get(0).get("capability").asText()).isEqualTo(Capability.CONFIGURATION_CHANGE.tag()); assertThat(deviations.get(0).get("summary").asText()).isNotEmpty(); assertThat(deviations.get(0).has("issue")) .as("the fixture's deviation is untracked, and the field is omitted rather than empty") @@ -521,8 +521,14 @@ private static TckRunMetadata metadata(Set declared) { declared, "a test double", ControlApi.HTTP, + // Withheld-and-skipped, which is the shape a fixture may legitimately depict: this + // provider cannot attempt the behaviour, because it never subscribes to the thing + // that would tell it. A narrowing-coercion deviation used to stand here, and that + // is the shape the guidance discourages -- a provider that attempts a behaviour and + // gets it wrong declares the capability and lets the scenario fail. Collections.singletonList(KnownDeviation.untracked( - Capability.NUMERIC_COERCION, "the fixture provider narrows a float to an integer"))); + Capability.CONFIGURATION_CHANGE, + "the fixture provider swallows its child's change events and can never " + "report one"))); metadata.recordProviderName("My Provider"); return metadata; } From f61c939ce856f3f59bdca6c10941b508d9ab6a4b Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Sun, 13 Sep 2026 22:30:40 +0200 Subject: [PATCH 15/22] docs(tck): stop naming an adoption's class in the report's own docs The envelope section illustrated configuration() with `FlagdInProcessTckTest` -> `flagd-in-process`. That class is gone: the flagd adoption now lives in a `tck` package of its own and its suites are `RpcTest` and `InProcessTest`. It was the only line on this branch that reached into an adoption, and it is the reason to use a made-up class here instead. The configuration names in the rest of the section are untouched and still correct -- flagd's suites state `flagd-rpc` and `flagd-in-process` rather than deriving them, precisely so a report does not start saying `in-process` when a class is renamed. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tools/tck/README.md b/tools/tck/README.md index 6e056aeeb..e4b770454 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -460,7 +460,9 @@ standard results format identifies: suite name is chosen to read well in a failure message (`flagd-rpc`), which makes it the *configuration*, and it is reported as such. One provider with two materially different modes produces two reports that are not interchangeable. Derived from the suite class name - (`FlagdInProcessTckTest` → `flagd-in-process`); override `ProviderTckHarness.configuration()`. + (`MyProviderInProcessTest` → `my-provider-in-process`) unless the suite overrides + `ProviderTckHarness.configuration()`, which flagd's two do — a class in a package already called + `.../flagd/tck/` is named `InProcessTest`, and `in-process` on a report does not say whose. - **`sdk`** — read from the classpath rather than declared, because the TCK depends on an SDK version *range* so that adopting it can never force an upgrade. What a consumer actually ran against is only knowable at runtime. From a1a0dc267cdc04e4cea9f488589057fdcdcc2dca Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 06:58:25 +0200 Subject: [PATCH 16/22] docs(tck): follow the base README rewrite into the reporting section Three references the rewrite broke or made stale, all of them this branch pointing into prose the base branch now spells differently. "How the backend was driven" became "Identifying the run", and "Saying that a withheld capability is a defect" became "Known deviations", so both anchors dangled. And the report example still selected the flagd suites by the filename pattern the directory move removed - -Dtest='Flagd*TckTest' matches nothing now, so a reader copying it would have got an empty reports/ directory and no error. It is the documented -Ptck command instead. No content moved and no count changed. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/tools/tck/README.md b/tools/tck/README.md index e4b770454..97874c1d7 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -384,7 +384,7 @@ Set `TCK_REPORT_DIR` and each suite writes two files: an envelope conforming to [Cucumber Messages][messages] stream. ```console -$ TCK_REPORT_DIR=./reports mvn test -Dtest='Flagd*TckTest' +$ TCK_REPORT_DIR=./reports mvn -Ptck -pl providers/flagd test $ ls reports/ flagd-in-process.json flagd-in-process.ndjson flagd-rpc.json flagd-rpc.ndjson ``` @@ -476,8 +476,8 @@ standard results format identifies: contracts drove it. Always present, both of them: the same scenarios passing over the HTTP control API and passing through in-process manipulation of a provider that *does* have a backend are not the same claim, and this is the only field that separates them, so an omission would be an - unfalsifiable claim rather than no claim at all. See [How the backend was - driven](#how-the-backend-was-driven). + unfalsifiable claim rather than no claim at all. See [Identifying the + run](#identifying-the-run). - **`declaration`** — the capability set the provider claims. This is an **input** to reading the results, not a summary of them, which is why it cannot be derived from the stream. The stream says a scenario was skipped; only the declaration says whether that is because the provider declines the @@ -494,8 +494,7 @@ standard results format identifies: [Declaring capabilities](#declaring-capabilities). `knownDeviations` is the one thing neither the stream nor the declaration can express: whether a -withheld capability is a limitation or a bug. See -[Saying that a withheld capability is a defect](#saying-that-a-withheld-capability-is-a-defect). +withheld capability is a limitation or a bug. See [Known deviations](#known-deviations). `results.digest` covers the `.ndjson`, so a consumer that fetched the two separately can tell that what it has is what the envelope describes. From 39ed77c69f1bd46443ba530fe23a200e0de9e72c Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 07:10:06 +0200 Subject: [PATCH 17/22] docs(tck): stop naming a canonical count the suite has already outgrown The sentence illustrating why the canonical set cannot be reduced said "fifty-four of the fifty-six questions". The canonical set was 56 when that was written and is 65 now, so the illustration named a number the suite had already left behind -- and would do so again at the next addition. The point does not need a count, so it no longer carries one. Signed-off-by: Simon Schrottner --- tools/tck/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/tck/README.md b/tools/tck/README.md index 97874c1d7..79d7dafae 100644 --- a/tools/tck/README.md +++ b/tools/tck/README.md @@ -348,8 +348,8 @@ narrowing alone leaves the exclusion in force and runs nothing. ## The canonical set cannot be reduced Extending the suite is safe by convention. Shrinking it is what a conformance suite has to prevent, -because a run that asks fifty-four of the fifty-six questions and reports success is -indistinguishable, in every artifact it produces, from one that asked all fifty-six. +because a run that asks all but two of the canonical questions and reports success is +indistinguishable, in every artifact it produces, from one that asked every question there is. `CanonicalScenarioGuard` is an ordinary JUnit test that the suite selects, and it fails the build if this run is set up to execute less than the canonical set: From 9e9122f6197c53b0dfa174f4a24cff481dfcf6ce Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Mon, 14 Sep 2026 08:35:38 +0200 Subject: [PATCH 18/22] docs(tck): drop two notes about what a previous revision said One recorded that an earlier revision of this class described `backend` as omitted for a backend-less provider; the field is required and the javadoc now just says so. The other named Go's and Python's Cucumber Messages pins, where what the field needs to convey is that the pins differ at all. Comments only. 279 tests / 43 skipped, unchanged. Signed-off-by: Simon Schrottner --- .../contrib/tools/tck/ConformanceReport.java | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java index 3eedf2231..05c62d46d 100644 --- a/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java +++ b/tools/tck/src/main/java/dev/openfeature/contrib/tools/tck/ConformanceReport.java @@ -50,10 +50,9 @@ public final class ConformanceReport { /** * What the provider was pointed at. * - *

Always present: {@code backend} is in the schema's top-level {@code required} array. An - * earlier revision described it as "omitted for a provider with no backend", which contradicted - * the {@code controlApi} enum whose {@code in-process} member exists for exactly that provider — - * so the one value most worth knowing could never legally appear. + *

Always present: {@code backend} is in the schema's top-level {@code required} array, + * including for a provider with no backend — that is what {@code controlApi}'s + * {@code in-process} member is for. */ public final Backend backend; @@ -220,10 +219,9 @@ public static final class Results { /** * The Cucumber Messages release the stream was produced against. * - *

Messages is versioned and the four TCK implementations pin different releases -- this - * one takes whatever cucumber-jvm bundles, while the Go TCK builds against v21 and the - * Python one against 34.2.0 -- so a consumer holding two reports cannot assume one schema - * validates both. + *

Messages is versioned and the TCK implementations pin different releases -- this one + * takes whatever cucumber-jvm bundles -- so a consumer holding two reports cannot assume one + * schema validates both. * *

Guessing is worse than not validating. A later schema accepts messages this producer * could not have emitted, and an earlier one rejects messages that are perfectly valid, so a From 92a7ea8fac7a34040c4d83da52d8515055faab09 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Tue, 15 Sep 2026 22:18:13 +0200 Subject: [PATCH 19/22] chore(tck): follow the re-pin to d47a66eb in the revision a report names tck.spec.revision is what a conformance report publishes as the source of its scenarios, and ConformanceReportPluginTest checks it against PINNED_REVISION where it reads it back, so it moves with the pin or the build says so. Two expectations the pin moved with it. The type-mismatch matrix is eight rows rather than eleven, the three "requested as a String" rows having become @string-typing scenarios of their own; and the maximal declaration a Java provider can publish now carries @string-typing between @numeric-coercion and @targeting, which is the assertion that would otherwise let a new capability reach a report unnoticed. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 2 +- .../contrib/tools/tck/CanonicalScenarioGuardTest.java | 6 ++++-- .../contrib/tools/tck/ConformanceReportPluginTest.java | 5 +++++ 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index 61d6faa1b..f39c66948 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -57,7 +57,7 @@ line fails the build rather than producing a report that names a revision which did not produce its scenarios. --> - aa2ad24f5a14ae2b5756df0b6d23f493f39507e6 + d47a66ebb9500706e5bded7799d0e49aa1e86bfd 3.27.7 4.3.0 2.22.1 diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java index a53a0b033..d95bb943b 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/CanonicalScenarioGuardTest.java @@ -50,11 +50,13 @@ void theCanonicalSetIsReadFromTheArtifact() { .as("every canonical scenario comes from a packaged feature file") .allSatisfy(ref -> assertThat(ref.toString()).startsWith(ProviderTck.FEATURES + "/")); - // The eleven rows of the type-mismatch matrix share one name, so a set that counted scenarios + // The eight rows of the type-mismatch matrix share one name, so a set that counted scenarios // by name would see one of them. Each Examples row is its own line and its own entry. + // Eleven until specification revision d47a66eb moved the three "requested as a String" rows + // onto @string-typing, where a backend that stores every value as a string can withhold them. assertThat(canonical) .filteredOn(ref -> ref.toString().contains("Requesting the wrong type returns the code default")) - .hasSize(11); + .hasSize(8); } @Test diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index 03f7989fd..378bbc8c6 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -347,6 +347,11 @@ void aReservedCapabilityCannotReachTheDeclaration() { Capability.DISABLED_FLAGS.tag(), Capability.UNAVAILABLE_INIT.tag(), Capability.NUMERIC_COERCION.tag(), + // @string-typing gates the four scenarios that ask a non-string flag + // through the String accessor. Gated because every value has a string + // representation, so a backend that stores flag values as strings has no + // mismatch to report and withholds it without being non-conformant. + Capability.STRING_TYPING.tag(), // @large-integers is absent, and for a different reason from @caching's. // Scenarios do carry it -- Go and JavaScript run them -- but the Java SDK's // integer accessor is 32 bits, so no Java provider can be asked for 2^53 - 1 From da38b4f42ad5ab7833673cad438b8736f54d8f61 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Wed, 16 Sep 2026 08:58:24 +0200 Subject: [PATCH 20/22] chore(tck): follow the re-pin to bda599f1 in the revision a report names tck.spec.revision is the fourth thing a re-pin moves, after the submodule gitlink, PINNED_REVISION and PINNED_DIGEST. ConformanceReportPluginTest already fails the build when it disagrees with PINNED_REVISION, so this is the check reporting rather than a convention being remembered. The maximal-claim assertion gains @fully-typed-values, which is not bookkeeping: it is containsExactly over Capability.declarable() in vocabulary order, so a new declarable capability has to be named here or the report's most complete declaration is not the most complete declaration. CanonicalScenarioGuardTest's count of eight is unchanged and its note about d47a66eb still describes what that revision did, so neither moves. The split took a row out of one outline and gave it a scenario of its own, which leaves the canonical total where it was. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 2 +- .../tools/tck/ConformanceReportPluginTest.java | 13 +++++++++---- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index f39c66948..0e2f20508 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -57,7 +57,7 @@ line fails the build rather than producing a report that names a revision which did not produce its scenarios. --> - d47a66ebb9500706e5bded7799d0e49aa1e86bfd + bda599f1db440aa8d395d1d3af7b9b3cc3103b98 3.27.7 4.3.0 2.22.1 diff --git a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java index 378bbc8c6..dbf970cf6 100644 --- a/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java +++ b/tools/tck/src/test/java/dev/openfeature/contrib/tools/tck/ConformanceReportPluginTest.java @@ -347,11 +347,16 @@ void aReservedCapabilityCannotReachTheDeclaration() { Capability.DISABLED_FLAGS.tag(), Capability.UNAVAILABLE_INIT.tag(), Capability.NUMERIC_COERCION.tag(), - // @string-typing gates the four scenarios that ask a non-string flag - // through the String accessor. Gated because every value has a string - // representation, so a backend that stores flag values as strings has no - // mismatch to report and withholds it without being non-conformant. + // @string-typing gates the two outline rows that ask a boolean and an + // integer flag through the String accessor. Gated because every value has a + // string representation, so a backend that stores flag values as strings has + // no mismatch to report and withholds it without being non-conformant. Capability.STRING_TYPING.tag(), + // @fully-typed-values asks the same of a float and a structure, and is a + // separate claim because a store can record booleans and integers natively + // while keeping those two as text. Both scenarios carry @string-typing as + // well, so the maximal claim has to include both tags to reach them. + Capability.FULLY_TYPED_VALUES.tag(), // @large-integers is absent, and for a different reason from @caching's. // Scenarios do carry it -- Go and JavaScript run them -- but the Java SDK's // integer accessor is 32 bits, so no Java provider can be asked for 2^53 - 1 From c114e34013811b007fd53ede3001301e2728e3df Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Wed, 16 Sep 2026 21:48:54 +0200 Subject: [PATCH 21/22] chore(tck): follow the spec pin in the revision a report names The fourth of the coordinated moves: gitlink and PINNED_REVISION are on the base branch, this property is what a report records as tck.specRevision. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index 0e2f20508..cc2a5e02d 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -57,7 +57,7 @@ line fails the build rather than producing a report that names a revision which did not produce its scenarios. --> - bda599f1db440aa8d395d1d3af7b9b3cc3103b98 + ff68adb4c7617ad2d980988241e92603bc247926 3.27.7 4.3.0 2.22.1 From 46a278565043aa053acfae74b55b16d962ccff30 Mon Sep 17 00:00:00 2001 From: Simon Schrottner Date: Thu, 1 Oct 2026 10:10:48 +0200 Subject: [PATCH 22/22] chore(tck): pin the assets branch in the revision a report names The fourth coordinated place: gitlink and PINNED_REVISION are on the base branch, this property is what a report records as tck.specRevision. Signed-off-by: Simon Schrottner --- tools/tck/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/tck/pom.xml b/tools/tck/pom.xml index cc2a5e02d..d07282ce5 100644 --- a/tools/tck/pom.xml +++ b/tools/tck/pom.xml @@ -57,7 +57,7 @@ line fails the build rather than producing a report that names a revision which did not produce its scenarios. --> - ff68adb4c7617ad2d980988241e92603bc247926 + 8374621763f04a3d0785a911f12035db4794179e 3.27.7 4.3.0 2.22.1