feat(provider-tck): emit a machine-readable conformance report

tools/tck/README.md

@@ -345,6 +345,176 @@ the exclusion **and** narrows Surefire's includes to it:
 **Both halves are needed.** Dropping alone runs the module's unit tests alongside the suites;
 narrowing alone leaves the exclusion in force and runs nothing.
 
+## The canonical set cannot be reduced
+
+Extending the suite is safe by convention. Shrinking it is what a conformance suite has to prevent,
+because a run that asks all but two of the canonical questions and reports success is
+indistinguishable, in every artifact it produces, from one that asked every question there is.
+
+`CanonicalScenarioGuard` is an ordinary JUnit test that the suite selects, and it fails the build if
+this run is set up to execute less than the canonical set:
+
+- a feature file added to `gherkin/`, or shadowing a canonical one — the selected scenarios no
+  longer match what this artifact ships, which it reads from its own JAR rather than through the
+  classpath
+- `cucumber.filter.tags` or `cucumber.filter.name` — Cucumber applies these by skipping scenarios at
+  execution, so the run is filtered however the plan looks
+- selectors or glue overridden in your `junit-platform.properties`
+
+It checks the setup rather than counting afterwards: both the discovered plan and the run's filter
+configuration are settled before the first scenario, so the check needs no backend and takes no
+measurable time. Where its result appears in the run depends on the order the JUnit Platform executes
+the suite's two engines in, which is not specified. Extension scenarios are ignored: the check is
+defined over `gherkin/` alone.
+
+Narrowing a run legitimately is what `capabilities()` is for — those scenarios are reported as
+skipped with a reason, which a filtered scenario is not. To filter anyway while debugging, set
+`-Dtck.partial=true` (or `TCK_PARTIAL`). The guard then reports itself as
+**skipped** rather than passed, so the run states that its canonical set was not verified.
+
+What the guard does not establish is that the canonical files contain what they should — a
+replacement placing its scenarios on the same lines would satisfy it. That is covered better
+elsewhere: the results stream carries the `source` of every feature that executed, and
+`tck.specRevision` says which revision it should match.
+
+## Conformance reports
+
+Set `TCK_REPORT_DIR` and each suite writes two files: an envelope conforming to the
+[report schema][report-schema] in the specification, and the run's results as a
+[Cucumber Messages][messages] stream.
+
+```console
+$ TCK_REPORT_DIR=./reports mvn -Ptck -pl providers/flagd test
+$ ls reports/
+flagd-in-process.json  flagd-in-process.ndjson  flagd-rpc.json  flagd-rpc.ndjson
+```
+
+`-Dtck.report.dir=...` does the same thing and is often easier to pass through Maven. The
+environment variable is the portable spelling — every language's TCK reads it, so one cross-language
+CI job can set one thing.
+
+It is an environment variable rather than a method on `ProviderTckHarness` so that emitting a report
+is a property of the run and not of the code: CI sets it, a developer running the suite locally does
+not, and no adopter changes a line to publish one. Unset means no report, which is not an error.
+Several suites in one JVM each write their own pair, so flagd's two resolvers do not collide.
+
+### The results are not a format this project defines
+
+The `.ndjson` is a Cucumber Messages stream, produced by Cucumber's own `MessageFormatter` — the
+same class the built-in `message:<path>` plugin instantiates, so the bytes are what
+`--plugin message:...` would have written. It already carries everything a per-scenario report would
+have had to invent: the outcome of every scenario, its tags including any set on an individual
+`Examples` block, an exact Scenario Outline row identity, and the source of every feature that ran.
+
+The plugin exists rather than the built-in one because a `@ConfigurationParameter` value is a
+compile-time constant, so the built-in plugin's path cannot be derived from the directory the run
+asked for — and flagd's two suites would write to the same file.
+
+Reading it needs no special tooling, but it does need one thing understood: **a scenario's outcome
+is the most severe result among its steps**, hooks included. `testCaseFinished` carries no status of
+its own. That is what makes a capability-gated skip truthful, because the aborted `@Before` hook
+contributes a `SKIPPED` result that outranks every step it stopped from running.
+
+```console
+$ jq -c 'select(.testStepFinished) | .testStepFinished
+         | {c: .testCaseStartedId, s: .testStepResult.status}' reports/flagd-rpc.ndjson \
+    | jq -s 'group_by(.c) | map({s: (map(.s) | if any(. == "FAILED") then "FAILED"
+                                              elif any(. == "SKIPPED") then "SKIPPED"
+                                              else "PASSED" end)})
+             | group_by(.s) | map({(.[0].s): length}) | add'
+{
+  "PASSED": 28,
+  "SKIPPED": 1
+}
+```
+
+The [`cucumber-query`](https://github.com/cucumber/messages/tree/main/java) helpers do this properly
+and in several languages; the above is only to show that the fact is in the file.
+
+### What identifies a scenario
+
+`pickle.astNodeIds`. For a scenario compiled from a Scenario Outline it is
+`[scenario id, table row id]`, and the second entry resolves in the `gherkinDocument` message to the
+`Examples` row the scenario was built from. Feature and name are not enough — the type-mismatch
+matrix in `errors.feature` is eleven rows sharing one name — and this is exact rather than derived:
+
+```console
+$ jq -c 'select(.pickle) | .pickle
+         | select(.name == "Requesting the wrong type returns the code default")
+         | {id, row: .astNodeIds[1]}' reports/flagd-rpc.ndjson | head -3
+{"id":"6c8debd2-...","row":"ab8b4a4b-..."}
+{"id":"a7c76b0a-...","row":"63d6d6c8-..."}
+{"id":"fecd333d-...","row":"bbd7f5ee-..."}
+```
+
+An earlier version of this module reverse-engineered the same fact by re-parsing the feature source
+and matching a pickle's reported line number against the Examples tables. The stream states it
+outright, which is the whole argument for a standard format over one we maintain.
+
+### What the envelope is for
+
+A Messages stream cannot say what it was a test *of*. The envelope carries the five things no
+standard results format identifies:
+
+- **`provider`** — what the provider calls itself through its own metadata, not the suite name. The
+  suite name is chosen to read well in a failure message (`flagd-rpc`), which makes it the
+  *configuration*, and it is reported as such. One provider with two materially different modes
+  produces two reports that are not interchangeable. Derived from the suite class name
+  (`MyProviderInProcessTest` → `my-provider-in-process`) unless the suite overrides
+  `ProviderTckHarness.configuration()`, which flagd's two do — a class in a package already called
+  `.../flagd/tck/` is named `InProcessTest`, and `in-process` on a report does not say whose.
+- **`sdk`** — read from the classpath rather than declared, because the TCK depends on an SDK version
+  *range* so that adopting it can never force an upgrade. What a consumer actually ran against is
+  only knowable at runtime.
+- **`tck`** — which implementation asked the questions, and `specRevision`, the open-feature/spec
+  commit the packaged artifacts came from. Baked into the JAR at build time from this module's POM:
+  the artifacts travel in the JAR, the repository they came from does not. The executed Gherkin no
+  longer rests on that pin alone — the stream carries the `source` of every feature, so it can be
+  diffed against the revision — but the pin is what identifies the two artifacts the stream does not
+  carry, `flags/canonical-flags.json` and `openapi/control-api.yaml`.
+- **`backend`** — what the provider was pointed at, and `controlApi`, which of the two control
+  contracts drove it. Always present, both of them: the same scenarios passing over the HTTP control
+  API and passing through in-process manipulation of a provider that *does* have a backend are not
+  the same claim, and this is the only field that separates them, so an omission would be an
+  unfalsifiable claim rather than no claim at all. See [Identifying the
+  run](#identifying-the-run).
+- **`declaration`** — the capability set the provider claims. This is an **input** to reading the
+  results, not a summary of them, which is why it cannot be derived from the stream. The stream says
+  a scenario was skipped; only the declaration says whether that is because the provider declines the
+  capability it needed. Given the declaration and a scenario's tags — both present — the reason for
+  each skip follows, so it does not have to be transported per scenario.
+
+  **One skip does not follow from it, and that one is why the reason travels anyway.** A capability
+  this SDK cannot express is absent from every Java declaration, and absent for a reason that says
+  nothing about the provider — `@large-integers` is not there because `Client.getIntegerDetails` is
+  32 bits, not because anyone declined it. A reader who inferred *"the provider declined"* from that
+  absence would be reading a decision into something no Java provider was ever offered. The gate's
+  reason is carried on the hook result that produced the skip, and its wording is deliberately unlike
+  an undeclared capability's, so the two are distinguishable in the results themselves. See
+  [Declaring capabilities](#declaring-capabilities).
+
+`knownDeviations` is the one thing neither the stream nor the declaration can express: whether a
+withheld capability is a limitation or a bug. See [Known deviations](#known-deviations).
+
+`results.digest` covers the `.ndjson`, so a consumer that fetched the two separately can tell that
+what it has is what the envelope describes.
+
+### What the report is for
+
+This suite promises that a scenario skipped for an undeclared capability is reported as skipped with
+the reason and *never* as passed — and a promise is not a check. The stream records every scenario
+individually, so a consumer can verify the rule instead of trusting a runner's headline number. Go's
+runner counts capability-gated skips in its **passed** tally, which is exactly the failure mode this
+makes impossible to hide.
+
+Every scenario appears exactly once, whatever happened to it. A report that quietly omitted the
+scenarios it did not run would satisfy every rule above and still mislead, because a reader would
+have no way to know how many questions went unasked. `ConformanceReportPluginTest` runs a fixture
+suite through the real Cucumber engine and asserts both properties over the emitted stream.
+
+[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json
+[messages]: https://github.com/cucumber/messages
+
 ## Extending it
 
 A provider with features of its own — flagd's `fractional` targeting, a vendor's proprietary mode —

tools/tck/pom.xml

@@ -25,6 +25,39 @@
         -->
         <tck.spec.checkout.skip>false</tck.spec.checkout.skip>
 
+        <!--
+            THE CONFORMANCE ARTIFACTS THIS BUILD CARRIES
+            ============================================
+            A conformance report has to say which questions were asked, not only what the answers
+            were, so it records the open-feature/spec commit the packaged Gherkin, flag set and
+            control API came from. It is filtered into
+            src/main/resources-filtered/.../tck-build.properties and packaged in the JAR,
+            because the repository it came from is not.
+
+            The executed Gherkin no longer rests on this pin alone: the results stream carries the
+            `source` of every feature that ran, so a consumer can diff what executed against what
+            this revision contains. The pin is what identifies the two artifacts the stream does not
+            carry — flags/canonical-flags.json and openapi/control-api.yaml.
+
+            This must equal the `spec` submodule's pinned commit, which is where the artifacts
+            actually come from — the copies under src/main/resources are produced from it at
+            generate-resources and are gitignored. Checkable in one command, from this directory:
+
+              git -C spec rev-parse HEAD
+
+            A property rather than a value read from git during the build because Maven has no way
+            to capture a command's output into a property without another plugin, and the submodule
+            pin already moves only by deliberate commit.
+
+            It is no longer only a convention that they agree: this value reaches the classpath
+            through tck-build.properties, and ConformanceReportPluginTest asserts that what
+            TckBuildInfo reads back equals CanonicalAssetDigestTest.PINNED_REVISION, which that
+            test has in turn checked against the packaged assets by digest. So the chain from the
+            gitlink to the revision a report publishes is closed, and a re-pin that forgets this
+            line fails the build rather than producing a report that names a revision which did not
+            produce its scenarios.
+        -->
+        <tck.spec.revision>8374621763f04a3d0785a911f12035db4794179e</tck.spec.revision>
         <assertj.version>3.27.7</assertj.version>
         <awaitility.version>4.3.0</awaitility.version>
         <jackson-databind.version>2.22.1</jackson-databind.version>
@@ -109,6 +142,55 @@
             <artifactId>cucumber-junit-platform-engine</artifactId>
         </dependency>
 
+        <!--
+            Cucumber core and the message types — already on the classpath as transitive
+            dependencies of cucumber-junit-platform-engine, and declared here because
+            ConformanceReportPlugin compiles against them directly.
+
+            The conformance report carries its results as a Cucumber Messages stream and produces
+            that stream with io.cucumber.core.plugin.MessageFormatter, which is the same class the
+            built-in `message:<path>` plugin instantiates. Cucumber's own formatter rather than one
+            of ours, so the results cannot drift from what the built-in message plugin writes; only
+            the output path differs, because a plugin configured through an annotation cannot have a
+            path derived from the run. Versions are managed by cucumber-bom in the parent POM.
+        -->
+        <dependency>
+            <groupId>io.cucumber</groupId>
+            <artifactId>cucumber-core</artifactId>
+            <!-- version managed by cucumber-bom in parent -->
+        </dependency>
+        <dependency>
+            <groupId>io.cucumber</groupId>
+            <artifactId>messages</artifactId>
+            <!-- version managed by cucumber-bom in parent -->
+        </dependency>
+
+        <!--
+            Gherkin — CanonicalScenarios compiles this artifact's own feature files to work out what
+            the canonical scenario set is, so that CanonicalScenarioGuard can tell whether a run
+            discovered all of it. Cucumber's own parser rather than a hand-rolled one, because
+            "which scenarios does this file contain" has to mean exactly what it means to the runner.
+            Already on the classpath transitively through cucumber-core; declared because we compile
+            against it. Version managed by cucumber-bom in the parent POM.
+        -->
+        <dependency>
+            <groupId>io.cucumber</groupId>
+            <artifactId>gherkin</artifactId>
+            <scope>compile</scope>
+        </dependency>
+
+        <!--
+            JUnit Jupiter API — CanonicalScenarioGuard is a Jupiter test that the suite selects, so
+            that a reduced canonical set fails the build the way any other failing test does. A
+            listener cannot do it: the JUnit Platform catches and logs whatever a TestExecutionListener
+            throws, which is precisely a silent pass.
+        -->
+        <dependency>
+            <groupId>org.junit.jupiter</groupId>
+            <artifactId>junit-jupiter-api</artifactId>
+            <scope>compile</scope>
+        </dependency>
+
         <!-- PicoContainer object factory — injects TckState into every step class -->
         <dependency>
             <groupId>io.cucumber</groupId>
@@ -217,6 +299,23 @@
     </dependencies>
 
     <build>
+        <!--
+            The conformance artifacts are copied verbatim; only the generated build-info properties
+            are filtered. Filtering the feature files would be a portability hazard rather than a
+            convenience: a scenario is a byte-for-byte shared definition across four languages, and
+            a stray ${...} in one would silently make this language's copy different.
+        -->
+        <resources>
+            <resource>
+                <directory>src/main/resources</directory>
+                <filtering>false</filtering>
+            </resource>
+            <resource>
+                <directory>src/main/resources-filtered</directory>
+                <filtering>true</filtering>
+            </resource>
+        </resources>
+
         <plugins>
             <!--
                 Phase 1 – move the spec submodule's WORKING TREE to the pinned commit.