Skip to content

test(flow): prove compatibility and provider acceptance scenarios - #61

Merged
szmyty merged 1 commit into
mainfrom
test/flow-30-acceptance-scenarios
Sep 26, 2026
Merged

szmyty merged 1 commit into
mainfrom
test/flow-30-acceptance-scenarios

Conversation

@szmyty

@szmyty szmyty commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Flow's provider kit had strong individual tests but no executable, completeness-checked coverage record for the single-execution acceptance boundary. This adds the Flow #30 matrix and makes missing or contradictory scenario evidence fail the PR gate.

Changes

  • Add 81 stable scenarios: 19 resolution, 22 contract, 22 artifact, 14 provider, and 4 privacy cases.
  • Run each recipe in two fresh roots and compare normalized evidence, including the actual kit package/executable identities.
  • Exercise compatibility and pin disagreements, conflict and fallback policy, malformed/versioned contracts, artifact escape/link/kind/digest/identity/staleness failures, exit-zero refusal, bounded streams, and a real argv/environment/stderr privacy probe.
  • Produce a versioned JSON report only after Rust assertions, receipt completeness, expected outcomes, budgets, and unchanged source bytes all pass. Add five report-integrity regression tests.
  • Retain successful portable reports in the existing Rust CI jobs for seven days. Raw failure logs stay local.
  • Document scope, budgets, recovery boundaries, and gaps; keep FLO-Q03 active and identify [FLO-3.3] Persist versioned plans, runs, checkpoints, and recovery decisions聽#49 as the next central checkpoint after merge.

No production library API or provider contract changes. The test profile omits debug symbols because the kit repeatedly copies and hashes executable bytes.

Validation

Local validation passed:

  • Rust 1.85 and stable: all-target tests, including 81 scenarios 脳 two fresh roots.
  • Strict Clippy on both toolchains; formatting.
  • Rust documentation tests and all three executable examples.
  • Locked offline package build/verification, including the new fixtures and runner.
  • Contract validation, deterministic scenario sources, architecture specifications, repository skills/agents, and five Python report-integrity tests.
  • git diff --check.

Base: d7d8a6d38a287fd41e64304c777a22a9d28a311d, with green main CI. Hosted CI 36206682449 passed on final head 20dcfaea55c8fd9f82f299b8ce4c79942b369f85: Rust 1.85.0, stable Rust, and repository validators. Both normalized-report artifacts are retained. The PR is open, non-draft, and mergeable for maintainer review.

Boundaries and rollback

Synthetic conformance does not establish real released-provider compatibility, provider-native format validation, publisher authentication, an OS sandbox, filesystem atomicity, descriptor-bound launch, process-tree containment, or durable recovery. Free-form provider evidence and authority profiles are not a generic redaction surface; only the allowlisted receipts are portable. Artifact acceptance covers declared bindings, not arbitrary unreported filesystem effects.

Revert this PR to remove the matrix, report gate, documentation additions, and test-profile size adjustment. It introduces no persisted product state or migration.

Closes #30
Parent: #13
Suite coordinator: #11
Roadmap-Step: FLO-Q03

Add the Flow #30 compatibility, artifact, provider-failure, and privacy matrix with deterministic receipts and PR coverage evidence.

Roadmap-Step: FLO-Q03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FLO-13.3] Prove compatibility, artifact, and provider-failure scenarios

1 participant