GATE conformance checks, self-assessment, operational runbooks, and conformance runner - v1.3.0
20 conformance checks with test procedures and evidence requirements, a fillable conformance report template, BigQuery evidence correlation queries, 9 operational runbooks for Day-2 operations, four standards- mapping YAML files (OWASP AISVS, MITRE ATLAS, NIST SSDF), and a runner that automates 9 of 20 checks by default (11 of 20 when quality and baseline bundle stores are configured).
Framework: https://deterministicagents.ai
Organisation: https://github.com/deterministic-agents
Documentation: CC BY 4.0 - Andrew Stevens · Code: MIT
Compatible with GATE v1.4. Ships Check20 (C20 output classification),
conditional automation for Check17 (C18) and Check18 (C19), the four
standards-mapping YAML files under mappings/, and the classification
field backfilled on every check entry in self_assessment.yaml.
Conformance runner: python -m runner.cli run --config gate-conformance.yaml.
The runner reports 9 AUTOMATED and 11 PARTIAL by default. With
quality_bundle_store_uri and baseline_bundle_store_uri configured
in the runner config (see runner/templates/config.example.yaml), the
runner additionally verifies the active quality and baseline bundle
signatures inline; Check17 and Check18 upgrade to AUTOMATED and the
totals become 11 AUTOMATED and 9 PARTIAL. Both totals assume the
operator's deployment includes at least one bounded or high_privilege
tier agent. Sandbox-only deployments see Check20 reported as SKIP per
the C20 adoption-path placement; the totals become 9 AUTOMATED and 10
PARTIAL in that case.
Backward-compatible with v1.2.0 operator configs: the two new config
keys default to absent and reproduce the v1.2.0 PARTIAL behaviour for
Check17 / Check18 exactly. See CHANGELOG-v1.3.0.md for the full
release entry.
Compatible with GATE v1.3. First gate-conformance Release Object since v1.0.0 - the v1.1.0 tag carried Check16-Check19 (C17 / C18 / C19), seven new evidence correlation queries, and three new runbooks (RB-07 C17 candidate backlog, RB-08 C18 quality gate outage, RB-09 C19 drift response) onto main but no Release was cut. v1.2.0 ships that content together with the new conformance runner.
Conformance runner: python -m runner.cli run --config gate-conformance.yaml.
Automates 9 of the 19 checks against your evidence store (Check01, 03,
04, 05, 08, 09, 10, 12, 13) and returns PARTIAL with structured
manual_steps for the other 10. Supports sqlite (built in) and
BigQuery (optional dependency). Runner output matches the v1.1
conformance report template shape. See runner/README.md for the full
quickstart and the per-tier behaviour notes on Check16, Check17, and
Check18.
gate-conformance/
├── self_assessment.yaml # 20 checks with test procedures
├── conformance_report_template.yaml # Fillable report for audit submission
├── evidence_correlation.sql # BigQuery queries for evidence chain traversal
├── operational_runbooks.yaml # 9 Day-2 runbooks
├── mappings/ # Standards mappings (per-requirement detail)
│ ├── owasp-aisvs.yaml # OWASP AISVS per-requirement mapping
│ ├── owasp-aisvs-triage.yaml # Triage source for v1.5 W4 work
│ ├── mitre-atlas.yaml # MITRE ATLAS per-technique mapping
│ └── nist-ssdf.yaml # NIST SSDF narrow intersection (C03, C05)
├── runner/ # Conformance runner CLI (v1.3.0)
│ ├── cli.py, config.py, report.py
│ ├── adapters/ (sqlite, BigQuery)
│ ├── checks/ (Check01..Check20)
│ ├── templates/ (config.example.yaml with bundle_stores: block)
│ └── README.md (quickstart, tier behaviour, extension guide)
└── tests/runner/ # 85 pytest cases against sqlite fixtures
# (84 pass + 1 skip; the skip is a drift
# guard for cross-workstream identifier
# consistency that requires the W3 tree
# adjacent on disk)
| Tier | Required checks |
|---|---|
sandbox |
Check04, Check08, Check12 |
bounded |
All sandbox + Check01-03, Check05-08, Check10-13, Check16-19, Check20 |
high_privilege |
All bounded + Check09, Check14, Check15 |
Check16-19 (the v1.3 controls) apply at bounded and high_privilege.
Check20 (the v1.4 control) applies at bounded and high_privilege;
it is N/A at sandbox per the C20 adoption-path placement (paper-update
07). Check17 and Check18 branch on tier inside the check, and v1.3.0
adds a conditional automation upgrade when their respective bundle
store URIs are configured. See runner/README.md for the per-tier
behaviour notes and the bundle-store configuration.
For each check in your tier, follow the test_procedure instructions,
collect the listed evidence_required artifacts, and set status to
PASS, FAIL, or NOT_APPLICABLE.
Complete the deployment, checks, metrics, and retention_profile
sections. Set summary.overall_status to CONFORMANT, NON_CONFORMANT,
or PARTIAL.
Run the queries against your evidence store (BigQuery or adapt for your query engine). The most important query for conformance is the Check01 query - zero rows means zero tool calls without a policy decision record.
| Check | Control | What it verifies |
|---|---|---|
| Check01 | C05 | Zero tool executions without a policy decision record |
| Check02 | C05 | No bypass paths to tools (network + IAM + SDK) |
| Check03 | C01 | 100% verified workload identity on privileged requests |
| Check04 | C05 | Schema validation rejects malformed tool inputs |
| Check05 | C11 | Ledger chain integrity verifies PASS; WORM retention active |
| Check06 | C10 | Replay reproduces incident run with matching hashes |
| Check07 | C06 | Circuit breaker stops side effects within SLO |
| Check08 | C07 | Budget exhaustion denies tool calls (not just logs) |
| Check09 | C12 | 100% signature coverage for financial/irreversible/infra tools |
| Check10 | C01 | Memory ACLs enforced at retrieval time; cross-tenant blocked |
| Check11 | C08 | Poisoning detection quarantines injected documents |
| Check12 | C13 | Evidence chain traversable: semantic → ledger → policy → replay |
| Check13 | C03 | Policy bundle hash in evidence matches deployed bundle |
| Check14 | C09 | HITL-required tool calls blocked without signed approval |
| Check15 | C14 | Multi-agent messages: signature + nonce + expiry enforced |
| Check16 | C17 | Unenrolled workloads are detected and remediated within TTL |
| Check17 | C18 | Memory retrievals pass quality gates before reaching the model |
| Check18 | C19 | Model behaviour is baselined and monitored for drift at cadence |
| Check19 | C19/C16 | Drift and adversarial events are emitted as distinct ledger event types |
| Check20 | C20 | Every final agent response classified and gated; fail-closed default at high_privilege |
| Metric | Target |
|---|---|
| Tool calls without policy decision record | 0 |
| Bypass detections | 0 |
| Attestation coverage | 100% |
| Ledger integrity status | PASS |
| Replay success rate | ≥ 95% |
| Breaker containment SLO | ≥ 99% |
| High-impact signature coverage | 100% |
| HITL-required without approval | 0 |
| Cross-tenant memory violations | 0 |
| Unenrolled workload identities outside remediation TTL (C17) | 0 |
| Memory retrievals without a quality_decision_id at bounded+ (C18) | 0 |
| Days without a drift_decision event at bounded+ (C19) | 0 |
| Drift / adversarial event-type crossover events (C19/C16) | 0 |
| High_privilege classification events with empty obligations (C20 fail-closed guardrail) | 0 |
| Output classification events whose bundle_hash does not resolve to a signed bundle (C20) | 0 |
operational_runbooks.yaml contains the minimum set of Day-2 runbooks:
| Runbook | Trigger |
|---|---|
| RB-01 Break-glass stop | Suspected compromise, runaway execution, unsafe actions |
| RB-02 Policy bundle rollback | Policy change causes unexpected denials or allows |
| RB-03 Incident replay | Investigating a run, validating a fix before re-enabling |
| RB-04 HITL outage | HITL service unavailable; approver unavailability |
| RB-05 Invariant bundle update | New high-impact tool; financial limit adjustment |
| RB-06 Agent decommission | Agent purpose complete; version replacement; retirement |
| RB-07 C17 candidate backlog escalation | Discovered-but-unenrolled candidates exceed threshold or TTL expires |
| RB-08 C18 quality gate outage | Memory quality gate service unavailable or failing open |
| RB-09 C19 drift response | Drift threshold breach requires tier reduction or escalation |
Each runbook includes: trigger, severity, SLO, step-by-step actions, evidence capture requirements, and exit criteria.
The CLI conformance runner ships in this release (v1.3.0). Run all 20 checks against your evidence store with:
python -m runner.cli run --config gate-conformance.yamlThe runner automates 9 of the 20 checks by default (Check01, 03, 04,
05, 08, 09, 10, 12, 13) and returns PARTIAL with structured
manual_steps for the other 11. PARTIAL is not failure - the report
carries the specific artefact each PARTIAL check still needs from the
operator. With quality_bundle_store_uri and baseline_bundle_store_uri
configured in the runner config (see runner/templates/config.example.yaml),
Check17 and Check18 upgrade to AUTOMATED; the runner reports 11
AUTOMATED and 9 PARTIAL.
See runner/README.md for the full quickstart, per-tier behaviour
notes on Check16-18-20, the bundle-store configuration, exit codes,
and how to add a custom check or evidence-store backend.
self_assessment.yaml remains the normative manual baseline. The
runner and the self-assessment are designed to be used together:
runner output for the automatable subset, self-assessment for the
rest.
The framework paper carries 6-12 line orientation entries for each
mapped standard in the Standard Mappings appendix. The per-requirement
and per-technique detail lives under mappings/:
| File | Standard | Detail |
|---|---|---|
mappings/owasp-aisvs.yaml |
OWASP AISVS v1.0 (commit aadf83a) | Per-requirement mapping for C01-C20 |
mappings/owasp-aisvs-triage.yaml |
OWASP AISVS triage | Structured input for v1.5 W4 |
mappings/mitre-atlas.yaml |
MITRE ATLAS 2026.05 (data format 6.0.0) | Per-technique mapping; C16 spec carries the CI harness coverage list |
mappings/nist-ssdf.yaml |
NIST SSDF SP 800-218 v1.1 | Narrow intersection at C03 and C05 |
Operators reading these mappings should treat them as operational orientation, not legal or compliance certification. The framework paper's Standard Mappings appendix carries the same caveat.
| Repo | What it is |
|---|---|
| gate-contracts | JSON Schema contracts (canonical dependency) |
| gate-python | Python reference library |
| gate-policies | OPA/Rego policy and invariant bundles |
| gate | Framework paper, spec site source |