From 201cf1cc6689d74c059bd79c7a590d4369c149c1 Mon Sep 17 00:00:00 2001 From: Michael Lieberman Date: Thu, 27 Aug 2026 17:03:12 -0400 Subject: [PATCH 1/2] docs: CEL context reference for control authors (closes #357) Adds `docs/CEL_CONTEXT.md` with a per-handler breakdown of the variables CEL sees in a control pass's `expr` field. Sourced from the actual handler evidence + CELContext dataclass -- not from CLAUDE.md's audience-mismatched notes. Covers what the current implementation actually exposes: - `exec`: `output.stdout`, `output.stderr`, `output.exit_code`, `output.command`, `output.json` (only when `output_format = "json"`) - `file_exists`: `output.found_file`, `output.relative_path`, `output.files_checked` - `regex` / `pattern`: `output.files_found`, `output.found_files`, `output.files_checked` - `mcp`: `result.*` (evaluated by `_eval_cel_over_result` before the handler returns, not the standard post-step path) - Ambient bindings: `project.*`, `repo.*`, `context.*` - Custom functions: `file_exists(path)`, `json_path(obj, jmespath)` Explicitly calls out which handlers do NOT evaluate `expr` today (`llm_eval`, `llm_extract`, `manual_steps`, `file_create`, `api_call`, `project_update`, `yaml_inject`) so authors do not silently write a no-op. Cross-linked from `docs/architecture/framework-design.md` (the exec handler's `expr` row) and `docs/HANDLER_AUTHORING.md` (the CEL basics section), with the old inline stale variable list in HANDLER_AUTHORING replaced by a short summary pointing at the reference. --- docs/CEL_CONTEXT.md | 196 ++++++++++++++++++++++++++ docs/HANDLER_AUTHORING.md | 31 ++-- docs/architecture/framework-design.md | 2 +- 3 files changed, 212 insertions(+), 17 deletions(-) create mode 100644 docs/CEL_CONTEXT.md diff --git a/docs/CEL_CONTEXT.md b/docs/CEL_CONTEXT.md new file mode 100644 index 00000000..b874e693 --- /dev/null +++ b/docs/CEL_CONTEXT.md @@ -0,0 +1,196 @@ +# CEL Context Reference for Control Authors + +This page lists every context variable and custom function available in +a CEL `expr` field on a control pass. If you are authoring a new +control, this is the reference. If you are grepping other controls for +"what can I write in `expr`?", start here instead. + +Cross-linked from [`docs/HANDLER_AUTHORING.md`](./HANDLER_AUTHORING.md) +and [`docs/architecture/framework-design.md`](./architecture/framework-design.md). + +## Where `expr` runs + +Two placement rules: + +1. **Post-handler CEL (most passes).** After the pass's `handler` runs, + the orchestrator evaluates `expr` against the handler's evidence + (see [`orchestrator.py`](../packages/darnit/src/darnit/sieve/orchestrator.py)). + Truthy `expr` + PASS = PASS. Falsy `expr` + PASS = INCONCLUSIVE + (handler and CEL disagree; keep going down the pass list). See + [`specs/020-definitive-fail-verdict/contracts/cel-post-step.md`](../specs/020-definitive-fail-verdict/contracts/cel-post-step.md) + for the full truth table. + +2. **In-handler CEL (`mcp` only).** `handler = "mcp"` evaluates `expr` + against the JSON result of the MCP tool call before deciding + PASS/FAIL. See + [`_eval_cel_over_result` in `builtin_handlers.py`](../packages/darnit/src/darnit/sieve/builtin_handlers.py). + +## Available variables per handler + +The variables a CEL expression can reference depend on which handler +produced the evidence. + +### `exec` handler + +The runtime injects `output.*` from the subprocess result: + +| Variable | Type | Description | +|---------------------|-------------------|----------------------------------------------------------------------------------------| +| `output.stdout` | string | First 2000 chars of stdout | +| `output.stderr` | string | First 2000 chars of stderr | +| `output.exit_code` | int | Subprocess exit code | +| `output.command` | list | Resolved command (after `$VAR` substitution) | +| `output.json` | any | Parsed JSON body -- only present when the pass sets `output_format = "json"` AND stdout parses cleanly | + +```toml +# Exit-code + stdout non-empty +{ handler = "exec", command = ["git", "tag", "--list", "v*"], expr = 'output.stdout != ""' } + +# JSON-body assertion (declare output_format so `output.json` is populated) +[[controls."OSPS-AC-01.01".passes]] +handler = "exec" +command = ["gh", "api", "/orgs/$OWNER/settings"] +output_format = "json" +expr = 'has(output.json.two_factor_requirement_enabled) && output.json.two_factor_requirement_enabled == true' +``` + +Guard `output.json.foo` access with `has()`; a missing key raises a CEL +evaluation error, not a false PASS. + +### `file_exists` handler + +`output.*` carries the file-presence outcome: + +| Variable | Type | Description | +|-------------------------|--------------|--------------------------------------------------------------------| +| `output.found_file` | string | Absolute path of the match (present only when the handler PASSes) | +| `output.relative_path` | string | Path relative to repo root | +| `output.files_checked` | list | Every candidate the handler looked at | + +Most `file_exists` passes need no `expr` (the presence check is +already the verdict). Reach for CEL only when you want to constrain +which file matched, e.g. `expr = 'output.relative_path == "SECURITY.md"'`. + +### `regex` handler (`pattern` alias) + +| Variable | Type | Description | +|-------------------------|----------------------------|------------------------------------------------------------| +| `output.files_found` | int | Number of files matched | +| `output.found_files` | list | Relative paths of files that matched at least one pattern | +| `output.files_checked` | list | Files the handler scanned | + +The `matches` field the CLAUDE.md notes referenced is not present in +the current handler's evidence -- match structure lives inside the +handler's confidence + evidence shaping. If you need per-file match +detail in `expr`, add it to the handler's evidence first. + +### `mcp` handler + +`expr` is evaluated against the raw JSON return of the MCP tool. The +top-level variable is `result`, so: + +```toml +[[controls."OSPS-XX-YY".passes]] +handler = "mcp" +server = "scorecard" +tool = "get_repo_score" +args = { owner = "$OWNER", repo = "$REPO" } +expr = 'result.score >= 7.0' +``` + +There is no `output.*` binding for `handler = "mcp"` passes -- only +`result.*` and any project/repo bindings below. + +### Handlers that do NOT evaluate `expr` + +`llm_eval`, `llm_extract`, `manual_steps`, `file_create`, `api_call`, +`project_update`, `yaml_inject` do not run a post-step CEL evaluator on +their evidence. Writing an `expr` on their pass config is silently +ignored today; use the handler's own config keys to shape the verdict. + +## Bindings available regardless of handler + +The [`CELContext`](../packages/darnit/src/darnit/sieve/cel_evaluator.py) +dataclass carries additional bindings the runtime can inject when the +orchestrator constructs a full context (not the trimmed post-step +context most passes see). Ambient bindings that a control author may +reference: + +| Variable | Type | Populated when | +|-------------|---------------------------|-----------------------------------------------------------------| +| `project.*` | dict | `.project/project.yaml` was read for this audit run | +| `repo.*` | dict (path, owner, name) | Set by the audit driver on every audit | +| `context.*` | dict | Values the user answered via `darnit collect-context` / harness | + +Typical use: + +```toml +# Only apply this pass when project language is Python +{ handler = "exec", command = ["python", "-c", "print('ok')"], expr = 'project.language == "python"' } +``` + +`project.*` is populated from the `.project/` reader +([`dot_project.py`](../packages/darnit/src/darnit/context/dot_project.py)) +plus any control-side auto-detected values (`language`, +`ci_provider`, `platform`). + +## Custom CEL functions + +Both are registered in +[`CELEvaluator._build_custom_functions()`](../packages/darnit/src/darnit/sieve/cel_evaluator.py). + +### `file_exists(path: string) -> bool` + +Returns true if `path`, resolved relative to the repo root, exists. +Useful as a boolean guard inside a larger expression: + +```toml +# Only PASS if the release-workflow file AND a CHANGELOG both exist +expr = 'file_exists(".github/workflows/release.yml") && (file_exists("CHANGELOG.md") || file_exists("CHANGES.md"))' +``` + +Returns `false` (never raises) if the repo path is unavailable to the +evaluator. + +### `json_path(obj: any, path: string) -> any` + +Evaluates a [JMESPath](https://jmespath.org/) expression against `obj`. +Returns the extracted value or `null` on any failure (missing key, +type mismatch, invalid JMESPath). + +```toml +[[controls."OSPS-XX-YY".passes]] +handler = "exec" +command = ["gh", "api", "/repos/$OWNER/$REPO/branches/main/protection"] +output_format = "json" +expr = 'json_path(output.json, "required_pull_request_reviews.required_approving_review_count") >= 1' +``` + +## Common patterns + +- **Exit-code + stdout combined:** `output.exit_code == 0 && output.stdout != ""` +- **JSON field with guard:** `has(output.json.foo) && output.json.foo == "expected"` +- **Negated grep (exec):** `!output.stdout.contains("suspicious-string")` +- **Prefix / suffix:** `output.stdout.startsWith("https://")` / + `output.stdout.endsWith(".pem")` +- **Substring:** `output.stdout.contains("bazel")` +- **File-existence as a guard:** `file_exists("Dockerfile") && output.exit_code == 0` +- **JMESPath extraction:** `json_path(output.json, "runs[0].tool.driver.version")` +- **Project-scoped when clause:** `project.language == "python"` (put this on a `when` field, not `expr`, if you want the whole pass to be skipped rather than resolved INCONCLUSIVE) + +## Getting a CEL error + +If your expression fails to evaluate (unknown identifier, type +mismatch, missing custom function), the orchestrator logs a warning +and preserves the handler's original verdict. It does NOT flip PASS to +FAIL. Failing CEL is an authoring bug the log surfaces; check the +darnit audit log at `--log-level=DEBUG` if a pass isn't behaving as +you expect. + +## Not covered here + +- **Sandboxing / timeouts:** CEL runs under a 1-second default timeout + ([`DEFAULT_TIMEOUT_SECONDS`](../packages/darnit/src/darnit/sieve/cel_evaluator.py)). + Bump on the `CELEvaluator` init; TOML has no per-pass override yet. +- **New context variables:** adding a variable is a `CELContext` change, + not a docs change; see the spec-implementation sync gate. diff --git a/docs/HANDLER_AUTHORING.md b/docs/HANDLER_AUTHORING.md index cc6f82ac..04297291 100644 --- a/docs/HANDLER_AUTHORING.md +++ b/docs/HANDLER_AUTHORING.md @@ -159,25 +159,24 @@ orchestrator applies `expr` after a handler returns `PASS` or `FAIL`. - CEL `false` turns the result into `INCONCLUSIVE`, so later passes can still run. - CEL evaluation errors fall back to the handler's original verdict. -Common context values available in CEL: - -- `output.stdout` -- `output.stderr` -- `output.exit_code` -- `output.json` -- `response.status_code` -- `response.body` -- `response.headers` -- `files` -- `matches` -- `project` -- `context` -- `repo` +For the full per-handler variable reference (`exec`, `file_exists`, +`regex`, `mcp`), the two custom functions (`file_exists`, `json_path`), +and copy-pasteable patterns, see [`docs/CEL_CONTEXT.md`](./CEL_CONTEXT.md). + +Quick summary of the most common variables: + +- `output.stdout`, `output.stderr`, `output.exit_code` -- populated by `exec` +- `output.json` -- populated by `exec` only when `output_format = "json"` +- `output.relative_path`, `output.found_file` -- populated by `file_exists` +- `output.files_found`, `output.found_files` -- populated by `regex` / `pattern` +- `result.*` -- top-level binding for `handler = "mcp"` (JSON of the tool result) +- `project.*` -- from `.project/project.yaml` +- `repo.*`, `context.*` -- ambient audit + user-collected context Custom helper functions available to CEL: -- `file_exists("PATH")` -- `json_path(OBJECT, "JMESPATH_EXPRESSION")` +- `file_exists("PATH")` -- returns `bool`, false if repo path is unavailable +- `json_path(OBJECT, "JMESPATH_EXPRESSION")` -- JMESPath extractor Examples: diff --git a/docs/architecture/framework-design.md b/docs/architecture/framework-design.md index cdca502e..a59f96ad 100644 --- a/docs/architecture/framework-design.md +++ b/docs/architecture/framework-design.md @@ -255,7 +255,7 @@ env = { "TOOL_VERBOSE" = "true" } | `fail_if_output_matches` | `str` | Regex pattern - if matches stdout → FAIL | | `pass_if_json_path` | `str` | JSONPath to extract value | | `pass_if_json_value` | `str` | Expected value at JSON path for PASS | -| `expr` | `str` | CEL expression for pass logic (see Section 3.7) | +| `expr` | `str` | CEL expression for pass logic (see Section 3.7; full context/function reference in [docs/CEL_CONTEXT.md](../CEL_CONTEXT.md)) | | `timeout` | `int` | Timeout in seconds (default: 300) | | `env` | `dict` | Additional environment variables | From b022943eefea6f51b070704334c5ba83a3db6a21 Mon Sep 17 00:00:00 2001 From: Michael Lieberman Date: Fri, 28 Aug 2026 13:34:37 -0400 Subject: [PATCH 2/2] docs(cel): correct binding scope + regex evidence shape (Marc-cn review) Two substantive fixes from PR #401 review: 1. project.*/repo.*/context.* are NOT bound in the `expr` path. The post-step evaluator at orchestrator.py:130 builds its context as literally `{"output": handler_result.evidence or {}}`. The mcp handler similarly binds `{"result": raw_response}`. Neither path constructs a CELContext, so referring to `project.language` etc. in `expr` fails with `undeclared reference to 'project'`, and `_apply_cel_expr` swallows the failure to a warning log -- silently preserving the handler's original verdict. Restructures the doc: adds a "What is NOT bound in expr" section with an explicit call-out; moves the old (misleading) "regardless of handler" list into a "would be available if a caller passed a full CELContext" section labelled as aspirational; recommends `when` on the control instead of `expr` for project-scoped skips. 2. regex handler evidence shape was conflated across three code paths. Splits it into three tables: - Standard match path (`_regex_match_files`): `output.files_checked` (int count), `output.patterns_checked`, `output.any_match`, `output.results` (list of per-file records) - Exclude-globs path (`_regex_exclude_evidence`): `output.exclude_globs`, `output.files_found`, `output.found_files` - No-files path (`_regex_no_files_result`): `output.files_checked` (list, not int) The `files_checked` type flip between int and list is called out as a `has()`-or-type-check risk. Also updates the HANDLER_AUTHORING quick-summary to match. --- docs/CEL_CONTEXT.md | 100 ++++++++++++++++++++++++++++---------- docs/HANDLER_AUTHORING.md | 6 +-- 2 files changed, 77 insertions(+), 29 deletions(-) diff --git a/docs/CEL_CONTEXT.md b/docs/CEL_CONTEXT.md index b874e693..c27b9e52 100644 --- a/docs/CEL_CONTEXT.md +++ b/docs/CEL_CONTEXT.md @@ -73,16 +73,40 @@ which file matched, e.g. `expr = 'output.relative_path == "SECURITY.md"'`. ### `regex` handler (`pattern` alias) -| Variable | Type | Description | -|-------------------------|----------------------------|------------------------------------------------------------| -| `output.files_found` | int | Number of files matched | -| `output.found_files` | list | Relative paths of files that matched at least one pattern | -| `output.files_checked` | list | Files the handler scanned | +The `regex` handler has three internal code paths, each with a +different evidence shape. Which one fires depends on config: + +**Standard match path** -- the default, used when `files` resolve to +existing content and `pattern`/`patterns` is set: + +| Variable | Type | Description | +|-----------------------------|-------------------------|----------------------------------------------------------------------| +| `output.files_checked` | int | Count (NOT list) of files scanned | +| `output.patterns_checked` | list | Names of the patterns evaluated | +| `output.any_match` | bool | True if any pattern matched in any file | +| `output.results` | list | Up to 20 per-(file, pattern) records: `file`, `pattern_name`, `pattern`, `match_count`, `matched`, `matches_preview` | + +**Exclude-globs path** -- used when the pass sets `exclude_globs` +instead of `pattern`s to test: + +| Variable | Type | Description | +|------------------------|-----------------|--------------------------------------------------------------------| +| `output.exclude_globs` | list | The globs the pass declared | +| `output.files_found` | int | Number of files that matched an exclude glob | +| `output.found_files` | list | Up to 10 relative paths of the matched files | + +**No-files path** -- when the pass's `files` list resolves to zero +matches on disk. Returns INCONCLUSIVE and only exposes: + +| Variable | Type | Description | +|-------------------------|---------------|---------------------------------------------------| +| `output.files_checked` | list | The candidate list that produced no matches | The `matches` field the CLAUDE.md notes referenced is not present in -the current handler's evidence -- match structure lives inside the -handler's confidence + evidence shaping. If you need per-file match -detail in `expr`, add it to the handler's evidence first. +the current handler's evidence -- per-file match detail lives inside +`output.results[]` on the match path. `output.files_checked` is an +`int` on the match path and a `list[string]` on the no-files path; +guard with `has()` or a type check before deep access. ### `mcp` handler @@ -99,7 +123,8 @@ expr = 'result.score >= 7.0' ``` There is no `output.*` binding for `handler = "mcp"` passes -- only -`result.*` and any project/repo bindings below. +`result.*`. `project.*`/`repo.*`/`context.*` are also NOT available +here (see the next section for why). ### Handlers that do NOT evaluate `expr` @@ -108,13 +133,44 @@ There is no `output.*` binding for `handler = "mcp"` passes -- only their evidence. Writing an `expr` on their pass config is silently ignored today; use the handler's own config keys to shape the verdict. -## Bindings available regardless of handler +## What is NOT bound in `expr` The [`CELContext`](../packages/darnit/src/darnit/sieve/cel_evaluator.py) -dataclass carries additional bindings the runtime can inject when the -orchestrator constructs a full context (not the trimmed post-step -context most passes see). Ambient bindings that a control author may -reference: +dataclass declares fields for ambient bindings (`project`, `repo`, +`context`, `files`, `matches`, `response`) that a full-context CEL +call would receive. **None of these are populated on the `expr` path +used by controls today.** + +The post-step `expr` evaluator at +[`orchestrator.py:130`](../packages/darnit/src/darnit/sieve/orchestrator.py) +builds its context as literally +`{"output": handler_result.evidence or {}}` -- only `output` is bound. +The `mcp` handler's in-handler CEL similarly binds only +`{"result": raw_response}` +([`_eval_cel_over_result`](../packages/darnit/src/darnit/sieve/builtin_handlers.py)). +Neither path constructs a `CELContext`, so referring to `project.*` or +`repo.*` in an `expr` fails with `undeclared reference to 'project'`. +CEL failure logs a warning and preserves the handler's original +verdict ([`_apply_cel_expr`](../packages/darnit/src/darnit/sieve/orchestrator.py)), +so a broken reference produces a silent no-op rather than a visible +error -- worth flagging in your control tests. + +If you need a project-context-scoped skip, put the check on the +control's `when` field instead of the pass's `expr`: + +```toml +[controls."OSPS-XX-YY"] +when = { language = "python" } +``` + +`when` runs against the audit's full project context before the +pass loop starts; `expr` runs on the trimmed post-step context and +does not. + +## Bindings that WOULD be available if a caller passes a full CELContext + +For completeness -- these are wired in the code but no +production caller (post-step, mcp) uses them today: | Variable | Type | Populated when | |-------------|---------------------------|-----------------------------------------------------------------| @@ -122,17 +178,9 @@ reference: | `repo.*` | dict (path, owner, name) | Set by the audit driver on every audit | | `context.*` | dict | Values the user answered via `darnit collect-context` / harness | -Typical use: - -```toml -# Only apply this pass when project language is Python -{ handler = "exec", command = ["python", "-c", "print('ok')"], expr = 'project.language == "python"' } -``` - -`project.*` is populated from the `.project/` reader -([`dot_project.py`](../packages/darnit/src/darnit/context/dot_project.py)) -plus any control-side auto-detected values (`language`, -`ci_provider`, `platform`). +If these ever get wired into the post-step path, existing controls +that reference them today (there are currently none in the shipped +framework) would start seeing them. Track that change here. ## Custom CEL functions @@ -176,7 +224,7 @@ expr = 'json_path(output.json, "required_pull_request_reviews.required_approving - **Substring:** `output.stdout.contains("bazel")` - **File-existence as a guard:** `file_exists("Dockerfile") && output.exit_code == 0` - **JMESPath extraction:** `json_path(output.json, "runs[0].tool.driver.version")` -- **Project-scoped when clause:** `project.language == "python"` (put this on a `when` field, not `expr`, if you want the whole pass to be skipped rather than resolved INCONCLUSIVE) +- **Project-scoped skip:** put `project.language == "python"` on the control's `when` field, not on a pass's `expr` -- `expr` cannot see `project.*` in the post-step path (see "What is NOT bound in `expr`" above). ## Getting a CEL error diff --git a/docs/HANDLER_AUTHORING.md b/docs/HANDLER_AUTHORING.md index 04297291..8eb797ad 100644 --- a/docs/HANDLER_AUTHORING.md +++ b/docs/HANDLER_AUTHORING.md @@ -168,10 +168,10 @@ Quick summary of the most common variables: - `output.stdout`, `output.stderr`, `output.exit_code` -- populated by `exec` - `output.json` -- populated by `exec` only when `output_format = "json"` - `output.relative_path`, `output.found_file` -- populated by `file_exists` -- `output.files_found`, `output.found_files` -- populated by `regex` / `pattern` +- `output.files_checked` (int), `output.patterns_checked`, `output.any_match`, `output.results` -- populated by the `regex` / `pattern` match path +- `output.exclude_globs`, `output.files_found`, `output.found_files` -- populated by the `regex` handler's `exclude_globs` path - `result.*` -- top-level binding for `handler = "mcp"` (JSON of the tool result) -- `project.*` -- from `.project/project.yaml` -- `repo.*`, `context.*` -- ambient audit + user-collected context +- `project.*`, `repo.*`, `context.*` are **NOT bound** in the `expr` path most controls use; put project-scoped skips on the control's `when` field instead. See [`CEL_CONTEXT.md`](./CEL_CONTEXT.md) for details. Custom helper functions available to CEL: