Skip to content

docs: CEL context reference for control authors - #401

Merged
Marc-cn merged 2 commits into
darnitdevorg:mainfrom
mlieberman85:docs-357-cel-context-vars
Aug 28, 2026
Merged

docs: CEL context reference for control authors#401
Marc-cn merged 2 commits into
darnitdevorg:mainfrom
mlieberman85:docs-357-cel-context-vars

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Summary

Closes #357. Adds docs/CEL_CONTEXT.md with a per-handler breakdown of what a control pass's expr can reference. Sourced from the actual handler evidence dicts and CELContext, not from stale notes in CLAUDE.md that were aimed at a different audience.

Why this doc exists

A control author writing expr = 'output.json.foo' gets no discoverable reference for what output.* contains -- they either grep other TOMLs or guess. Guessing produces silently-wrong CEL (e.g., referencing output.json on a pass that never set output_format = "json", or reaching for matches on a regex pass that no longer exposes it).

Content sourced from code, not CLAUDE.md

The old CLAUDE.md notes claimed:

  • pattern: files, matches -- but matches is not in the current regex_handler's evidence.
  • api: response.status_code, response.body, response.headers -- but there is no verification api handler; api_call is a remediation handler with no expr evaluator.

The new doc covers what the current implementation actually exposes and calls out which handlers do NOT evaluate expr today so authors don't silently write a no-op.

What landed

  • docs/CEL_CONTEXT.md (new, +212 lines): per-handler variable tables (exec, file_exists, regex/pattern, mcp), ambient bindings (project, repo, context), the two custom functions with signatures + failure modes, common patterns, and an error-behavior note.
  • docs/HANDLER_AUTHORING.md: replaced the old inline variable enumeration with an accurate one-line summary + a link to the new reference.
  • docs/architecture/framework-design.md: added a link to the reference on the expr field row.

Non-goals

  • Not modifying any handler code -- pure docs.
  • Not proposing new context variables. Adding one is a CELContext change (spec-implementation sync gate applies) and belongs in a separate PR.
  • No mention of a hypothetical api verification handler or a matches array on regex; if those land later, add them here.

Closes #357.

@mlieberman85
mlieberman85 force-pushed the docs-357-cel-context-vars branch from a3b87cf to 1b318c5 Compare August 28, 2026 01:35
@Marc-cn

Marc-cn commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Checked the tables against the handler evidence dicts, exec, file_exists and mcp all match. Two things before merge:

  1. "Bindings available regardless of handler" doesn't hold on the post-step path. orchestrator.py:130 builds cel_context = {"output": handler_result.evidence or {}} only output is bound, and nothing constructs a CELContext for_apply_cel_expr. Running the doc's own example against that context:

'project.language == "python"' success=False undeclared reference to 'project'
'repo.owner == "x"' success=False undeclared reference to 'repo'
'output.exit_code == 0' success=True value=True

And per _apply_cel_expr the failure just logs a warning and preserves the handler verdict, so an author gets no signal. Worth scoping that section to "not available in post-step expr" or dropping it.

  1. files_found / found_files come from the exclude-globs branch (builtin_handlers.py:400-404), not the regex/pattern match path the table describes. A regex pass that matched files exposes files_checked; a pass thatresolved nothing exposes files_checked only (line 502)

rest lgtm

)

Adds `docs/CEL_CONTEXT.md` with a per-handler breakdown of the
variables CEL sees in a control pass's `expr` field. Sourced from the
actual handler evidence + CELContext dataclass -- not from CLAUDE.md's
audience-mismatched notes.

Covers what the current implementation actually exposes:
- `exec`: `output.stdout`, `output.stderr`, `output.exit_code`,
  `output.command`, `output.json` (only when `output_format = "json"`)
- `file_exists`: `output.found_file`, `output.relative_path`,
  `output.files_checked`
- `regex` / `pattern`: `output.files_found`, `output.found_files`,
  `output.files_checked`
- `mcp`: `result.*` (evaluated by `_eval_cel_over_result` before the
  handler returns, not the standard post-step path)
- Ambient bindings: `project.*`, `repo.*`, `context.*`
- Custom functions: `file_exists(path)`, `json_path(obj, jmespath)`

Explicitly calls out which handlers do NOT evaluate `expr` today
(`llm_eval`, `llm_extract`, `manual_steps`, `file_create`, `api_call`,
`project_update`, `yaml_inject`) so authors do not silently write a
no-op.

Cross-linked from `docs/architecture/framework-design.md` (the exec
handler's `expr` row) and `docs/HANDLER_AUTHORING.md` (the CEL basics
section), with the old inline stale variable list in HANDLER_AUTHORING
replaced by a short summary pointing at the reference.
Two substantive fixes from PR darnitdevorg#401 review:

1. project.*/repo.*/context.* are NOT bound in the `expr` path. The
   post-step evaluator at orchestrator.py:130 builds its context as
   literally `{"output": handler_result.evidence or {}}`. The mcp
   handler similarly binds `{"result": raw_response}`. Neither path
   constructs a CELContext, so referring to `project.language` etc.
   in `expr` fails with `undeclared reference to 'project'`, and
   `_apply_cel_expr` swallows the failure to a warning log --
   silently preserving the handler's original verdict.

   Restructures the doc: adds a "What is NOT bound in expr" section
   with an explicit call-out; moves the old (misleading) "regardless
   of handler" list into a "would be available if a caller passed a
   full CELContext" section labelled as aspirational; recommends
   `when` on the control instead of `expr` for project-scoped skips.

2. regex handler evidence shape was conflated across three code paths.
   Splits it into three tables:
   - Standard match path (`_regex_match_files`):
     `output.files_checked` (int count), `output.patterns_checked`,
     `output.any_match`, `output.results` (list of per-file records)
   - Exclude-globs path (`_regex_exclude_evidence`):
     `output.exclude_globs`, `output.files_found`, `output.found_files`
   - No-files path (`_regex_no_files_result`):
     `output.files_checked` (list, not int)

   The `files_checked` type flip between int and list is called out
   as a `has()`-or-type-check risk.

Also updates the HANDLER_AUTHORING quick-summary to match.
@mlieberman85
mlieberman85 force-pushed the docs-357-cel-context-vars branch from 1b318c5 to b022943 Compare August 28, 2026 17:34
@Marc-cn
Marc-cn merged commit 5f7ff7e into darnitdevorg:main Aug 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: surface CEL context variables and helpers for control authors

2 participants