llm_eval receives empty file_contents when preceded by pattern/regex (no file_exists sibling)

[mlieberman85]

Symptom

On repos where a control's deterministic tier resolves INCONCLUSIVE rather than PASS/FAIL, the follow-on llm_eval pass fires with file_contents = {} in the consultation request. Every downstream LLM call reasons about a list of file paths plus dependency statuses but no actual file content, and returns inconclusive at 0.20-0.30 confidence. In a survey of 29 repos, only tqdm produced any PENDING_LLM at all (4 controls); renaming flask's README.md -> README.rst moved it from 0 to 5 PENDING controls, all with empty consultations.

9 Sonnet calls, 9 non-answers, all because the LLM tier fires precisely when the file it would have read isn't reachable.

Root cause

packages/darnit/src/darnit/sieve/builtin_handlers.py:626-640 -- llm_eval_handler resolves only one dynamic token:

files_to_include = config.get("files_to_include", [])
file_contents: dict[str, str] = {}
for f in files_to_include[:5]:
    resolved = f
    if f == "$FOUND_FILE":
        resolved = context.gathered_evidence.get("found_file", "")
    if not resolved:
        continue

gathered_evidence["found_file"] is set only by file_exists_handler on PASS (lines 143, 160, 173). The regex/pattern handler evidence carries files_checked, found_files, files_found, matched etc. -- but never found_file.

9 out of 9 llm_eval passes in packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml use files_to_include = ["$FOUND_FILE"].

The pipeline that reaches llm_eval with empty file_contents is:

pattern -> llm_eval    (no file_exists sibling; $FOUND_FILE never set)

This is the shape of DO-03.02, DO-04.01, DO-05.01, VM-02.01, GV-01.01, SA-01.01, SA-02.01 -- the exact controls that fire on the tqdm/flask-with-rst repros.

Why it fires precisely when there's nothing to show

_regex_no_files_result (builtin_handlers.py:485-508) returns INCONCLUSIVE with evidence={"files_checked": files_list} when the candidate list produces zero on-disk matches. INCONCLUSIVE -> ATTACH_EVIDENCE_AND_CONTINUE -> llm_eval fires -> $FOUND_FILE empty -> consultation ships with file_contents: {}.

flask README.md vs README.rst is the same mechanism: with README.md present, DO-03.02's files = ["SECURITY.md", "README.md", "docs/releases.md", "CONTRIBUTING.md", "docs/RELEASE-VERIFICATION.md"] matches something -> pattern PASSes/FAILs conclusively -> orchestrator stops. With README.rst only, no candidate matches -> INCONCLUSIVE -> llm_eval fires -> empty consultation.

Fix options, ordered by scope

1. TOML-only: change every llm_eval block's files_to_include from ["$FOUND_FILE"] to enumerate real candidate paths (superset of the sibling pattern's files). The handler already skips missing files via except OSError, so listing candidates that may not exist is safe. Zero framework code change; downside is TOML duplicates the pattern's file list.

2. Wire regex evidence to llm_eval: enrich regex_handler evidence with an existing_files key (candidates that actually exist on disk regardless of match state). Teach llm_eval to fall back to that when $FOUND_FILE is empty. One evidence key + ~5 lines of resolution logic. Preserves single-source-of-truth for the file list.

3. use_locator = true on llm_eval: give llm_eval the same locator resolution file_exists has. Consistent declarative surface; largest scope of the three.

Repro

• Any repo without a README.md, SECURITY.md, or the other .md files a DO-* / VM-* / GV-* / SA-* control lists in its pattern.files, but with equivalent .rst/.txt content. tqdm is a real-world example; mv README.md README.rst on any other repo reproduces.
• Run darnit audit (or the MCP audit_openssf_baseline tool) with the harness or any driver that dispatches PENDING_LLM. Inspect the consultation payload -- file_contents will be {}.

Related

• packages/darnit/src/darnit/sieve/builtin_handlers.py: llm_eval_handler, regex_handler, _regex_no_files_result
• packages/darnit/src/darnit/sieve/orchestrator.py: apply_check_execution_rule INCONCLUSIVE branch
• packages/darnit-baseline/src/darnit_baseline/openssf-baseline.toml: 9 [[controls."...".passes]] blocks with handler = "llm_eval"

Surfaced during a survey of 29 repos (10 general + 19 CNCF).

[Marc-cn]

Looked at this. There may be a fourth option. The controls that don't hit this bug (DO-01.01, VM-02.01, GV-01.01, SA-03.02) all share a shape the broken ones lack:

[controls."X".locator]
discover = [...]            # DO-01.01's already covers .rst/.txt

[[controls."X".passes]]
handler = "file_exists"
use_locator = true          # sets found_file

[[controls."X".passes]]
handler = "regex"
file = "$FOUND_FILE"

use_locator copies locator.discover into files at load time (control_loader.py:89), so file_exists sets found_file and llm_eval resolves. DO-01.01 is why flask-with-rst shows 5 PENDING and not 6, its discover list already has README.rst.

The five that break (DO-03.02, DO-04.01, DO-05.01, SA-01.01, SA-02.01) start with pattern over an inline files list, so found_file is never set.

Proposal: give those five a locator with .rst/.adoc/.txt variants and a file_exists + use_locator first pass, TOML only, same machinery the working controls use. Two effects: docs in non-.md formats get found and llm_eval gets real content; and where no doc exists in any format, file_exists FAILs at confidence 1.0 and terminates, so the LLM is never called for a guaranteed non-answer.

The semantic change: those five currently land WARN/PENDING on a repo with no such doc, and would become FAIL. That matches DO-01.01/VM-02.01 today, but flagging it since it moves numbers. What do you think? On which option can I work on?