Skip to content

fix(baseline): enumerate real candidate paths in llm_eval passes - #406

Open
mlieberman85 wants to merge 2 commits into
darnitdevorg:mainfrom
mlieberman85:fix-402-llm-eval-file-contents
Open

fix(baseline): enumerate real candidate paths in llm_eval passes#406
mlieberman85 wants to merge 2 commits into
darnitdevorg:mainfrom
mlieberman85:fix-402-llm-eval-file-contents

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Summary

Closes #402. TOML-only fix (Option 1 from the issue). Every llm_eval pass in openssf-baseline.toml previously declared files_to_include = ["\$FOUND_FILE"]. \$FOUND_FILE is populated only by file_exists_handler on PASS; the pattern/regex handler never sets gathered_evidence["found_file"]. When the deterministic tier resolved INCONCLUSIVE, llm_eval fired with file_contents = {} and the LLM had nothing to look at.

Survey across 29 repos surfaced 9 non-answers at 0.20-0.30 confidence, one per PENDING_LLM. tqdm produced 4 (README.rst only, no .md files in the pattern lists); renaming flask's README.md to README.rst took it from 0 to 5 PENDING controls.

Changes

  • 9 llm_eval blocks: files_to_include = ["\$FOUND_FILE"] -> enumerated real candidate paths per control:
    • DO-01.01, VM-02.01, GV-01.01, SA-03.02: mirror the locator's discover list.
    • DO-03.02, DO-04.01, DO-05.01, SA-01.01, SA-02.01: mirror the sibling pattern handler's files list.
    • .rst variants added where the deterministic tier only listed .md files (DO-01.01, DO-03.02, DO-04.01, DO-05.01) so llm_eval sees rst content directly.
    • Every list stays within the handler's 5-entry cap (builtin_handlers.py:628).
  • tests/darnit_baseline/controls/test_llm_eval_file_contents.py (new, 3 tests):
    • test_no_pass_ships_lone_found_file: parses TOML, fails if any llm_eval pass ships with files_to_include == ["\$FOUND_FILE"].
    • test_every_llm_eval_declares_files_to_include: mandatory field guard.
    • test_files_to_include_within_handler_cap: warns on over-cap declarations.

Test plan

  • pytest tests/darnit_baseline/controls/test_llm_eval_file_contents.py -v -> 3/3 pass.
  • python scripts/validate_sync.py --verbose -> TOML schema valid, 66 controls, 7 handlers in sync.
  • Full workspace sweep: pytest tests/ -q -> 3005 pass, 26 skip, 0 fail.

Not in scope (follow-ups)

  1. Framework code changes to remove the 5-entry cap or generalize the handler's file resolution (Options 2 or 3 from the issue).
  2. Widening the sibling pattern handler's file lists to accept .rst / .txt / no-extension README variants deterministically. That would make the pattern tier PASS/FAIL conclusively on more repos and reduce how often llm_eval is reached at all -- but it changes the pattern handler's verdict distribution and belongs in a separate PR.

Closes #402.

 darnitdevorg#402)

Every llm_eval pass in openssf-baseline.toml previously declared
`files_to_include = ["$FOUND_FILE"]`. `$FOUND_FILE` is populated
only by `file_exists_handler` on PASS; the `pattern`/`regex` handler
never sets `gathered_evidence["found_file"]`. When the deterministic
tier resolved INCONCLUSIVE (candidate `files` list produced no on-disk
matches -- e.g. a .md-only pattern list on an rst-based repo like
tqdm), the pipeline continued to `llm_eval` and shipped a consultation
with `file_contents = {}`. The LLM reasoned about a list of file paths
and dependency statuses but never saw actual file content.

Symptom (surveyed across 29 repos): 9 Sonnet calls, 9 non-answers at
0.20-0.30 confidence, one per PENDING_LLM. tqdm produced 4 (README.rst
only, no .md files in the pattern lists); renaming flask's README.md
to README.rst took it from 0 to 5 PENDING controls.

Fix: enumerate real candidate paths per issue darnitdevorg#402 option 1 (TOML-only,
no framework change). Each `files_to_include` list now mirrors the
sibling `pattern` handler's `files` list (for the buggy pattern ->
llm_eval controls) or the locator's `discover` list (for the file_exists
-> ... -> llm_eval controls that were unreachable but still misleading).

Adds .rst variants where the deterministic tier only listed .md files
(DO-01.01, DO-03.02, DO-04.01, DO-05.01) so llm_eval can see rst
content directly. The llm_eval handler caps at 5 files
(builtin_handlers.py:628); every list stays within the cap.

Regression guard: tests/darnit_baseline/controls/test_llm_eval_file_contents.py
- test_no_pass_ships_lone_found_file: parses the TOML and asserts no
  llm_eval pass has files_to_include == ["$FOUND_FILE"].
- test_every_llm_eval_declares_files_to_include: files_to_include is
  mandatory for llm_eval.
- test_files_to_include_within_handler_cap: warns if any pass exceeds
  the handler's 5-entry cap.

Not in scope (follow-up):
- Framework code changes to remove the 5-entry cap or generalize the
  handler's file resolution (option 2 or option 3 from the issue).
- Widening the sibling `pattern` handler's file lists to accept .rst /
  .txt / no-extension README variants deterministically. That would
  make the pattern tier PASS/FAIL conclusively on more repos and
  reduce how often llm_eval is reached at all, but it changes the
  pattern handler's verdict distribution and belongs in a separate PR.
@Marc-cn

Marc-cn commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Tested on tqdm (the repo from the original survey) by driving the sieve directly and inspecting the consultation payload. Partial fix- 2 of the 4 controls now carry content:

OSPS-DO-04.01   file_contents: {README.rst: 10000}
OSPS-DO-05.01   file_contents: {README.rst: 10000, CONTRIBUTING.md: 10000}
OSPS-SA-01.01   file_contents: {}
OSPS-SA-02.01   file_contents: {}

SA-01.01 and SA-02.01 mirror the sibling pattern list without .rst variants, so on a repo with README.rst none of the five candidates exist and the consultation still ships empty. Same for the SA-02.01 list, which is .md/openapi only.

Two suggestions:

  • Add .rst variants to SA-01.01 and SA-02.01 too (ARCHITECTURE.rst, docs/architecture.rst, README.rst, docs/api.rst), same as you did for DO-01.01/03.02/04.01/05.01. That would close the tqdm case entirely.
  • test_no_pass_ships_lone_found_file only asserts the literal ["$FOUND_FILE"] is gone; it would still pass on a list that resolves to nothing. Worth a test that runs a control against a fixture repo and asserts file_contents is non-empty.
    LGTM: full suite unchanged on my box, and the DO-* improvement is real.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

llm_eval receives empty file_contents when preceded by pattern/regex (no file_exists sibling)

2 participants