fix(baseline): enumerate real candidate paths in llm_eval passes - #406
Open
mlieberman85 wants to merge 2 commits into
Open
fix(baseline): enumerate real candidate paths in llm_eval passes#406mlieberman85 wants to merge 2 commits into
mlieberman85 wants to merge 2 commits into
Conversation
darnitdevorg#402) Every llm_eval pass in openssf-baseline.toml previously declared `files_to_include = ["$FOUND_FILE"]`. `$FOUND_FILE` is populated only by `file_exists_handler` on PASS; the `pattern`/`regex` handler never sets `gathered_evidence["found_file"]`. When the deterministic tier resolved INCONCLUSIVE (candidate `files` list produced no on-disk matches -- e.g. a .md-only pattern list on an rst-based repo like tqdm), the pipeline continued to `llm_eval` and shipped a consultation with `file_contents = {}`. The LLM reasoned about a list of file paths and dependency statuses but never saw actual file content. Symptom (surveyed across 29 repos): 9 Sonnet calls, 9 non-answers at 0.20-0.30 confidence, one per PENDING_LLM. tqdm produced 4 (README.rst only, no .md files in the pattern lists); renaming flask's README.md to README.rst took it from 0 to 5 PENDING controls. Fix: enumerate real candidate paths per issue darnitdevorg#402 option 1 (TOML-only, no framework change). Each `files_to_include` list now mirrors the sibling `pattern` handler's `files` list (for the buggy pattern -> llm_eval controls) or the locator's `discover` list (for the file_exists -> ... -> llm_eval controls that were unreachable but still misleading). Adds .rst variants where the deterministic tier only listed .md files (DO-01.01, DO-03.02, DO-04.01, DO-05.01) so llm_eval can see rst content directly. The llm_eval handler caps at 5 files (builtin_handlers.py:628); every list stays within the cap. Regression guard: tests/darnit_baseline/controls/test_llm_eval_file_contents.py - test_no_pass_ships_lone_found_file: parses the TOML and asserts no llm_eval pass has files_to_include == ["$FOUND_FILE"]. - test_every_llm_eval_declares_files_to_include: files_to_include is mandatory for llm_eval. - test_files_to_include_within_handler_cap: warns if any pass exceeds the handler's 5-entry cap. Not in scope (follow-up): - Framework code changes to remove the 5-entry cap or generalize the handler's file resolution (option 2 or option 3 from the issue). - Widening the sibling `pattern` handler's file lists to accept .rst / .txt / no-extension README variants deterministically. That would make the pattern tier PASS/FAIL conclusively on more repos and reduce how often llm_eval is reached at all, but it changes the pattern handler's verdict distribution and belongs in a separate PR.
This was referenced Aug 29, 2026
Collaborator
|
Tested on tqdm (the repo from the original survey) by driving the sieve directly and inspecting the consultation payload. Partial fix- 2 of the 4 controls now carry content: SA-01.01 and SA-02.01 mirror the sibling pattern list without .rst variants, so on a repo with README.rst none of the five candidates exist and the consultation still ships empty. Same for the SA-02.01 list, which is .md/openapi only. Two suggestions:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #402. TOML-only fix (Option 1 from the issue). Every
llm_evalpass inopenssf-baseline.tomlpreviously declaredfiles_to_include = ["\$FOUND_FILE"].\$FOUND_FILEis populated only byfile_exists_handleron PASS; thepattern/regexhandler never setsgathered_evidence["found_file"]. When the deterministic tier resolved INCONCLUSIVE,llm_evalfired withfile_contents = {}and the LLM had nothing to look at.Survey across 29 repos surfaced 9 non-answers at 0.20-0.30 confidence, one per PENDING_LLM. tqdm produced 4 (README.rst only, no .md files in the pattern lists); renaming flask's README.md to README.rst took it from 0 to 5 PENDING controls.
Changes
llm_evalblocks:files_to_include = ["\$FOUND_FILE"]-> enumerated real candidate paths per control:discoverlist.patternhandler'sfileslist.builtin_handlers.py:628).tests/darnit_baseline/controls/test_llm_eval_file_contents.py(new, 3 tests):test_no_pass_ships_lone_found_file: parses TOML, fails if anyllm_evalpass ships withfiles_to_include == ["\$FOUND_FILE"].test_every_llm_eval_declares_files_to_include: mandatory field guard.test_files_to_include_within_handler_cap: warns on over-cap declarations.Test plan
pytest tests/darnit_baseline/controls/test_llm_eval_file_contents.py -v-> 3/3 pass.python scripts/validate_sync.py --verbose-> TOML schema valid, 66 controls, 7 handlers in sync.pytest tests/ -q-> 3005 pass, 26 skip, 0 fail.Not in scope (follow-ups)
patternhandler's file lists to accept .rst / .txt / no-extension README variants deterministically. That would make the pattern tier PASS/FAIL conclusively on more repos and reduce how oftenllm_evalis reached at all -- but it changes the pattern handler's verdict distribution and belongs in a separate PR.Closes #402.