feat(sieve): wire regex evidence to llm_eval file_contents (issue #402 Option 2) - #408
Open
mlieberman85 wants to merge 1 commit into
Open
Conversation
…nitdevorg#402 Option 2) Follow-up to darnitdevorg#406 and darnitdevorg#407. Where those PRs fixed the empty-file_contents bug in TOML on both the `pattern` files list and the `llm_eval` files_to_include list, this changes the framework so the two lists don't need to duplicate each other going forward. ## Change `regex_handler` (`_regex_match_files`) now emits `resolved_files: list[str]` in evidence -- the absolute paths of every candidate that actually existed on disk and was scanned. `_resolve_regex_files` was already computing that list; we just surface it. `llm_eval_handler` gains two connected pieces: 1. A new sentinel `"$RESOLVED_FILES"` in `files_to_include`. Expands to every path in `gathered_evidence["resolved_files"]`. 2. An automatic fallback: when `files_to_include` produces no file contents at all (e.g. `["$FOUND_FILE"]` with no preceding `file_exists` PASS -- the exact shape from darnitdevorg#402), the handler reads `gathered_evidence["resolved_files"]` on its own. Existing TOMLs with lone `["$FOUND_FILE"]` (before darnitdevorg#406 landed) start working correctly with no config change. Both branches share the same `_read` helper and honor the 5-file cap and 10000-byte truncation the handler already had. ## Why this shape Preserves single-source-of-truth for the file list: the sibling `pattern`/`regex` pass already declares `files = [...]`; `llm_eval` doesn't need to enumerate them again. The follow-up work of teaching `pattern` to also accept `.rst`/`.txt` variants (darnitdevorg#407) automatically benefits `llm_eval` too, since `resolved_files` grows. ## Tests 5 new llm_eval tests + 1 new regex_handler test: - `test_resolved_files_sentinel_reads_all_evidence_paths`: `$RESOLVED_FILES` fans out to every entry. - `test_resolved_files_fallback_when_files_to_include_yields_empty`: the core darnitdevorg#402 reproducer -- `["$FOUND_FILE"]` with no `found_file` now reads the pattern-resolved list. - `test_no_fallback_when_files_to_include_already_produced_content`: explicit paths win; fallback only fires when file_contents is empty. - `test_fallback_respects_five_file_cap`: cap holds through the fallback. - `test_no_fallback_when_gathered_evidence_has_no_resolved_files`: absent `resolved_files` is a no-op. - `test_evidence_includes_resolved_files_on_match_path`: `resolved_files` lists paths that existed on disk; missing entries are excluded. ## Relationship to darnitdevorg#406 and darnitdevorg#407 Complementary: - **darnitdevorg#406**: enumerated real paths in `llm_eval` blocks (TOML-only, fixes what the LLM sees when it fires). - **darnitdevorg#407**: widened `pattern` file lists to include .rst / .txt / no-ext README variants (TOML-only, fixes how often the LLM fires). - **This PR**: makes the framework carry the file list forward automatically so future TOMLs don't need to duplicate it. All three can merge independently; together they close the whole empty-consultation class of bug and reduce the maintenance burden on future control authors.
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment Thanks for integrating Codecov - We've got you covered ☂️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Framework-level follow-up to #406 and #407 (both TOML-only). #402 Option 2. Where those PRs fixed the empty-
file_contentsbug in TOML on both sides -- enumerating real paths insidellm_evalblocks (#406) and widening the siblingpattern.fileslists (#407) -- this PR changes the framework so the two lists don't need to duplicate each other going forward.Change
regex_handler(_regex_match_files) now emitsresolved_files: list[str]in evidence: the absolute paths of every candidate that actually existed on disk and was scanned._resolve_regex_filesalready computed that list internally; this exposes it.llm_eval_handlergains two connected pieces:New sentinel
\"$RESOLVED_FILES\"infiles_to_include-- fans out to every path ingathered_evidence[\"resolved_files\"]. Explicit opt-in for controls that want the behavior.Automatic fallback when
files_to_includeproduces nofile_contents(empty\$FOUND_FILE, missing literal paths, no\$RESOLVED_FILESsentinel). Existing TOMLs with lone[\"\$FOUND_FILE\"](i.e. the pre-fix(baseline): enumerate real candidate paths in llm_eval passes #406 shape) start working correctly with no config change -- the pattern's file list is threaded forward.Both branches share the same
_readhelper and honor the 5-file cap + 10000-byte truncation the handler already had.Why this shape
Preserves single-source-of-truth for the file list: the sibling
pattern/regexpass already declaresfiles = [...];llm_evaldoesn't need to enumerate them again. #407's follow-up work of wideningpatternto accept.rst/.txtvariants automatically benefitsllm_evaltoo, sinceresolved_filesgrows.Test plan
6 new tests:
test_resolved_files_sentinel_reads_all_evidence_paths:\$RESOLVED_FILESfans out.test_resolved_files_fallback_when_files_to_include_yields_empty: the exact llm_eval receives empty file_contents when preceded by pattern/regex (no file_exists sibling) #402 reproducer.[\"\$FOUND_FILE\"]with nofound_filenow ships real content.test_no_fallback_when_files_to_include_already_produced_content: explicit paths win; fallback fires only when empty.test_fallback_respects_five_file_cap: cap holds through the fallback path.test_no_fallback_when_gathered_evidence_has_no_resolved_files: absent evidence is a no-op.test_evidence_includes_resolved_files_on_match_path:resolved_fileslists paths that existed on disk; missing entries excluded.pytest tests/darnit/sieve/test_builtin_handlers.py -v-> 87 pass.Full workspace sweep:
pytest tests/ -q-> 3008 pass, 26 skip, 0 fail.ruff check .clean (repo-wide before push).Relationship to #406 and #407
Three complementary fixes for the same bug class:
All three can merge independently. Together they close the whole empty-consultation class of bug and reduce future control-author burden.
Not in scope
\$FOUND_FILEin favor of\$RESOLVED_FILES(both are useful; the former guarantees one specific file).Closes #402.