Skip to content

feat(sieve): wire regex evidence to llm_eval file_contents (issue #402 Option 2) - #408

Open
mlieberman85 wants to merge 1 commit into
darnitdevorg:mainfrom
mlieberman85:fix-402-option-2-regex-to-llm-eval
Open

feat(sieve): wire regex evidence to llm_eval file_contents (issue #402 Option 2)#408
mlieberman85 wants to merge 1 commit into
darnitdevorg:mainfrom
mlieberman85:fix-402-option-2-regex-to-llm-eval

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Summary

Framework-level follow-up to #406 and #407 (both TOML-only). #402 Option 2. Where those PRs fixed the empty-file_contents bug in TOML on both sides -- enumerating real paths inside llm_eval blocks (#406) and widening the sibling pattern.files lists (#407) -- this PR changes the framework so the two lists don't need to duplicate each other going forward.

Change

regex_handler (_regex_match_files) now emits resolved_files: list[str] in evidence: the absolute paths of every candidate that actually existed on disk and was scanned. _resolve_regex_files already computed that list internally; this exposes it.

llm_eval_handler gains two connected pieces:

  1. New sentinel \"$RESOLVED_FILES\" in files_to_include -- fans out to every path in gathered_evidence[\"resolved_files\"]. Explicit opt-in for controls that want the behavior.

  2. Automatic fallback when files_to_include produces no file_contents (empty \$FOUND_FILE, missing literal paths, no \$RESOLVED_FILES sentinel). Existing TOMLs with lone [\"\$FOUND_FILE\"] (i.e. the pre-fix(baseline): enumerate real candidate paths in llm_eval passes #406 shape) start working correctly with no config change -- the pattern's file list is threaded forward.

Both branches share the same _read helper and honor the 5-file cap + 10000-byte truncation the handler already had.

Why this shape

Preserves single-source-of-truth for the file list: the sibling pattern/regex pass already declares files = [...]; llm_eval doesn't need to enumerate them again. #407's follow-up work of widening pattern to accept .rst/.txt variants automatically benefits llm_eval too, since resolved_files grows.

Test plan

6 new tests:

  • test_resolved_files_sentinel_reads_all_evidence_paths: \$RESOLVED_FILES fans out.

  • test_resolved_files_fallback_when_files_to_include_yields_empty: the exact llm_eval receives empty file_contents when preceded by pattern/regex (no file_exists sibling) #402 reproducer. [\"\$FOUND_FILE\"] with no found_file now ships real content.

  • test_no_fallback_when_files_to_include_already_produced_content: explicit paths win; fallback fires only when empty.

  • test_fallback_respects_five_file_cap: cap holds through the fallback path.

  • test_no_fallback_when_gathered_evidence_has_no_resolved_files: absent evidence is a no-op.

  • test_evidence_includes_resolved_files_on_match_path: resolved_files lists paths that existed on disk; missing entries excluded.

  • pytest tests/darnit/sieve/test_builtin_handlers.py -v -> 87 pass.

  • Full workspace sweep: pytest tests/ -q -> 3008 pass, 26 skip, 0 fail.

  • ruff check . clean (repo-wide before push).

Relationship to #406 and #407

Three complementary fixes for the same bug class:

PR Layer What it fixes
#406 TOML What the LLM sees when it fires (enumerated real paths per control)
#407 TOML How often the LLM fires (widened pattern lists to .rst/.txt/no-ext)
This Framework The list is threaded forward automatically -- no duplication needed

All three can merge independently. Together they close the whole empty-consultation class of bug and reduce future control-author burden.

Not in scope

  • Removing the 5-file cap or generalizing to configurable-per-control cap (option 3 from the issue).
  • Deprecating \$FOUND_FILE in favor of \$RESOLVED_FILES (both are useful; the former guarantees one specific file).

Closes #402.

…nitdevorg#402 Option 2)

Follow-up to darnitdevorg#406 and darnitdevorg#407. Where those PRs fixed the empty-file_contents
bug in TOML on both the `pattern` files list and the `llm_eval`
files_to_include list, this changes the framework so the two lists
don't need to duplicate each other going forward.

## Change

`regex_handler` (`_regex_match_files`) now emits `resolved_files:
list[str]` in evidence -- the absolute paths of every candidate that
actually existed on disk and was scanned. `_resolve_regex_files` was
already computing that list; we just surface it.

`llm_eval_handler` gains two connected pieces:

1. A new sentinel `"$RESOLVED_FILES"` in `files_to_include`. Expands to
   every path in `gathered_evidence["resolved_files"]`.

2. An automatic fallback: when `files_to_include` produces no file
   contents at all (e.g. `["$FOUND_FILE"]` with no preceding
   `file_exists` PASS -- the exact shape from darnitdevorg#402), the handler reads
   `gathered_evidence["resolved_files"]` on its own. Existing TOMLs
   with lone `["$FOUND_FILE"]` (before darnitdevorg#406 landed) start working
   correctly with no config change.

Both branches share the same `_read` helper and honor the 5-file cap
and 10000-byte truncation the handler already had.

## Why this shape

Preserves single-source-of-truth for the file list: the sibling
`pattern`/`regex` pass already declares `files = [...]`; `llm_eval`
doesn't need to enumerate them again. The follow-up work of teaching
`pattern` to also accept `.rst`/`.txt` variants (darnitdevorg#407) automatically
benefits `llm_eval` too, since `resolved_files` grows.

## Tests

5 new llm_eval tests + 1 new regex_handler test:

- `test_resolved_files_sentinel_reads_all_evidence_paths`: `$RESOLVED_FILES`
  fans out to every entry.
- `test_resolved_files_fallback_when_files_to_include_yields_empty`: the
  core darnitdevorg#402 reproducer -- `["$FOUND_FILE"]` with no `found_file` now
  reads the pattern-resolved list.
- `test_no_fallback_when_files_to_include_already_produced_content`:
  explicit paths win; fallback only fires when file_contents is empty.
- `test_fallback_respects_five_file_cap`: cap holds through the fallback.
- `test_no_fallback_when_gathered_evidence_has_no_resolved_files`:
  absent `resolved_files` is a no-op.
- `test_evidence_includes_resolved_files_on_match_path`: `resolved_files`
  lists paths that existed on disk; missing entries are excluded.

## Relationship to darnitdevorg#406 and darnitdevorg#407

Complementary:

- **darnitdevorg#406**: enumerated real paths in `llm_eval` blocks (TOML-only, fixes
  what the LLM sees when it fires).
- **darnitdevorg#407**: widened `pattern` file lists to include .rst / .txt / no-ext
  README variants (TOML-only, fixes how often the LLM fires).
- **This PR**: makes the framework carry the file list forward
  automatically so future TOMLs don't need to duplicate it.

All three can merge independently; together they close the whole
empty-consultation class of bug and reduce the maintenance burden on
future control authors.
@codecov-commenter

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

llm_eval receives empty file_contents when preceded by pattern/regex (no file_exists sibling)

2 participants