Skip to content

feat(check): cap every finding list in the JSON payload - #60

Merged
tybot02 merged 1 commit into
mainfrom
feat/check-output-cap
Aug 26, 2026
Merged

tybot02 merged 1 commit into
mainfrom
feat/check-output-cap

Conversation

@lucasmaan

Copy link
Copy Markdown
Contributor

Problem

Pointed at a 1024-document knowledge base, kb-ai check printed 482KB of JSON, of
which 1024 rows were {"document": ..., "reason": "missing"} — the same reason every
time, for a condition check_extractions itself documents as not a fault.

A diagnostic that takes a screenful to say "nothing is wrong" does not get read. The
one thing that run had to say — 0 mismatched, and grounding unable to check any of the
698 articles it found — was buried in it.

Change

Every finding list is cut to 20 items by default.

The cap is a display limit and never a measurement:

  • count stays the true total whatever --limit does;
  • truncated says outright that rows were dropped, so a capped list cannot be misread
    as a complete one;
  • the stderr summaries always speak for the whole set;
  • --limit 0 restores the full payload, --limit N takes a different slice.

Truncating into a bare list would have been worse than not truncating: twenty rows that
look like the whole set is a wrong answer, where twenty rows labelled "of 1024" is a
short one.

The cap is uniform, so the actionable lists (mismatched, unsourced) are cut at the
same 20 as the benign missing that motivated it. That is the right default — a KB
with thousands of genuine faults has a bigger problem than its report length — but it
is why --limit 0 exists.

Verification

  • 27 tests; kb_ai.commands.check at 100% statement coverage (50 statements, 0 missing).
  • The invariant was checked directly rather than assumed: across --limit
    unset/0/5/1000, count is identical for every list, rows equal
    min(limit, count), truncated is true exactly when rows were dropped, and the
    stderr summaries are byte-identical.
  • Measured on the real 1024-document KB: 482KB uncapped, 10.6KB at the default.
  • Bad --limit values (negative, non-numeric, float) all exit 2 with argparse usage.
  • Six mutations all killed with a green control and reverse control.

🤖 Generated with Claude Code

@tybot02
tybot02 force-pushed the feat/check-output-cap branch from 1375b93 to 89a832a Compare August 26, 2026 10:23
Pointed at a 1024-document knowledge base, `check` printed 482KB of JSON,
of which 1024 rows were `{"document": ..., "reason": "missing"}` -- the same
reason every time, for a condition check_extractions itself documents as not
a fault. The one thing the run had to say (0 mismatched, and grounding unable
to check any of the 698 articles it found) was buried in it.

Every finding list is now cut to 20 items by default. The cap is a display
limit and never a measurement: `count` stays the true total whatever --limit
does, and `truncated` says outright that rows were dropped, so a capped list
cannot be misread as a complete one. `--limit 0` restores the full payload.

The cap is uniform, so the actionable lists (`mismatched`, `unsourced`) are
cut at the same 20 as the benign `missing` that motivated it -- the right
default, since a KB with thousands of genuine faults has a bigger problem
than its report length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tybot02
tybot02 force-pushed the feat/check-output-cap branch from 89a832a to da9d971 Compare August 26, 2026 11:35
@tybot02
tybot02 merged commit 273a2bb into main Aug 26, 2026
10 checks passed
@tybot02
tybot02 deleted the feat/check-output-cap branch August 26, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants