Skip to content

fix(behavior): stop scoring frames whose window straddles a bout edge - #104

Merged
gbradham merged 1 commit into
mainfrom
fix/cv-clean-window
Aug 6, 2026
Merged

fix(behavior): stop scoring frames whose window straddles a bout edge#104
gbradham merged 1 commit into
mainfrom
fix/cv-clean-window

Conversation

@gbradham

@gbradham gbradham commented Aug 6, 2026

Copy link
Copy Markdown
Member

What

The rolling window is causal: row i's features summarise frames
i-window+1 .. i. For the first window-1 frames of every bout, that span
still covers the previous behaviour. Cross-validation was asking the model
to name a behaviour from features that mostly describe a different one, then
counting it wrong when it obliged.

Those frames cannot be classified correctly however good the model is. Scoring
them measures where the annotator drew the boundary, not the classifier.

Only the leading edge is affected — the end of a bout is fine, its window
lies wholly inside it.

Impact

On a 34-session cohort with ~25-frame bouts at window 8, these are 28% of all
scored frames
. Applying the same rule to a held-out evaluation moved macro F1
+0.022 (0.685 → 0.707) and accuracy +0.022 — performance that was always
there and was being hidden by frames the model had no way to get right.

This shifts reported CV numbers upward. Runs measured before and after are
not directly comparable. The change is toward the honest figure, but it is a
behaviour change and worth knowing when comparing against previously recorded
results.

Training is untouched

Deliberately. A partly-filled window is still a real input the deployed model
will meet at a genuine behaviour transition, and dropping those rows would
shrink the training set for no gain. This is a scoring change only.

n_rows_scored

n_rows_kept counts rows assembled for training, which now differs from what
was measured. The result dict gains n_rows_scored so nobody divides by the
wrong denominator.

One existing test asserted sum(support) == n_rows_kept — that invariant is
precisely what this changes, so it now asserts == n_rows_scored plus
n_rows_scored <= n_rows_kept. Worth a look, since it changes an existing
assertion rather than adding one.

Tests

Ten cases covering the rule directly: leading frames excluded, trailing frames
kept, bouts shorter than the window never scorable, session boundaries not
bridged, frame-number gaps restarting the window, window=1 a no-op,
out-of-order rows handled, and the excluded count equalling window-1
across several window sizes.

2813 passed, 1 skipped; ruff clean.

An intermittent single failure appeared in one full-suite run and did not
recur with a fixed seed. Same pre-existing flake noted on #102, unrelated to
this change.

The rolling window is causal: row i's features summarise frames
i-window+1 .. i. For the first window-1 frames of every bout that span still
covers the PREVIOUS behavior, so cross-validation was asking the model to name
a behavior from features that mostly describe a different one, then counting
it wrong when it obliged.

Those frames cannot be classified correctly however good the model is. Scoring
them measures where the annotator drew the boundary, not the classifier. On a
34-session cohort with ~25-frame bouts at window 8 they are 28% of all scored
frames, and excluding them moved measured macro F1 by about +0.02 on a held-out
set -- performance that was always there and was being hidden.

Only the leading edge is affected; the end of a bout is fine, its window lies
wholly inside it. Training is deliberately untouched: a partly-filled window is
still a real input the deployed model will meet, and dropping those rows would
shrink the training set for no gain.

Adds n_rows_scored to the CV result. n_rows_kept counts rows assembled for
training, which now differs from what was measured, and one existing test was
asserting the two were equal -- that invariant is exactly what changed, so it
now asserts support sums to n_rows_scored instead.

Note this shifts reported CV numbers upward. Runs measured before and after are
not directly comparable, and the change is toward the honest figure.
@gbradham
gbradham merged commit 0e52b00 into main Aug 6, 2026
4 checks passed
@gbradham
gbradham deleted the fix/cv-clean-window branch August 6, 2026 12:44
gbradham added a commit that referenced this pull request Aug 6, 2026
#104 landed the clean-window scoring rule this branch had copied ahead of it,
so both sides defined _clean_window_rows identically and git took one copy.
The only real conflict was the helper's name: main still calls it
_bout_recall, this branch makes it public bout_metrics so evaluation and
cross-validation can share one definition of a detected bout. Kept the rename,
which is the point of the branch.

n_rows_scored from #104 comes through untouched.
gbradham added a commit that referenced this pull request Aug 6, 2026
_run_cv_folds has reported n_rows_scored since #104, but the summary written
into the model bundle never copied it. A report could therefore only say how
many rows were assembled -- n_rows_kept, which counts mirrored copies and
window-contaminated frames that train but never score -- leaving the reader to
work the real denominator out from the per-class supports.

On model11 that was 17,129 scored against 25,005 kept, a 31% difference.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant