Skip to content

feat(behavior): score a saved model against annotated sessions - #105

Merged
gbradham merged 2 commits into
mainfrom
feature/evaluation-module
Aug 6, 2026
Merged

feat(behavior): score a saved model against annotated sessions#105
gbradham merged 2 commits into
mainfrom
feature/evaluation-module

Conversation

@gbradham

@gbradham gbradham commented Aug 6, 2026

Copy link
Copy Markdown
Member

What

There was no way to ask "how does this saved .pkl do on these
annotations". Cross-validation fits N fold models and discards them; the
held-out path measures a model it just fitted. BehaviorModel.load()/
.predict() existed and benchmarks/metrics.py had an evaluator, but nothing
joined them — validating a bundle meant a one-off script.

evaluate_model(model_path, sessions, ...) rebuilds features from the bundle's
own spec/window/stats (so the columns match what it was fitted on, not
what the caller would have picked), predicts, and reports per-class
precision/recall/F1, pooled macro F1, confusion matrix, and bout metrics.

Three deliberate choices

Declined frames are counted, not charged. predict() returns "" where
the rolling window could not be filled. Those land in n_unscored rather than
being dropped silently or scored as errors — "declined to answer" is not
"wrong".

Window-contaminated frames are excluded, the same rule cross-validation
scores under (#104), so a CV score and an evaluation score mean the same thing
and can be compared. Reported as n_window_contaminated.

Thin classes are reported but excluded from the macro average. A class with
25 frames averaged as an equal fifth of macro F1 measures its own luck. This is
the lesson from a real cohort where a behaviour present in 4 of 34 sessions
scored 0.0 in the two folds containing none of it and pulled mean-of-folds from
0.739 to 0.679. The exclusion is stated in thin_classes, not left to infer.

Structure

The metric assembly is a separate pure function, summarise_predictions, so it
can be tested on hand-built label sequences where the expected numbers are
arithmetic rather than whatever the classifier happened to do.

_bout_recall becomes public bout_metrics. Evaluation and cross-validation
must agree on what counts as a detected bout, so they share one definition
rather than keeping two that can drift. Two existing tests referenced it by the
private name and now use the public one.

Tests

14 cases. Exact per-class arithmetic; accuracy; macro over scored classes only;
a thin class reported but excluded; a class the model never predicts still
appearing with zero recall; a class only ever predicted charged against its own
precision; confusion square over every label; bouts counted per session rather
than across them; empty input returning a result instead of raising. Plus
end-to-end loading of a bundle this suite trains itself.

2816 passed, 1 skipped; ruff clean.

Note on scope

This has no in-app consumer yet — it was designed as the engine behind an
Evaluate tab, and the immediate need was met by a standalone script instead. It
is tested and self-contained, but if you would rather not carry an unconsumed
module, holding this until the tab exists is a reasonable call. Flagging rather
than assuming.

Unrelated: tests/unit/nodes/test_delay_node_precision.py::test_delay_node_is_accurate_under_realistic_loop_pressure
fails intermittently (2 of 4 runs) on unmodified main. Pre-existing timing
flake, worth its own fix.

Nothing could answer "how does this .pkl do on these annotations".
Cross-validation fits N fold models and discards them; training with a holdout
measures a model it just fitted. BehaviorModel.load/predict existed and
benchmarks/metrics.py had an evaluator, but nothing joined them, so validating
a bundle meant a one-off script.

evaluate_model() rebuilds features from the bundle's own spec/window/stats --
matching what it was fitted on rather than what the caller would have picked --
predicts, and reports per-class precision/recall/F1, pooled macro F1, a
confusion matrix and bout metrics.

Three deliberate choices:

Frames the model declines on (a window it could not fill) are counted in
n_unscored, not dropped silently and not charged as errors. "Declined to
answer" is not "wrong".

Frames whose causal window still covers the previous behavior are excluded,
the same rule cross-validation scores under, so a CV score and an evaluation
score mean the same thing.

Classes below support_floor are reported but kept out of the macro average. A
25-frame class averaged as an equal fifth measures its own luck; the exclusion
is stated in the result rather than left to be inferred.

The metric assembly is a separate pure function so it can be tested on
hand-built label sequences where the expected numbers are arithmetic.

_bout_recall becomes public bout_metrics: evaluation and cross-validation must
agree on what counts as a detected bout, so they share one definition.
#104 landed the clean-window scoring rule this branch had copied ahead of it,
so both sides defined _clean_window_rows identically and git took one copy.
The only real conflict was the helper's name: main still calls it
_bout_recall, this branch makes it public bout_metrics so evaluation and
cross-validation can share one definition of a detected bout. Kept the rename,
which is the point of the branch.

n_rows_scored from #104 comes through untouched.
@gbradham
gbradham merged commit fdc969c into main Aug 6, 2026
3 of 4 checks passed
@gbradham
gbradham deleted the feature/evaluation-module branch August 6, 2026 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant