feat(behavior): score a saved model against annotated sessions - #105
Merged
Conversation
Nothing could answer "how does this .pkl do on these annotations". Cross-validation fits N fold models and discards them; training with a holdout measures a model it just fitted. BehaviorModel.load/predict existed and benchmarks/metrics.py had an evaluator, but nothing joined them, so validating a bundle meant a one-off script. evaluate_model() rebuilds features from the bundle's own spec/window/stats -- matching what it was fitted on rather than what the caller would have picked -- predicts, and reports per-class precision/recall/F1, pooled macro F1, a confusion matrix and bout metrics. Three deliberate choices: Frames the model declines on (a window it could not fill) are counted in n_unscored, not dropped silently and not charged as errors. "Declined to answer" is not "wrong". Frames whose causal window still covers the previous behavior are excluded, the same rule cross-validation scores under, so a CV score and an evaluation score mean the same thing. Classes below support_floor are reported but kept out of the macro average. A 25-frame class averaged as an equal fifth measures its own luck; the exclusion is stated in the result rather than left to be inferred. The metric assembly is a separate pure function so it can be tested on hand-built label sequences where the expected numbers are arithmetic. _bout_recall becomes public bout_metrics: evaluation and cross-validation must agree on what counts as a detected bout, so they share one definition.
#104 landed the clean-window scoring rule this branch had copied ahead of it, so both sides defined _clean_window_rows identically and git took one copy. The only real conflict was the helper's name: main still calls it _bout_recall, this branch makes it public bout_metrics so evaluation and cross-validation can share one definition of a detected bout. Kept the rename, which is the point of the branch. n_rows_scored from #104 comes through untouched.
This was referenced Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
There was no way to ask "how does this saved
.pkldo on theseannotations". Cross-validation fits N fold models and discards them; the
held-out path measures a model it just fitted.
BehaviorModel.load()/.predict()existed andbenchmarks/metrics.pyhad an evaluator, but nothingjoined them — validating a bundle meant a one-off script.
evaluate_model(model_path, sessions, ...)rebuilds features from the bundle'sown
spec/window/stats(so the columns match what it was fitted on, notwhat the caller would have picked), predicts, and reports per-class
precision/recall/F1, pooled macro F1, confusion matrix, and bout metrics.
Three deliberate choices
Declined frames are counted, not charged.
predict()returns""wherethe rolling window could not be filled. Those land in
n_unscoredrather thanbeing dropped silently or scored as errors — "declined to answer" is not
"wrong".
Window-contaminated frames are excluded, the same rule cross-validation
scores under (#104), so a CV score and an evaluation score mean the same thing
and can be compared. Reported as
n_window_contaminated.Thin classes are reported but excluded from the macro average. A class with
25 frames averaged as an equal fifth of macro F1 measures its own luck. This is
the lesson from a real cohort where a behaviour present in 4 of 34 sessions
scored 0.0 in the two folds containing none of it and pulled mean-of-folds from
0.739 to 0.679. The exclusion is stated in
thin_classes, not left to infer.Structure
The metric assembly is a separate pure function,
summarise_predictions, so itcan be tested on hand-built label sequences where the expected numbers are
arithmetic rather than whatever the classifier happened to do.
_bout_recallbecomes publicbout_metrics. Evaluation and cross-validationmust agree on what counts as a detected bout, so they share one definition
rather than keeping two that can drift. Two existing tests referenced it by the
private name and now use the public one.
Tests
14 cases. Exact per-class arithmetic; accuracy; macro over scored classes only;
a thin class reported but excluded; a class the model never predicts still
appearing with zero recall; a class only ever predicted charged against its own
precision; confusion square over every label; bouts counted per session rather
than across them; empty input returning a result instead of raising. Plus
end-to-end loading of a bundle this suite trains itself.
2816 passed, 1 skipped; ruff clean.Note on scope
This has no in-app consumer yet — it was designed as the engine behind an
Evaluate tab, and the immediate need was met by a standalone script instead. It
is tested and self-contained, but if you would rather not carry an unconsumed
module, holding this until the tab exists is a reasonable call. Flagging rather
than assuming.