fix(eval): include ragged prompt tails in PE and unified evaluation - #464
Open
Yuanjia-Lee wants to merge 1 commit into
Open
Yuanjia-Lee wants to merge 1 commit into
Yuanjia-Lee wants to merge 1 commit into
Conversation
_eval_pass floored the eval set to whole chunks of the training batch size, so any non-divisible tail never reached generation or scoring and the scored set silently changed with the batch config. Pad the last chunk with replicated roots (unique ":eval-pad:" lineage ids, mirroring ARTrainer), verify the per-batch reward count against the dispatch roots times the fan-out, and drop pad leaves from the mean. Below-chunk sets, divisible sets, and existing per-scorer means are unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_eval_passin PE and UnifiedModel floored the eval set to whole chunks ofthe training batch size (
usable = n_prompts - n_prompts % chunk or n_prompts). The ragged tail was dropped before generation, so what gotscored depended on the training batch config: 13 prompts at batch size 8 are
scored as 8, at batch size 4 as 12. The default
eval_num_prompts = batch_sizeis always divisible, which is why this never showed up.On a 13-prompt set whose full-set winner is B but whose first 8 prompts favor
A, chunk=8 scores A=1.00 vs B=0.75 and picks the wrong model; the full set is
0.615 vs 0.846. Checkpoint selection via the #202 eval suites is affected.
The fix pads the tail chunk with replicated roots given fresh
:eval-pad:ids (the
ARTrainer._pad_eval_inputspattern, moved intoeval_suites.py),keeps every dispatch at the training batch size, verifies each batch's reward
count against
roots x fan-out(so an engine-side mismatch fails loudlyinstead of skewing the mean), and drops pad descendants, so the scored set is
the configured set. Below-chunk, divisible, and per-scorer mean semantics
are unchanged.
Related Issue
N/A
Test Plan
Sampletypes: with thefix every configured prompt is scored and the mean matches the per-sample
oracle; divisible and below-chunk sets match legacy exactly.
_eval_passdrives a@distributedrollout+scorer role in a real worker; each root is scoredexactly once and pad leaves stay out of the mean.
ruff check/ruff format --check/check-docstring-lines/py_compilepass.Compatibility / Risk
No config, checkpoint, data-format, or API changes. Eval numbers on
non-divisible sets change (that is the fix); divisible and below-chunk sets
are byte-identical.
Reviewer Notes
eval_suites.pygainspad_eval_inputs;pe.pyandunified_model.pycallit in parallel in
_eval_pass.ARTrainer._pad_eval_inputsis untouched (itpads to the DP lcm, not the batch chunk). No open PR touches evaluation
batching.
Checklist