Skip to content

fix(eval): include ragged prompt tails in PE and unified evaluation - #464

Open
Yuanjia-Lee wants to merge 1 commit into
Tencent-Hunyuan:mainfrom
Yuanjia-Lee:fix/eval-tail
Open

Yuanjia-Lee wants to merge 1 commit into
Tencent-Hunyuan:mainfrom
Yuanjia-Lee:fix/eval-tail

Conversation

@Yuanjia-Lee

Copy link
Copy Markdown
Contributor

Summary

_eval_pass in PE and UnifiedModel floored the eval set to whole chunks of
the training batch size (usable = n_prompts - n_prompts % chunk or n_prompts). The ragged tail was dropped before generation, so what got
scored depended on the training batch config: 13 prompts at batch size 8 are
scored as 8, at batch size 4 as 12. The default eval_num_prompts = batch_size is always divisible, which is why this never showed up.

On a 13-prompt set whose full-set winner is B but whose first 8 prompts favor
A, chunk=8 scores A=1.00 vs B=0.75 and picks the wrong model; the full set is
0.615 vs 0.846. Checkpoint selection via the #202 eval suites is affected.

The fix pads the tail chunk with replicated roots given fresh :eval-pad:
ids (the ARTrainer._pad_eval_inputs pattern, moved into eval_suites.py),
keeps every dispatch at the training batch size, verifies each batch's reward
count against roots x fan-out (so an engine-side mismatch fails loudly
instead of skewing the mean), and drops pad descendants, so the scored set is
the configured set. Below-chunk, divisible, and per-scorer mean semantics
are unchanged.

Related Issue

N/A

Test Plan

  • Local harness (uncommitted, test: remove tests directory #99/test: remove tests directory #267) over real Sample types: with the
    fix every configured prompt is scored and the mean matches the per-sample
    oracle; divisible and below-chunk sets match legacy exactly.
  • Real dispatch on one 4090: the patched _eval_pass drives a
    @distributed rollout+scorer role in a real worker; each root is scored
    exactly once and pad leaves stay out of the mean.
  • ruff check / ruff format --check / check-docstring-lines /
    py_compile pass.

Compatibility / Risk

No config, checkpoint, data-format, or API changes. Eval numbers on
non-divisible sets change (that is the fix); divisible and below-chunk sets
are byte-identical.

Reviewer Notes

eval_suites.py gains pad_eval_inputs; pe.py and unified_model.py call
it in parallel in _eval_pass. ARTrainer._pad_eval_inputs is untouched (it
pads to the DP lcm, not the batch chunk). No open PR touches evaluation
batching.

Checklist

  • I reviewed the changed code and removed unrelated/generated artifacts.
  • I updated tests, docs, and configs where needed, or explained why not.

_eval_pass floored the eval set to whole chunks of the training batch
size, so any non-divisible tail never reached generation or scoring and
the scored set silently changed with the batch config. Pad the last
chunk with replicated roots (unique ":eval-pad:" lineage ids, mirroring
ARTrainer), verify the per-batch reward count against the dispatch
roots times the fan-out, and drop pad leaves from the mean.

Below-chunk sets, divisible sets, and existing per-scorer means are
unchanged.
@github-actions github-actions Bot added the need review Ready and waiting for review label Sep 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

need review Ready and waiting for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant