Skip to content

Fix lm-eval continuation scoring after partial context truncation - #1902

Merged
michalk8 merged 1 commit into
ml-explore:mainfrom
sqliangx:fix-lm-eval-partial-truncation
Sep 21, 2026
Merged

michalk8 merged 1 commit into
ml-explore:mainfrom
sqliangx:fix-lm-eval-partial-truncation

Conversation

@sqliangx

@sqliangx sqliangx commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR fixes a scoring bug in the lm-eval adapter's MLXLM.loglikelihood() method. Currently, MLXLM.loglikelihood() uses the truncated prefix length to slice untruncated full sequences. As a result, it incorrectly includes context tokens in continuation scoring after partial context truncation.

For example, with a character tokenizer, context abc, continuation de, and max_tokens=3, the context becomes bc. The code should score de, but it currently scores cde instead.

The PR fixes the bug by saving the original continuation boundary before truncating the prefix. This keeps the scoring focused on the continuation. The existing truncation policy and fully truncated (-inf, False) behavior stay the same.

Testing

This PR adds a test covering no truncation and two partial-truncation cases. Both partial-truncation cases fail before the fix and pass afterward.

  • ☑️ I understand it is strictly prohibited to use AI to write PR description
  • AI usage disclosure: bug finding

@michalk8 michalk8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for spotting this!

@michalk8
michalk8 merged commit c69d128 into ml-explore:main Sep 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants