feat(report): state the grid probabilities arrived on, and show it costs the floor nothing - #98
Merged
Merged
Conversation
…sts the floor nothing Hosted Jev rounds to two decimals. The report now says so at the point of use: when every probability an arm returned sits on a grid of 0.001 or coarser, over at least 20 values, a Resolution line names it, and warns when the bins are narrower than the grid. scripts/quantization_floor.py answers whether the rounding raises the ECE floor. Across 40 to 10,000 rows, 10 or 20 bins, and grids of 0.01 or 0.05, a calibrated model reported on the grid clears the floor's 95th percentile at the nominal 5 percent, so the floor needs no correction. METHODOLOGY carries the table. Part of #9: whether the grid varies by model or option count needs live runs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #9. It answers two of the three open questions.
Should the report state the observed grid? Yes.
calibration.probability_grid(values)returns the coarsest of 0.1, 0.05, 0.01, 0.005 and 0.001 that every value sits on, orNone. It needs at least 20 values, because a handful landing on a grid says nothing about the vendor. When there is a grid, the arm's section gets a line like this:When the bins are narrower than the grid, the line adds a warning that the binning measures the rounding.
Is the achievable ECE floor higher than the one computed? No.
scripts/quantization_floor.pysimulates a perfectly calibrated model with continuous true probabilities, sends them rounded to the grid, and reads ECE on the rounded values againstcalibration_floorof those same values, as the report does. If rounding cost calibration that the floor doesn't model, the calibrated model would clear the floor's 95th percentile more than 5% of the time.With 200 trials per cell:
The false-alarm rate stays at the nominal 5% (sampling error is about 1.5 points), even on a grid five times coarser than the vendor's. Rounding moves each value by at most half a step, in both directions, and since the floor is computed from the rounded values those errors cancel within a bin. METHODOLOGY has the full table, and the script reproduces it.
Still open on #9: whether the grid varies by model or by option count. That needs live runs against more than one model version.
Tests cover grid detection at 0.05, 0.01 and 0.001, no grid for values finer than that or for too few values, and the report line for an adapter on the 0.01 grid.
docs/example-report.mdis unchanged, since the mock's probabilities are continuous. The gate, the site checks and the prose checks all pass.🤖 Generated with Claude Code