fix: REQ sketch loses min/max when merging into a new sketch - #170
Merged
Conversation
NewSketch left minItem/maxItem at Go's zero value while Merge and Reset use NaN as the unset sentinel, so merging into a freshly created sketch computed min(0, other.minItem) and max(0, other.maxItem). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
req.NewSketch()leavesminItem/maxItemat Go's zero value, butMergeandResetboth use NaN as the "unset" sentinel. So merging into a freshly created sketch computesmin(0, other.minItem)andmax(0, other.maxItem), and the 0 wins whenever the stream doesn't straddle it.It's not just the getters. The bogus 0 is reinserted into the sorted view as a retained quantile, so
Quantile,Rank,CDFandPMFall shift, and it survivesMarshalBinary+Decode. Chained merges into an empty accumulator, which is the usual map-reduce shape, hit it too.NewSketch()thenReset()thenMergeworks fine, which is the tell.ReqSketch.javainitialises the fields at declaration:Go has no field initialisers so the port dropped it. I set them in
NewSketchinstead. That covers every construction site: the decoder's empty and raw-items paths both go throughNewSketch, and the exact and estimation paths read min/max out of the buffer.Updateguards onIsEmpty()so it was never affected, and every reader of the fields (MinItem,MaxItem,refreshSortedView, the estimation-format encoder) is already behind an emptiness check, so no path can now observe the NaN.How I found it: I built an invariant harness over the quantile families asserting that min/max after merging into a fresh sketch equals min/max after merging into a reset sketch, and equals the source's. 5 value patterns across REQ, KLL and t-digest. KLL and t-digest are clean; REQ fails 4 of 5. The one that passes is the case whose data straddles zero, which is why the existing
Merge Multipletest misses it. I checked the intended semantics againstReqSketch.javarather than guessing at it. The same harness also ran 200-trial randomised sweeps per family for quantile monotonicity in rank, CDF within [0,1] and non-decreasing, PMF summing to 1, and serialize/deserialize identity; nothing else came out of it, and I confirmedtdigest.Quantilematchestdigest_impl.hppline for line.7 subtests added in
TestSketchMergeIntoEmptyMinMax. All 7 fail onmainwith only the test file applied, all 7 pass with the two-line change.make lint-checkandgo test -race ./req/are clean and the full suite is green.Written with AI assistance (Claude). I reviewed the change and ran everything above myself.