Fix swapped interpolation weights and tail branches in TDigestDouble - #755
Open
jaideeppyne wants to merge 2 commits into
Open
Fix swapped interpolation weights and tail branches in TDigestDouble#755jaideeppyne wants to merge 2 commits into
jaideeppyne wants to merge 2 commits into
Conversation
The left tail branch returned the interpolated weight without dividing by centroidsWeight_, unlike the mirror-image right tail branch and the reference implementation. Reachable via heapify() when the first centroid has weight greater than 1, making getRank() exceed its documented [0, 1] range.
The interpolation passed weightedAverage(mean[i], w1, mean[i+1], w2) where the reference passes the weights in the opposite order, so the estimate moved toward the lower centroid as the rank rose within a bracket. This made getQuantile() non-monotonic for ordinary update() sequences and cost about an order of magnitude of rank accuracy. The right tail also added the interpolated offset to maxValue_ instead of subtracting it, returning quantiles above getMaxValue().
This was referenced Aug 30, 2026
jaideeppyne
force-pushed
the
tdigest-rank-left-tail
branch
from
August 31, 2026 13:40
23920f2 to
cfd7ea1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three spots in
TDigestDoublediffer from the reference implementation (MergingDigest). The first affects ordinary use, the other two needheapify.I originally opened this for the third one only, then found the first while checking whether that fix was partial.
1. getQuantile() passes the interpolation weights in the wrong order
Reference:
We compute the same two quantities as
w1/w2and then callweightedAverage(centroidMeans_[i], w1, centroidMeans_[i + 1], w2). The weight onmean[i]should be the distance to the far end, not the near one. As the rank rises inside a bracket the estimate moves toward the lower centroid instead of the upper one.That makes
getQuantile()non-monotonic from a plainupdate()sequence. 1999 of 2000 random digests (k in {10, 20, 50, 100, 200}, n in [100, 50000], gaussian / uniform / lognormal / sequential) return somegetQuantile(r2) < getQuantile(r1)withr2 > r1. Worst seen: rank 0.9995 gives 16799.3 after rank 0.9990 gave 36135.9.It costs accuracy too. Rank error against exact quantiles, 29700 queries at k=100, n=20000:
2. getQuantile() right tail has the wrong sign
Reference is
max - (...), we havemaxValue_ + (...), so the branch returns quantiles abovegetMaxValue(). On min 0, centroids (10,100),(20,100),(30,100), max 40, rank 0.90 returns 45.92 where the reference gives 34.08.3. getRank() left tail is not normalized
It omits the
/ centroidsWeight_that both the mirror-image right tail below it and the reference have, sogetRank()leaves its documented [0, 1] range. Same digest,getRank(9.0)returns 45.1 instead of 0.1503.getCDF([5, 20, 35])returns[25.5, 0.5, 0.915, 1.0], andgetPMFreturns a negative mass of-25.0.2 and 3 are not reachable from
update()ormerge(), becausecompress()leaves both tail centroids singletons. They are reachable fromheapify(), which accepts a first or last centroid of weight greater than 1.Three named tests, each verified failing before its own fix. Full suite is 39 failures before and after, all
*CrossLanguageTestwith emptycpp_generated_filesfixtures, unrelated to this.C++ has all three the same way, and I reproduced the non-monotonic quantiles through the Python binding on the C++ core, 200 of 200 trials, so it probably wants the same changes.
I used Claude Code on this. The oracles were tdunning's
MergingDigestand monotonicity, which needs no second implementation.