(diversity) attribute diversity implementation with post process and adaptive L#1250
Draft
narendatha wants to merge 5 commits into
Draft
(diversity) attribute diversity implementation with post process and adaptive L#1250narendatha wants to merge 5 commits into
narendatha wants to merge 5 commits into
Conversation
Remove the experimental_diversity_search feature gate (graduate the diverse-search primitives to always-on) and add a post-processing-only attribute-bucket diversity path: a plain greedy search over the top-L pool followed by AttributeBucketDiversity, which keeps at most diverse_results_k results per distinct attribute value. Exposed as SearchMode::DiverseAttribute and wired into the disk benchmark via new attributes/diverse_attribute_id/diverse_results_k inputs.
Add DiverseAdaptiveSearch, a greedy graph search that samples attribute-bucket concentration during traversal and grows the search list L when few distinct buckets are seen (yield = sum min(count_b,k)/visited). The enlarged pool feeds the existing AttributeBucketDiversity post-processor. Wired through SearchMode::DiverseAttribute's optional adaptive_l and the disk benchmark. Reuses inline-filter compute_adaptive_l. Includes unit tests for the bucket-yield sizing.
AttributeBucketDiversity::post_process previously recomputed exact distances for every candidate in the L-pool, fetching each full-precision vector and re-evaluating the distance. This duplicated work already done during traversal and made Design A slower than the queue method at higher L. Mirror RerankAndFilter: consult scratch.distance_cache first and only fetch/recompute for cache misses, then sort and run the greedy per-bucket selection. Recall and IO counts are unchanged; redundant vector fetches and distance recomputation are eliminated. Add diverse-search-results.md documenting the Enron/Caselaw 3-way comparison (standard, queue, Design A, Design B) and this optimization.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
What does this implement/fix? Briefly explain your changes.
Any other comments?