Skip to content

[core] Add primary-key vector bucket search#8569

Open
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/pk-vector-04-bucket-search
Open

[core] Add primary-key vector bucket search#8569
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/pk-vector-04-bucket-search

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the bucket-local search kernel for primary-key vector indexes. A bucket can now merge ANN results with exact fallback results from active data files that are not covered by ANN yet, while sharing the existing vector metric semantics and returning physical source positions for the later read path.

Changes

  • Extract common vector metric normalization, exact score/distance computation, and ANN score-to-distance conversion from AbstractVectorRead into VectorSearchMetric, preserving the existing higher-is-better raw score behavior while keeping PK distances lower-is-better and bounded for cosine.
  • Introduce a shared PkVectorSearchResult containing data file name, zero-based physical row position, and normalized distance, and use it directly from ANN search instead of the unreleased nested candidate type.
  • Add bounded-memory exact Top-K search for l2, cosine, and inner_product, preserving physical positions across null vectors and applying deletion-vector exclusions.
  • Add bucket-local search that validates ANN source metadata against active files, avoids rescanning ANN-covered files, forwards deletion vectors and search options, and deterministically merges ANN and exact candidates.

Testing

  • Verify the shared metric contract for all configured metrics, zero vectors, float rounding boundaries, large L2 distances, and ANN score conversion.
  • Verify multi-source ANN results map to physical data-file positions through the shared result type.
  • Verify exact search supports all configured metrics and rejects invalid queries.
  • Verify null vectors and deletion vectors preserve physical row addressing.
  • Verify bucket search merges ANN and exact fallback Top-K without rescanning covered files.
  • Run the focused Core suite, including existing raw-vector scoring paths: 18 tests passed.
  • Run non-fast Core compilation with Checkstyle, Spotless, and Maven enforcers.

Notes

@JingsongLi JingsongLi force-pushed the codex/pk-vector-04-bucket-search branch from 53efbbb to 1e547f1 Compare July 12, 2026 00:12
@JingsongLi JingsongLi marked this pull request as ready for review July 12, 2026 00:17

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the latest head. The shared metric extraction preserves the existing raw-vector score semantics, exact fallback keeps physical row positions across nulls and deletion vectors, and bucket-level merging avoids rescanning ANN-covered files while applying deterministic lower-is-better ordering. The focused 18-test Core suite passes locally. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants