Skip to content

[core][flink] Publish primary-key vector indexes on write#8563

Open
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/pk-vector-03-write-lifecycle
Open

[core][flink] Publish primary-key vector indexes on write#8563
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/pk-vector-03-write-lifecycle

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Activates primary-key vector index maintenance in the write path on top of #8549 and #8562. Complete Level 1 compact files become ANN sources, and bucket-local ANN additions and removals are published with the same compact increment as their source data transition.

Changes

  • Force lookup compaction to rewrite Level 0 files for both deduplicate and partial-update tables, ensuring vector indexes are built only from complete Level 1 rows.
  • Create and restore a bucket-local vector index maintainer for primary-key table writers using the configured vector field, metric, algorithm, and field-scoped index options.
  • Publish ANN additions and removals through the matching compact increment and retain maintainer state across writer checkpoints.
  • Restore active source-backed ANN metadata from filesystem manifests without requiring an auxiliary directory index.
  • Extend the existing write-restore and Flink coordination signatures directly to round-trip active vector index payloads, adapting all callers without compatibility overloads.

Testing

  • Verify deduplicate and partial-update lookup compaction both rewrite Level 0 files.
  • Verify vector index additions and removals are attached to the same compact data transition.
  • Verify filesystem writer restore returns active source-backed ANN metadata.
  • Verify Flink coordinator restore round-trips ANN metadata with inclusive ordinal ranges.
  • Run the related Core suite: 19 tests passed.
  • Run the focused Flink coordinator test: 1 test passed.
  • Run non-fast Core/Flink compilation with Checkstyle, Spotless, and Maven enforcers.

Notes

@JingsongLi JingsongLi marked this pull request as ready for review July 11, 2026 15:30
@JingsongLi JingsongLi force-pushed the codex/pk-vector-03-write-lifecycle branch from 4adab77 to 989dee1 Compare July 11, 2026 15:46

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the latest head. The Level-0 rewrite is correctly scoped to the reachable deletion-vector lookup path, vector payload additions/removals are published with the matching compact increment, and filesystem/Flink restore round-trips the active source-backed payloads consistently. The focused Core tests and the new Flink coordinator test pass locally. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants