[core] Parse primary-key vector index source metadata (_SOURCE_META)#515
Merged
Merged
Conversation
This was referenced Jul 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: part of #514 — PR 1 of 5 (does not close the issue)
This is the first, read-only slice of primary-key vector (bucket-local ANN) search support. It mirrors Apache Paimon (Java) apache/paimon#8549 as of commit
a50a36ff8.Java primary-key vector indexes append a nullable
_SOURCE_METAblob toGlobalIndexMetalisting the ordered source data files and row counts of a bucket-local ANN payload; an ANN ordinal maps to a(data file, physical row position)pair. This PR gives paimon-rust the ability to parse that metadata and resolve an ordinal — the foundation PR 2 (bucket search) and PR 3 (physical-position read) build on.Read/parse only. Out of scope for this PR (tracked in #514): ANN search, manifest selection, physical-row materialization, index building, and engine SQL entry points.
Brief change log
source_meta: Option<Vec<u8>>field toGlobalIndexMeta(serde_SOURCE_META,serde_bytes), matching Java's 6thGlobalIndexMetafield.spec/pk_vector_source.rs:PkVectorSourceFile/PkVectorSourceMeta, a byte-for-byte parser for Java'sPkVectorSourceMetaframe (version1, count,writeUTFfilenames,writeLongrow counts, trailing-bytes check), and an ordinal-onlyresolve(ordinal) -> (file, position)prefix-scan resolver._SOURCE_METAthrough the hand-written Avro index-manifest decoder, schema-guarded: a pre-#8549 5-field_GLOBAL_INDEXwriter schema is detected from the file's writer schema and the 6th field is not read, so old manifests never misalign the byte stream._SOURCE_METAinner blob is JavaDataOutput(big-endian ints/longs, modified UTF-8), parsed independently of the outer Avro layer; parsing helpers are module-private (no sharedcommon/extraction in this PR).Tests
All in
spec/pk_vector_source.rsandspec/index_manifest.rs(22 tests; fullcargo test -p paimongreen):C0 80→ NUL; raw0x00accepted (JavareadUTFparity); truncated/bad-continuation and lone-surrogate rejected.checked_addoverflow branch._SOURCE_METAround-trips (Some/None); legacy 5-field_GLOBAL_INDEXdecodes without misalignment and yieldssource_meta = None.API and Format
_SOURCE_METAformat defined by Java [core] Add primary-key vector index foundation paimon#8549._SOURCE_METAis a nullable trailing field, so existing index manifests remain readable (verified by the legacy-schema test).PkVectorSourceFile,PkVectorSourceMeta); no existing API changed.Documentation
No user-facing documentation changes.