Skip to content

[core] Parse primary-key vector index source metadata (_SOURCE_META)#515

Merged
JingsongLi merged 5 commits into
apache:mainfrom
JunRuiLee:feat/pk-vector-source-meta
Jul 13, 2026
Merged

[core] Parse primary-key vector index source metadata (_SOURCE_META)#515
JingsongLi merged 5 commits into
apache:mainfrom
JunRuiLee:feat/pk-vector-source-meta

Conversation

@JunRuiLee

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: part of #514 — PR 1 of 5 (does not close the issue)

This is the first, read-only slice of primary-key vector (bucket-local ANN) search support. It mirrors Apache Paimon (Java) apache/paimon#8549 as of commit a50a36ff8.

Java primary-key vector indexes append a nullable _SOURCE_META blob to GlobalIndexMeta listing the ordered source data files and row counts of a bucket-local ANN payload; an ANN ordinal maps to a (data file, physical row position) pair. This PR gives paimon-rust the ability to parse that metadata and resolve an ordinal — the foundation PR 2 (bucket search) and PR 3 (physical-position read) build on.

Read/parse only. Out of scope for this PR (tracked in #514): ANN search, manifest selection, physical-row materialization, index building, and engine SQL entry points.

Brief change log

  • Add a nullable source_meta: Option<Vec<u8>> field to GlobalIndexMeta (serde _SOURCE_META, serde_bytes), matching Java's 6th GlobalIndexMeta field.
  • New spec/pk_vector_source.rs: PkVectorSourceFile / PkVectorSourceMeta, a byte-for-byte parser for Java's PkVectorSourceMeta frame (version 1, count, writeUTF filenames, writeLong row counts, trailing-bytes check), and an ordinal-only resolve(ordinal) -> (file, position) prefix-scan resolver.
  • Decode _SOURCE_META through the hand-written Avro index-manifest decoder, schema-guarded: a pre-#8549 5-field _GLOBAL_INDEX writer schema is detected from the file's writer schema and the 6th field is not read, so old manifests never misalign the byte stream.
  • The _SOURCE_META inner blob is Java DataOutput (big-endian ints/longs, modified UTF-8), parsed independently of the outer Avro layer; parsing helpers are module-private (no shared common/ extraction in this PR).

Tests

All in spec/pk_vector_source.rs and spec/index_manifest.rs (22 tests; full cargo test -p paimon green):

  • Frame parsing: single/multi source file; rejects bad version, zero count, trailing bytes, truncated input, negative row count, empty list.
  • Java modified UTF-8: ASCII/2-byte/3-byte/supplementary round-trips; C0 80 → NUL; raw 0x00 accepted (Java readUTF parity); truncated/bad-continuation and lone-surrogate rejected.
  • Resolver: single-file, multi-file prefix-sum boundaries, negative/out-of-range ordinal errors, checked_add overflow branch.
  • Avro: new-format _SOURCE_META round-trips (Some/None); legacy 5-field _GLOBAL_INDEX decodes without misalignment and yields source_meta = None.

API and Format

  • No storage-format change originated here: this reads the _SOURCE_META format defined by Java [core] Add primary-key vector index foundation paimon#8549. _SOURCE_META is a nullable trailing field, so existing index manifests remain readable (verified by the legacy-schema test).
  • Public API additions only (PkVectorSourceFile, PkVectorSourceMeta); no existing API changed.

Documentation

No user-facing documentation changes.

@JunRuiLee JunRuiLee marked this pull request as ready for review July 12, 2026 16:35

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit eed779e into apache:main Jul 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants