Skip to content

feat: support batch incremental delta scans#508

Merged
JingsongLi merged 1 commit into
apache:mainfrom
Pandas886:feat/batch-incremental-delta
Jul 13, 2026
Merged

feat: support batch incremental delta scans#508
JingsongLi merged 1 commit into
apache:mainfrom
Pandas886:feat/batch-incremental-delta

Conversation

@Pandas886

@Pandas886 Pandas886 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #503

I'd like a small public API for batch incremental reads — basically "files between snapshot A (exclusive) and B (inclusive)" — without callers reinventing snapshot walking on top of TableScan.

This PR is intentionally thin: Delta + Auto selection only. Changelog / Diff come in follow-up PRs so review stays focused.

Brief change log

  • Add IncrementalScanMode, IncrementalScan, IncrementalPlan, IncrementalSplit
  • Range semantics: left-open / right-closed, empty range, out-of-bounds errors
  • Delta: plan data files from APPEND snapshots in the range via existing scan helpers
  • Auto: changelog-producer=none → Delta; otherwise resolves to Changelog (planning Changelog still fails loud until the next PR)
  • Push partition/bucket filters through from ReadBuilder when planning
  • TableRead::to_incremental_arrow for the data-split case

No new manifest reader code — it reuses what TableScan already does.

Tests

cargo test -p paimon --test incremental_batch_scan_test
cargo test -p paimon --lib
cargo fmt --all -- --check
cargo clippy -p paimon --all-targets --features fulltext,vortex -- -D warnings

API and Format

  • New public scan types under paimon::table (also re-exported at crate root)
  • No storage format change

Documentation

None yet — waiting until the SQL TVF lands so we document the whole story once. Can add a short rustdoc note if preferred.

Follow-ups

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit 2760b14 into apache:main Jul 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch incremental scan API: Delta mode and Auto selection

2 participants