Skip to content

feat: support DataFusion paimon_incremental_query#511

Open
Pandas886 wants to merge 4 commits into
apache:mainfrom
Pandas886:feat/datafusion-incremental-query
Open

feat: support DataFusion paimon_incremental_query#511
Pandas886 wants to merge 4 commits into
apache:mainfrom
Pandas886:feat/datafusion-incremental-query

Conversation

@Pandas886

@Pandas886 Pandas886 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #506

Depends on: #508#509#510. This PR only adds the DataFusion SQL surface on top of that stack.

I wanted something you can actually type in SQL without dropping into the Rust API:

SELECT * FROM paimon_incremental_query('t', 0, 2);
SELECT * FROM paimon_incremental_query('t', 0, 2, 'diff');
SELECT rowkind, id, value FROM paimon_incremental_query('t$audit_log', 0, 2, 'changelog');

Brief change log

  • Register TVF paimon_incremental_query in SQLContext
  • 3-arg form defaults to auto; optional 4th arg: auto / delta / changelog / diff
  • Physical plan delegates to core IncrementalScan / AuditLogTableno second planning implementation in the DF crate
  • Support $audit_log suffix for rowkind output
  • Projection + residual filter (filter is not pushed into core incremental yet; kept conservative)
  • Update docs/src/sql.md with the function and a couple examples

I skipped the Java golden / lookup-compact E2E path on purpose: those need compact/lookup generation which isn't in this contribution series. Coverage is pure Rust / input-changelog based.

Tests

cargo test -p paimon-datafusion --test incremental_query
cargo fmt --all -- --check
cargo clippy -p paimon -p paimon-datafusion --all-targets --features fulltext,vortex -- -D warnings

API and Format

  • New SQL table function only; storage format unchanged
  • Relies on the core incremental APIs from the parent PRs

Documentation

Yes — docs/src/sql.md updated.

Review tip

If reviewing against main, the DF-only commit is the tip: feat: support DataFusion paimon_incremental_query. Happy to restack onto latest main once #508#510 land.

@JingsongLi

Copy link
Copy Markdown
Contributor

The Diff problem of inheriting '# 510'; The non audit mode will first push the SQL projection down to the core, directly triggering the issue of "only comparing projection columns/missing PK". In addition, the catalog section of 'catalog. db. table' is discarded after parsing, and the TVF global registration will be overwritten by the last registered catalog. Multiple catalog scenarios may result in reading the wrong catalog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataFusion: paimon_incremental_query table function

2 participants