feat: [iceberg] support iceberg metadata columns _file, _partition by parthchandra · Pull Request #4752 · apache/datafusion-comet

parthchandra · 2026-06-29T16:37:40Z

Which issue does this PR close?

Adds support for iceberg metadata columns

Closes #.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

hsiang-c · 2026-07-09T17:09:49Z

+          // Data under spec 0
+          spark.sql(s"INSERT INTO $table VALUES (1, 'US', 'A'), (2, 'EU', 'B')")
+
+          // Evolve partition spec: add category field -> spec 1


(nit) We might be able to use SparkSQL as well but it is not a must.

ALTER TABLE $table ADD PARTITION FIELD category;

ref: https://iceberg.apache.org/docs/nightly/spark-ddl/#alter-table-add-partition-field

hsiang-c · 2026-07-09T17:17:05Z

        withFallbackReason(scan, "Comet Scan is not enabled")

-      case scan if metadataCols(scan).nonEmpty =>
+      case scan: FileSourceScanExec if metadataCols(scan).nonEmpty =>


Maybe we can move it under the data source V1 branch?

hsiang-c · 2026-07-09T19:44:39Z

+
+          // _pos should be 0-indexed per file
+          val result = spark.sql(s"SELECT id, _pos FROM $table ORDER BY id").collect()
+          assert(result(0).getLong(1) == 0L)


Should be all zero according to my local test.

hsiang-c · 2026-07-09T20:02:59Z

+          val positions = result.map(_.getLong(0))
+          // First file has 0,1,2; second file has 0,1 (or vice versa)
+          assert(positions.contains(0L))
+          assert(positions.count(_ == 0L) == 2, "Each file should start _pos at 0")


I got this from Iceberg Java:

+----+ |_pos| +----+ |0 | |0 | |0 | |0 | |0 | +----+

hsiang-c · 2026-07-10T02:44:27Z

@@ -1,5 +1,5 @@
 diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
-index db659dc06b..78f5e3440f 100644
+index db659dc..78f5e34 100644


Nit, we need to use 10 digits hash: git config core.abbrev 10;

hsiang-c · 2026-07-10T02:44:36Z

+     List<SparkPlan> batchScans = SparkPlanUtil.collectBatchScans(plan);
+-    assertThat(batchScans).hasSizeGreaterThan(0).allMatch(SparkPlan::supportsColumnar);
+    // When Comet is enabled, its native scan replaces BatchScanExec nodes entirely
+    assertThat(batchScans).allMatch(SparkPlan::supportsColumnar);


## Which issue does this PR close?  - Partially closes #2607 ## What changes are included in this PR?  - If a projection includes `_spec_id`, which is a constant like the `_file` metadata column for all rows, add it to `RecordBatchTransformerBuilder` ## Are these changes tested?  - Yes, unit tests. - Iceberg Spark: apache/datafusion-comet#4752 Sample plan ```shell == Physical Plan == AdaptiveSparkPlan isFinalPlan=false +- CometSort [id#1247L, data#1248, _spec_id#1249], [id#1247L ASC NULLS FIRST] +- CometExchange rangepartitioning(id#1247L ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=3533] +- CometFilter [id#1247L, data#1248, _spec_id#1249], (id#1247L >= 10) +- CometIcebergNativeScan [id#1247L, data#1248, _spec_id#1249], /var/folders/d2/b93h6k7174ddqxltrxgb51040000gn/T/iceberg_warehouse16657344237870771906/iceberg_data/default/table/metadata/00004-70b616c5-1bbe-4e1d-bdd6-e9ac4910674a.metadata.json, testrest.default.table (branch=null) [filters=id IS NOT NULL, id >= 10, groupedBy=], 1 ```

## Which issue does this PR close?  - Partially closes apache#2607 ## What changes are included in this PR?  - If a projection includes `_spec_id`, which is a constant like the `_file` metadata column for all rows, add it to `RecordBatchTransformerBuilder` ## Are these changes tested?  - Yes, unit tests. - Iceberg Spark: apache/datafusion-comet#4752 Sample plan ```shell == Physical Plan == AdaptiveSparkPlan isFinalPlan=false +- CometSort [id#1247L, data#1248, _spec_id#1249], [id#1247L ASC NULLS FIRST] +- CometExchange rangepartitioning(id#1247L ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=3533] +- CometFilter [id#1247L, data#1248, _spec_id#1249], (id#1247L >= 10) +- CometIcebergNativeScan [id#1247L, data#1248, _spec_id#1249], /var/folders/d2/b93h6k7174ddqxltrxgb51040000gn/T/iceberg_warehouse16657344237870771906/iceberg_data/default/table/metadata/00004-70b616c5-1bbe-4e1d-bdd6-e9ac4910674a.metadata.json, testrest.default.table (branch=null) [filters=id IS NOT NULL, id >= 10, groupedBy=], 1 ```

## Which issue does this PR close?  - Partially closes #2607 ## What changes are included in this PR?  - Support `_pos` metadata column by delegating it to the Parquet reader's `RowNumber` virtual column, inspired by apache/datafusion#22026. - Refactor the `compare_schemas` function b/c `_pos` field id exists in both `source_schema` and `target_schema` and the original implementation might result in `SchemaComparison::NameChangesOnly`, which produces a `RecordBatch` with correct content but incorrect column order. ## Are these changes tested?  - Yes, unit tests. - Iceberg Spark: apache/datafusion-comet#4752 Sample plan ```shell == Physical Plan == AdaptiveSparkPlan isFinalPlan=false +- CometSort [id#863L, data#864, _pos#868L], [id#863L ASC NULLS FIRST] +- CometExchange rangepartitioning(id#863L ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=2634] +- CometFilter [id#863L, data#864, _pos#868L], (id#863L >= 10) +- CometIcebergNativeScan [id#863L, data#864, _pos#868L], file:/var/folders/d2/b93h6k7174ddqxltrxgb51040000gn/T/hive2622216829356851627/table/metadata/00004-95932e2b-bbe6-444e-81b8-069c5e748a20.metadata.json, spark_catalog.default.table (branch=null) [filters=id IS NOT NULL, id >= 10, groupedBy=], 1 ```

hsiang-c · 2026-07-21T05:01:54Z

Spark 4.2 fails probably b/c of #4969

parthchandra · 2026-07-21T14:26:16Z

Spark 4.2 fails probably b/c of #4969

Thanks for the pointer @hsiang-c !

The iceberg runtime jar is needed for RESTCatalogHelper and RESTCatalogAdapter to compile. The Spark 4.2 IncompatibleClassChangeError is a pre-existing issue tracked by apache#4969.

parthchandra force-pushed the iceberg-metadata-columns branch 2 times, most recently from d23bf63 to ec21656 Compare June 29, 2026 20:56

parthchandra changed the title ~~feat: support iceberg metadata columns _file, _partition~~ feat: [iceberg] support iceberg metadata columns _file, _partition Jun 29, 2026

parthchandra mentioned this pull request Jun 30, 2026

feat: add support for _partition metadata column apache/iceberg-rust#2668

Open

parthchandra force-pushed the iceberg-metadata-columns branch 2 times, most recently from 50fb6b3 to 0e34cf1 Compare July 6, 2026 18:23

parthchandra force-pushed the iceberg-metadata-columns branch from cf7b59f to 82d7c43 Compare July 9, 2026 16:54

hsiang-c reviewed Jul 9, 2026

View reviewed changes

hsiang-c reviewed Jul 10, 2026

View reviewed changes

This was referenced Jul 10, 2026

feat: add support for _spec_id metadata column apache/iceberg-rust#2695

Merged

feat: add support for _pos metadata column apache/iceberg-rust#2746

Merged

parthchandra force-pushed the iceberg-metadata-columns branch from e56d51d to 35f615b Compare July 20, 2026 23:43

parthchandra added 9 commits July 21, 2026 09:45

feat: [iceberg] support iceberg metadata columns _file, _partition

e38fca1

add _pos, _spec_id

795bf0a

update cargo.lock

bc0ca08

fix tests

2de6ab9

update diff

0541b31

Update

8b2985f

update cargo.lock

88b9ebb

fix tests

f80f2c9

fix one more test failure

175cd35

parthchandra added 3 commits July 21, 2026 09:45

skip spark 4.2

e72a920

revert spark-4.2 iceberg dep removal — compile failure without it

c153646

The iceberg runtime jar is needed for RESTCatalogHelper and RESTCatalogAdapter to compile. The Spark 4.2 IncompatibleClassChangeError is a pre-existing issue tracked by apache#4969.

update iceberg-rust branch

393754e

parthchandra force-pushed the iceberg-metadata-columns branch from dae0f75 to 393754e Compare July 21, 2026 16:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: [iceberg] support iceberg metadata columns _file, _partition#4752

feat: [iceberg] support iceberg metadata columns _file, _partition#4752
parthchandra wants to merge 12 commits into
apache:mainfrom
parthchandra:iceberg-metadata-columns

parthchandra commented Jun 29, 2026

Uh oh!

hsiang-c Jul 9, 2026 •

edited

Loading

Uh oh!

hsiang-c Jul 9, 2026

Uh oh!

hsiang-c Jul 9, 2026

Uh oh!

hsiang-c Jul 9, 2026

Uh oh!

hsiang-c Jul 9, 2026

Uh oh!

hsiang-c Jul 10, 2026 •

edited

Loading

Uh oh!

hsiang-c Jul 10, 2026

Uh oh!

hsiang-c commented Jul 21, 2026

Uh oh!

parthchandra commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

parthchandra commented Jun 29, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

hsiang-c Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

hsiang-c commented Jul 21, 2026

Uh oh!

parthchandra commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hsiang-c Jul 9, 2026 •

edited

Loading

hsiang-c Jul 10, 2026 •

edited

Loading