feat: Iceberg table format V3: native table decryption, fall back for other V3 features by mbutrovich · Pull Request #4991 · apache/datafusion-comet

mbutrovich · 2026-07-21T14:43:02Z

Which issue does this PR close?

Part of #3756. Closes #.

Rationale for this change

Comet's native Iceberg scan rejected V3 tables. This adds native reads of encrypted Iceberg tables (a V3 feature).

Iceberg-Java unwraps the key envelope on the driver during planning and stores each data file's plaintext data encryption key in the manifest's key_metadata field. So DataFile.keyMetadata() already holds the plaintext key when Comet reads the scan tasks. Comet forwards those bytes to iceberg-rust, which decodes StandardKeyMetadata and uses the key directly for Parquet decryption. No native KMS or JNI is needed (unlike plain-Parquet Modular Encryption in #2447, where the reader unwraps at read time).

What changes are included in this PR?

Bump arrow-rs to 58.4.0 and iceberg-rust to 4532da8 for the encryption read path (feat(encryption) [9/N] Read encrypted parquet data-files iceberg-rust#2584 adds encrypted-data-file reads and FileScanTask::key_metadata; feat(encryption): support AES-256 keys for reading Parquet data files iceberg-rust#2878 adds AES-256 data keys).
proto: optional bytes key_metadata on IcebergFileScanTask (field 6) and IcebergDeleteFile (field 8; 5 to 7 reserved for a separate deletion-vector branch).
serde (CometIcebergNativeScan): forward ContentFile.keyMetadata() for data and delete files. Unset when null.
planner: populate FileScanTask/FileScanTaskDeleteFile key_metadata from the proto.
CometScanRule / IcebergReflection: allow format V3 (reject V4+); fall back for V3 default values, for unsupported column types (allow-list over the full table schema), and for 192-bit data keys (no AES-192-GCM in the underlying crypto; 128- and 256-bit are supported); keep the delete gate Parquet-only so deletion-vector tables still fall back. Table encryption itself is not a fallback reason.
Tests: CometTestKMS (in-memory KeyManagementClient, spark-4.1 source set since KeyManagementClient is public only from Iceberg 1.10 and encryption needs 1.11); spark-hive added as a test dependency to stand up an embedded Hive metastore.
dev/diffs/iceberg/1.11.0.diff: testMissingRequiredWithoutDefault tolerates Comet's exception wrapping.

How are these changes tested?

Rust standard_key_metadata_roundtrips: StandardKeyMetadata encode/decode round-trips (API-stability guard after the bump).
Rust decodes_java_produced_key_metadata: decodes a real Iceberg-Java blob and recovers the identical 16-byte DEK and AAD prefix (cross-language wire-format guard).
Scala CometIcebergEncryptionSuite.data file key_metadata carries a plaintext DEK that round-trips: StandardEncryptionManager stores a plaintext (not wrapped) DEK in key_metadata and the blob round-trips.
Scala CometIcebergEncryptionSuite.encrypted V3 table is read natively and matches Spark: end-to-end. A type=hive Iceberg catalog (embedded Derby metastore) with CometTestKMS creates an encrypted V3 table; the test asserts key_metadata is populated, that results match Spark, and that the read runs through CometIcebergNativeScanExec.
Scala CometIcebergEncryptionSuite.encrypted V3 table with a 256-bit data key is read natively and matches Spark: same, with encryption.data-key-length=32.
Scala CometIcebergEncryptionSuite.encrypted V3 table with a 192-bit data key falls back to Spark: a table with encryption.data-key-length=24 falls back and still returns correct results.
Iceberg's own TestTableEncryption also runs under Comet in the iceberg-spark CI (via dev/diffs/iceberg/1.11.0.diff, which enables native Iceberg scan), reading a real encrypted V3 table.
Ran org.apache.iceberg.spark.sql.TestTableEncryption locally against Iceberg 1.11 with the diff applied (13 tests, all passing) and inspected the captured logs: the INFO org.apache.comet.serde.operator.CometIcebergNativeScan - IcebergScan: ... lines confirm the encrypted data reads (testSelect and testInsertAndDelete) engage CometIcebergNativeScan rather than falling back. Those tests assert exact result sets, so the reads are decrypted correctly through the native path. The remaining fallbacks are metadata-table scans (ALL_DATA_FILES, MANIFESTS, all_manifests), which are unrelated to the data path.

…ey data through proto

…Java's encryption suite locally

hsiang-c · 2026-07-21T20:02:40Z

+  // The native Parquet reader (arrow-rs 58.3) decrypts only 128-bit AES-GCM keys, so a table with a
+  // larger data key must fall back to Spark and still return correct results. Flip this to a native
+  // assertion when arrow-rs 58.4 (256-bit support, arrow-rs#10349) is picked up.
+  test("encrypted V3 table with a 256-bit data key falls back to Spark") {


hsiang-c

LGTM

mbutrovich · 2026-07-21T20:16:19Z

LGTM

Thanks for the review @hsiang-c!

# Conflicts: # native/Cargo.lock # native/Cargo.toml # native/core/src/execution/planner.rs

mbutrovich · 2026-07-22T14:45:43Z

+    # (cache-save) phase runs, and by then the test run has left trees under the
+    # workspace (e.g. the Hive-catalog warehouse from CometIcebergEncryptionSuite)
+    # that make a fresh '**/pom.xml' walk throw "Fail to hash files under
+    # directory". A precomputed string keeps the save phase from re-walking them.


I also added some cleanup logic for the Hive-catalog, but just in case I'm also addressing the footgun here. It fixed a failure in the Post Java test steps:

Post job cleanup. Error: The template is not valid. /home/runner/work/datafusion-comet/datafusion-comet/./.github/actions/java-test/action.yaml (Line: 63, Col: 14): hashFiles('**/pom.xml') failed. Fail to hash files under directory '/home/runner/work/datafusion-comet/datafusion-comet'

that I saw after adding the Hive catalog https://github.com/apache/datafusion-comet/actions/runs/29912607672/job/88905543943

andygrove · 2026-07-23T13:55:37Z

+
+  // Serialized StandardKeyMetadata for an encrypted delete file, from DeleteFile.keyMetadata().
+  // Absent for unencrypted tables. Decoded directly by iceberg-rust; no KMS unwrap needed.
+  // Field 8: fields 5-7 are reserved for the deletion-vector coordinates on the dv-read branch


Should we add an explicit reserved 5, 6, 7 ?

andygrove

LGTM with one suggested change. Thanks @mbutrovich

hsiang-c · 2026-07-23T17:24:45Z

+
+  // A 256-bit (32-byte) AES-GCM data key is supported natively, so the read must run through Comet
+  // and match Spark.
+  test("encrypted V3 table with a 256-bit data key is read natively and matches Spark") {


This updated test LGTM, thanks @mbutrovich

# Conflicts: # native/core/src/execution/planner.rs # spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala

)

mbutrovich added 8 commits July 21, 2026 10:40

preliminary iceberg decryption support: bump iceberg-rest dep, feed k…

5cde1d4

…ey data through proto

add encryption suite to CI

ef554e2

fix clippy

e13c7a4

add end-to-end test using hive catalog

0131329

add 256-bit fallback until arrow 58.4.0 lands

93617ce

format, again

e854424

fix hive warehouse initialization

a0a23e5

fix reflection issue on metadata tables spotted when running Iceberg …

7cf63eb

…Java's encryption suite locally

mbutrovich changed the title ~~feat: [DO NOT MERGE] preliminary Iceberg V3 support: native table decryption, fall back for other V3 features~~ feat: preliminary Iceberg V3 support: native table decryption, fall back for other V3 features Jul 21, 2026

mbutrovich marked this pull request as ready for review July 21, 2026 19:18

update docs.

8638d7b

hsiang-c reviewed Jul 21, 2026

View reviewed changes

Comment thread docs/source/user-guide/latest/iceberg.md Outdated

hsiang-c reviewed Jul 21, 2026

View reviewed changes

Comment thread spark/pom.xml

hsiang-c reviewed Jul 21, 2026

View reviewed changes

Comment thread spark/src/test/spark-4.1/org/apache/comet/CometIcebergEncryptionSuite.scala

hsiang-c approved these changes Jul 21, 2026

View reviewed changes

Address PR feedback.

9a3a994

Merge branch 'main' into iceberg_decryption

2fe0d3c

# Conflicts: # native/Cargo.lock # native/Cargo.toml # native/core/src/execution/planner.rs

mbutrovich self-assigned this Jul 22, 2026

mbutrovich added this to the 1.0.0 milestone Jul 22, 2026

clean up Hive warehouse files, and adjust Maven cache key computation

15f71f8

mbutrovich force-pushed the iceberg_decryption branch from b7ae106 to 15f71f8 Compare July 22, 2026 12:07

mbutrovich commented Jul 22, 2026

View reviewed changes

mbutrovich requested review from andygrove, comphead and parthchandra July 22, 2026 17:54

mbutrovich added 2 commits July 23, 2026 07:40

Merge branch 'main' into iceberg_decryption

61ed2ed

add 256-bit data file key support, bump arrow-rs to 58.4.0

df4d321

mbutrovich changed the title ~~feat: preliminary Iceberg V3 support: native table decryption, fall back for other V3 features~~ feat: Iceberg table format V3: native table decryption, fall back for other V3 features Jul 23, 2026

mbutrovich mentioned this pull request Jul 23, 2026

feat: [DO NOT MERGE] Iceberg table format V3: native deletion vector reads, fall back for other V3 features #4887

Draft

andygrove reviewed Jul 23, 2026

View reviewed changes

andygrove approved these changes Jul 23, 2026

View reviewed changes

mbutrovich added 2 commits July 23, 2026 10:57

Merge branch 'main' into iceberg_decryption

e7b39a8

address PR feedback

0868fba

hsiang-c reviewed Jul 23, 2026

View reviewed changes

mbutrovich added 2 commits July 23, 2026 13:55

Merge branch 'main' into iceberg_decryption

c78e5e8

# Conflicts: # native/core/src/execution/planner.rs # spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala

format

c57619c

mbutrovich merged commit bc53ad5 into apache:main Jul 23, 2026
116 of 118 checks passed

mbutrovich deleted the iceberg_decryption branch July 23, 2026 22:25

mbutrovich mentioned this pull request Jul 24, 2026

fix: apply 1.11 Iceberg diff changes from #4991 to other versions #5020

Merged

mbutrovich added a commit that referenced this pull request Jul 24, 2026

fix: apply 1.11 Iceberg diff changes from #4991 to other versions (#5020

8324fe9

)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Iceberg table format V3: native table decryption, fall back for other V3 features#4991

feat: Iceberg table format V3: native table decryption, fall back for other V3 features#4991
mbutrovich merged 18 commits into
apache:mainfrom
mbutrovich:iceberg_decryption

mbutrovich commented Jul 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

hsiang-c Jul 21, 2026

Uh oh!

Uh oh!

hsiang-c left a comment

Uh oh!

mbutrovich commented Jul 21, 2026

Uh oh!

mbutrovich Jul 22, 2026 •

edited

Loading

Uh oh!

andygrove Jul 23, 2026

Uh oh!

andygrove left a comment

Uh oh!

hsiang-c Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mbutrovich commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

Uh oh!

Uh oh!

hsiang-c Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hsiang-c left a comment

Choose a reason for hiding this comment

Uh oh!

mbutrovich commented Jul 21, 2026

Uh oh!

mbutrovich Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

hsiang-c Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mbutrovich commented Jul 21, 2026 •

edited

Loading

mbutrovich Jul 22, 2026 •

edited

Loading