Skip to content

chore: Update to arrow/parquet 59.1.0#23312

Merged
alamb merged 2 commits into
apache:mainfrom
alamb:alamb/update_arrow_59.1.0
Jul 8, 2026
Merged

chore: Update to arrow/parquet 59.1.0#23312
alamb merged 2 commits into
apache:mainfrom
alamb:alamb/update_arrow_59.1.0

Conversation

@alamb

@alamb alamb commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

(DRAFT until arrow is updated, I am using this PR to pre-test the release)

Which issue does this PR close?

Rationale for this change

Update to latest arrow

What changes are included in this PR?

Are these changes tested?

Yes by CI

Are there any user-facing changes?

No API change (this is a minor update of Arrow)

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates common Related to common crate proto Related to proto crate functions Changes to functions implementation spark auto detected api change Auto detected API change labels Jul 3, 2026
// add an initial FlightData message that sends schema
let options = arrow::ipc::writer::IpcWriteOptions::default();
let mut compression_context = CompressionContext::default();
let mut compression_context = IpcWriteContext::default();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Struct was renamed (old was deprecated) in


for (start, end) in valid_or_empty.set_slices() {
mutable_array_data.extend(
mutable_array_data.try_extend(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required because extend and extend_null was deprecated

(basically now errors are propagated rather than causing panics)

@Jefffrey

Jefffrey commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

@alamb

alamb commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

things that can now progress after this arrow-rs upgrade:

I expect the release to be finalized / approved on Monday at the earliest and then I will mark this PR ready

@Jefffrey

Jefffrey commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

@zhuqi-lucas

Copy link
Copy Markdown
Contributor

Nice — this also pulls in ParquetPushDecoder::peek_next_row_group(). I'll start on #23067 (Per-RG fully_matched RowFilter skip) once this lands!

@alamb alamb force-pushed the alamb/update_arrow_59.1.0 branch from d6e194c to a82451a Compare July 7, 2026 23:11
@alamb alamb marked this pull request as ready for review July 7, 2026 23:13
@alamb

alamb commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Nice — this also pulls in ParquetPushDecoder::peek_next_row_group(). I'll start on #23067 (Per-RG fully_matched RowFilter skip) once this lands!

This PR is now ready for review as arrow 59.1.0 has been released

@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Jul 7, 2026

@zhuqi-lucas zhuqi-lucas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @alamb !

@alamb alamb added this pull request to the merge queue Jul 8, 2026
@alamb

alamb commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for the review @zhuqi-lucas

Merged via the queue into apache:main with commit f34a676 Jul 8, 2026
40 checks passed
@alamb alamb deleted the alamb/update_arrow_59.1.0 branch July 8, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate functions Changes to functions implementation physical-expr Changes to the physical-expr crates proto Related to proto crate spark

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants