Union Array

[connortsui20]

Rationale for this change

Tracking issue: #8769

TODO

What changes are included in this PR?

What APIs are changed? Are there any user-facing changes?

---

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

[codspeed-hq]

Merging this PR will improve performance by 25.8%

⚡ 2 improved benchmarks
✅ 1668 untouched benchmarks
⏩ 53 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	copy_nullable[65536]	1.4 ms	1 ms	+32.02%
⚡	Simulation	copy_non_nullable[65536]	1,089.5 µs	908.9 µs	+19.87%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ct/union-array (66e354f) with ct/union-scalar (5e0113d)}

Footnotes

1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5b0a79a5f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Support nullable indices for union take

When a UnionArray is taken with nullable indices (for example a DictArray of union values with nullable codes, since Dict::execute passes those codes through take_canonical), this branch returns an error before preserving the null slots. Nullable take is a supported operation in the crate’s conformance expectations ([Some(0), None, Some(2)] should become [row0, null, row2]), so this makes nullable-coded dictionary unions and outer-join style takes fail. Please handle nullable indices by keeping the output type_ids nullable/masked while gathering children with non-null placeholder indices.

Useful? React with 👍 / 👎.

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	66e354f		1	Explore Profiling Data

Previous Runs (5)

Status	Commit	Job	Attempt	Link
🟢 Done	cfa9d17		1	Explore Profiling Data
🟢 Done	e2186b0		1	Explore Profiling Data
🟢 Done	9238470		1	Explore Profiling Data
🟢 Done	c33685a		1	Explore Profiling Data
🟢 Done	c5b0a79		1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[github-actions]

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +2.5%
Engines: DataFusion No clear signal (+2.3%, low confidence) · DuckDB No clear signal (+2.7%, low confidence)
Vortex (geomean): 0.896x ✅
Parquet (geomean): 0.885x ✅
Shifts: Parquet (control) -11.5% · Median polish -10.5%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

datafusion / vortex-file-compressed (0.841x ✅, 2↑ 0↓)

name	PR 66e354f (ns)	base e6eeaae (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed 🚀	9711199	11446321	0.85
vortex_q01/datafusion:vortex-file-compressed 🚀	6206128	7449028	0.83

datafusion / parquet (0.822x ✅, 2↑ 0↓)

name	PR 66e354f (ns)	base e6eeaae (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet 🚀	19883671	23584739	0.84
vortex_q01/datafusion:parquet 🚀	4500859	5616200	0.80

duckdb / vortex-file-compressed (0.979x ➖, 0↑ 0↓)

name	PR 66e354f (ns)	base e6eeaae (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	9896500	10387695	0.95
vortex_q01/duckdb:vortex-file-compressed	5915306	5885688	1.01

duckdb / parquet (0.952x ➖, 0↑ 0↓)

name	PR 66e354f (ns)	base e6eeaae (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23245745	24576114	0.95
vortex_q01/duckdb:parquet	9325737	9723548	0.96

---

No file size changes detected.