feat(vortex-bench): infra of SpatialBench on DuckDB, plain binary#8598
Merged
Conversation
HarukiMoriarty
changed the title
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
HarukiMoriarty
force-pushed
the
nemo/spatial-wkb
branch
from
f1bff93 to
dcdae4f
Compare
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | chunked_varbinview_into_canonical[(1000, 10)] |
168.9 µs | 205.7 µs | -17.87% |
| ⚡ | Simulation | chunked_varbinview_canonical_into[(100, 100)] |
259.5 µs | 224.4 µs | +15.64% |
| ⚡ | Simulation | chunked_varbinview_into_canonical[(100, 100)] |
306.6 µs | 271.5 µs | +12.95% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[128] |
273.6 ns | 244.4 ns | +11.93% |
| ⚡ | Simulation | rebuild_naive |
111.1 µs | 99.9 µs | +11.19% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing nemo/spatial-wkb (a9fb661) with develop (797b650)
Footnotes
-
4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
a10y
reviewed
Jun 26, 2026
Contributor
Author
|
Fixed, wire to |
Contributor
Author
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
Tag SpatialBench geometry columns with GeoParquet `geo` file metadata so DuckDB's read_parquet surfaces them as GEOMETRY, and re-type the matching WKB columns as the vortex.geo.wkb (WellKnownBinary) extension on the vortex lane so the vortex scan surfaces them as GEOMETRY too. Queries then reference geometry columns directly, dropping the ST_GeomFromWKB wrappers. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
danking
approved these changes
Jun 30, 2026
danking
left a comment
danking
left a comment
Contributor
There was a problem hiding this comment.
LGTM since the follow up PR will address the review request.
HarukiMoriarty
added a commit
that referenced
this pull request
Jul 2, 2026
) ## Summary <!-- Why are you proposing this change, and what is its impact? Is it part of a long term effort, or a bigger change? If this PR is related to a tracked effort or an open issue, please link to the relevant issue. --> Wires SpatialBench into the `vx-bench` / `bench-orchestrator` pipeline so it can be run end-to-end like the other benchmarks (datagen → Parquet → Vortex conversion → query). It builds on the WKB datagen landed in #8598. Running command: ``` uv run --project bench-orchestrator vx-bench run spatialbench --engine duckdb --format parquet,vortex --opt scale-factor=N --queries 1,2,3,4,5,6,7,8,9 --iterations 3 ``` ## Limitation <!-- No need to duplicate information from the previous section, but if you're touching many parts of the code base, its worth explicitly noting the important changes or how they are tested. --> - DuckDB-only. For now SpatialBench queries use DuckDB-specific ST_* spatial SQL that DataFusion has no functions for yet. There is a a single ad-hoc entry in `BENCHMARK_ENGINES = { SPATIALBENCH: {DUCKDB} }`. - No dictionary encoding / compaction on the WKB column. WKB geometry blobs are large and effectively unique, so running the dictionary builder over them balloons memory (tens of GB) for zero compression gain. The normal compaction path is preserved for every other column on every other benchmark. - Queries 10, 11, 12 is timeout simply because DuckDB poorly support on Spatial index. ## Performance SF=1.0 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ 1 │ 39.1ms │ 14.6ms (0.37x) │ │ 2 │ 72.1ms │ 24.9ms (0.35x) │ │ 3 │ 57.7ms │ 20.1ms (0.35x) │ │ 4 │ 113.1ms │ 70.6ms (0.62x) │ │ 5 │ 354.2ms │ 288.4ms (0.81x) │ │ 6 │ 169.5ms │ 91.6ms (0.54x) │ │ 7 │ 156.7ms │ 71.5ms (0.46x) │ │ 8 │ 196.5ms │ 80.3ms (0.41x) │ │ 9 │ 20.3ms │ 18.7ms (0.92x) │ └───────┴───────────────────────┴─────────────────┘ ``` SF=3 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ 1 │ 50.7ms │ 31.5ms (0.62x) │ │ 2 │ 126.2ms │ 60.3ms (0.48x) │ │ 3 │ 71.9ms │ 42.9ms (0.60x) │ │ 4 │ 539.9ms │ 64.9ms (0.12x) │ │ 5 │ 948.7ms │ 874.5ms (0.92x) │ │ 6 │ 656.2ms │ 121.7ms (0.19x) │ │ 7 │ 256.6ms │ 232.1ms (0.90x) │ │ 8 │ 273.8ms │ 244.6ms (0.89x) │ │ 9 │ 35.7ms │ 27.9ms (0.78x) │ └───────┴───────────────────────┴─────────────────┘ ``` SF=10 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ 1 │ 158.6ms │ 114.4ms (0.72x) │ │ 2 │ 255.1ms │ 219.3ms (0.86x) │ │ 3 │ 229.2ms │ 181.5ms (0.79x) │ │ 4 │ 184.5ms │ 134.3ms (0.73x) │ │ 5 │ 3.30s │ 3.08s (0.93x) │ │ 6 │ 476.4ms │ 348.9ms (0.73x) │ │ 7 │ 918.2ms │ 961.2ms (1.05x) │ │ 8 │ 980.6ms │ 926.7ms (0.94x) │ │ 9 │ 33.7ms │ 33.6ms (1.00x) │ └───────┴───────────────────────┴─────────────────┘ ``` <!-- Are there any user-facing changes that might require documentation updates Is any public API changed? --> --------- Signed-off-by: Nemo Yu <zyu379@wisc.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a SpatialBench benchmark that runs on DuckDB across two storage lanes, Vortex binary BLOB and Parquet binary BLOB.
Benchmark results
Local, warm, median of 5 (ms).
Takeaways
Vortex wins the scan/filter-bound queries (Q1–Q4, Q6) by up to 3.5×. Q5/Q7/Q8/Q9 are compute-bound (DuckDB spatial join dominates), so the formats converge. Q10–Q12 time out on both, because the unindexed spatial join is the wall, independent of storage format.