feat(vortex-bench): wire SpatialBench into the bench orchestrator#8607
Conversation
HarukiMoriarty
force-pushed
the
nemo/spatial-wire-vx-bench
branch
from
a0218b2 to
fdb0872
Compare
Merging this PR will improve performance by 11.93%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[128] |
273.6 ns | 244.4 ns | +11.93% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing nemo/spatial-wire-vx-bench (c282925) with develop (0a45777)
Footnotes
-
4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
HarukiMoriarty
force-pushed
the
nemo/spatial-wire-vx-bench
branch
from
fdb0872 to
b362551
Compare
DuckDB's GEOMETRY only accepts little-endian (NDR) WKB, but the externally sourced zone table (Overture Maps via spatialbench-cli) is big-endian, so the vortex lane failed spatial queries with "Only little-endian WKB is supported". Re-encode geometry columns to little-endian during the parquet->vortex conversion so the vortex file stores canonical little-endian WKB; columns that are already little-endian pass through without a copy. Also drop the best-effort warn around zone generation so data-generation failures propagate. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
|
|
||
| def targets_from_axes(engine: str, format: str) -> tuple[list[BenchmarkTarget], list[str]]: | ||
| def targets_from_axes( | ||
| engine: str, format: str, benchmark: Benchmark | None = None |
There was a problem hiding this comment.
I don't about this much, but why is this needed?
There was a problem hiding this comment.
To avoid running SpatialBench on engines (DataFusion) that can't execute its spatial queries for now, and fail early and clearly.
There was a problem hiding this comment.
Ideally we can remove this function, though users may need a bit aware of Datafusion and other engine cannot run SpatialBench now? seems not a big problem.
| /// | ||
| /// For SpatialBench (`skip_binary_dict`), the geometry blobs are large and | ||
| /// unique, so the dictionary builder balloons memory (tens of GB) for zero gain. | ||
| fn write_options_for( |
There was a problem hiding this comment.
this is clunky, not sure I have a better way if doing that right now :/
There was a problem hiding this comment.
Yeah, this is a way I thought is clean for now, otherwise people need to wait for like 1 hour of dictionary compact on binary for SF=1.
The zone table's data generation hand-rolled idempotency: an existence check plus a scratch dir cleared with remove_dir_all to survive interrupted runs. Reuse the shared idempotent_async helper instead, which skips when zone_0.parquet exists and writes via a unique temp path + atomic rename, so interrupted runs leave nothing to clean up. Zone is a single part (the CLI runs without --parts), so it is handled as one file. Signed-off-by: Nemo Yu <zyu379@wisc.edu>
4 participants
Summary
Wires SpatialBench into the
vx-bench/bench-orchestratorpipeline so it can be run end-to-end like the other benchmarks (datagen → Parquet → Vortex conversion → query). It builds on the WKB datagen landed in #8598.Running command:
Limitation
DuckDB-only. For now SpatialBench queries use DuckDB-specific ST_* spatial SQL that DataFusion has no functions for yet. There is a a single ad-hoc entry in
BENCHMARK_ENGINES = { SPATIALBENCH: {DUCKDB} }.No dictionary encoding / compaction on the WKB column. WKB geometry blobs are large and effectively unique, so running the dictionary builder over them balloons memory (tens of GB) for zero compression gain. The normal compaction path is preserved for every other column on every other benchmark.
Queries 10, 11, 12 is timeout simply because DuckDB poorly support on Spatial index.
Performance
SF=1.0
SF=3
SF=10