Use PiecewiseSequence indices for FSL take#8800
Conversation
Merging this PR will regress 1 benchmark
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
aba7e4d to
3edf6c8
Compare
07fde63 to
3ecd955
Compare
3edf6c8 to
1748e31
Compare
3ecd955 to
98a9180
Compare
01e95cb to
eb83451
Compare
eb83451 to
e79ec4b
Compare
robert3005
left a comment
There was a problem hiding this comment.
I think we have to audit the validation from the compute and verify that we do not do the work twice. We have to audit exactly but ideally we'd not validate and figure out how to make operations safe by default even given bad input.
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
6eb0c92 to
16ca5dc
Compare
b615895 to
6dcc52a
Compare
Signed-off-by: Daniel King <dan@spiraldb.com>
6dcc52a to
a2cbdf0
Compare
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
a2cbdf0 to
d694338
Compare
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
89e4da2 to
2a33748
Compare
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
|
I cannot replicate the two benchmark slowdowns locally. |
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
|
The copy_ benchmark slowdowns are resolved by #8832 |
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
| let validity = array.validity()?.take(indices_ref)?; | ||
| let output_len = indices_ref.len(); | ||
| let taken = match lengths { | ||
| ConstantOrArray::Constant(length) => { |
There was a problem hiding this comment.
we have a thing that we call Columnar
| indices_ref: &ArrayRef, | ||
| ctx: &mut ExecutionCtx, | ||
| ) -> VortexResult<Option<ArrayRef>> { | ||
| let Some((starts, lengths)) = maybe_contiguous_slices(indices, ctx)? else { |
There was a problem hiding this comment.
Maybe minor but in this method you double validate if this succeeds
| check_index_arrays(array.starts(), array.lengths(), array.multipliers())?; | ||
| let starts = array.starts().clone().execute::<PrimitiveArray>(ctx)?; | ||
| if let Some(length) = constant_unsigned_usize(array.lengths())? { | ||
| check_index_arrays(starts.as_ref(), array.lengths(), array.multipliers())?; | ||
| return Ok(Some((starts, ConstantOrArray::Constant(length)))); | ||
| } | ||
|
|
||
| let lengths = array.lengths().clone().execute::<PrimitiveArray>(ctx)?; | ||
| check_index_arrays(starts.as_ref(), lengths.as_ref(), array.multipliers())?; |
There was a problem hiding this comment.
I think I miss something - why we validate so hard here?
| check_index_arrays(array.starts(), array.lengths(), array.multipliers())?; | ||
| let starts = array.starts().clone().execute::<PrimitiveArray>(ctx)?; | ||
| if let Some(length) = constant_unsigned_usize(array.lengths())? { | ||
| check_index_arrays(starts.as_ref(), array.lengths(), array.multipliers())?; | ||
| return Ok(Some((starts, ConstantOrArray::Constant(length)))); | ||
| } | ||
|
|
||
| let lengths = array.lengths().clone().execute::<PrimitiveArray>(ctx)?; | ||
| check_index_arrays(starts.as_ref(), lengths.as_ref(), array.multipliers())?; |
There was a problem hiding this comment.
I think we don't want to serialise this array, this is purely optimisation for list takes
Summary
Ports the TakeSlices-style optimization to
Take(elements, PiecewiseSequenceArray)instead of introducing a value-wrapper API.1, then callselements.take(piecewise_indices).PiecewiseSequenceArrayindices and gather contiguous ranges only whenmultipliersis a constant unsigned1; non-unit multipliers fall back through normal materialized-index take.start + j * multipliersequences, while the current optimized consumers stay explicitly contiguous.Validation
cargo fmt -p vortex-arraycargo test -p vortex-array piecewise_sequencecargo check -p vortex-arraycargo check -p vortex-array --benchesgit diff --check