feat: add CalendarIntervalType support#4898
Conversation
|
Nice, clean approach to interval support. I traced the micros-to-nanos conversion through all four paths (both writers, both readers) and they are consistent, and the child-vector A few questions, none blocking:
Also heads up: the red This review was prepared with the assistance of an LLM (Claude). |
…o feat/calendar-interval-type
| _: Float4Vector | _: Float8Vector | _: DecimalVector | _: VarCharVector | | ||
| _: VarBinaryVector | _: DateDayVector | _: TimeStampMicroVector | | ||
| _: TimeStampMicroTZVector => | ||
| _: TimeStampMicroTZVector | _: IntervalMonthDayNanoVector => |
There was a problem hiding this comment.
self reminder: move this to CometLiteral.getSupportLevel after #4715 is merged.
Seems like |
andygrove
left a comment
There was a problem hiding this comment.
LGTM. Thanks @peterxcli. Could you fix conflicts?
Which issue does this PR close?
Part of #4540.
Rationale for this change
Comet does not currently support Spark's
CalendarIntervalType, causing queries carrying calendar interval columns through Comet operators to fall back to Spark.This change adds the foundational Arrow and codegen support needed to preserve calendar intervals—months, days, and microseconds—across the Spark, Arrow, and native execution boundaries.
What changes are included in this PR?
CalendarIntervalTypeto ArrowInterval(MonthDayNano).CALENDAR_INTERVALto the serialized data type protocol and native Arrow type conversion.CalendarIntervalTypesupport to Comet batch-kernel code generation for scalar and nested values.How are these changes tested?
Added coverage for:
CalendarIntervalTypethrough the Arrow writer andCometPlainVector.