Skip to content

ConstantEncodingDecoder silently replaces all-null constants with 0/false instead of null #246

Description

@dfa1

Root cause

ConstantEncodingDecoder.arrayFromScalar only special-cases the all-null dtype (DType.Null → NullArray). For a nullable primitive/bool dtype whose constant scalar value is null it falls through to constantPrimitive/decodeBool, which read the absent proto field as 0/false and produce an all-valid constant.

  • decodeBool line ~120: scalar.bool_value() != null && scalar.bool_value() → always false when field is absent
  • constantPrimitive: fills the output buffer with 0L bytes when the scalar number field is missing

Result: an all-null vortex.constant column exports 0.0/0/false instead of all-null — silent corruption.

Affected slugs (round 7 triage)

All confirmed against parquet oracle:

Slug Affected columns
behavioral-risk-factor-surveillance-system TOLDCFS / HAVECFS / WORKCFS (all-null F64)
cardiovascular-diseases-risk-prediction-dataset 11 all-null cols + COLGHOUS null constant chunk at row 393216
diabetes-health-indicators-dataset PAINACT2 / QLMENTL2 / QLSTRES2 / QLHLTH2 (all-null F64)
fhv_tripdata_2025 SR_Flag (all-null I64)
green_tripdata_2025 ehail_fee (all-null F64)
heart-disease-health-indicators-dataset PAINACT2 / QLMENTL2 / QLSTRES2 / QLHLTH2 (all-null F64)
openpowerlifting Squat4Kg etc. (null constant in all-null chunks at rows 655360/1048576)
personal-key-indicators-of-heart-disease TOLDCFS / HAVECFS / WORKCFS (all-null F64)
stackoverflow-tags IsModeratorOnly / IsRequired (all-null bool)

Fix

In arrayFromScalar, before dispatching on dtype, check whether the scalar value itself is null (all proto fields absent). If so, return a MaskedArray wrapping an all-zeros array with an all-invalid validity bitmap — or return a ConstantArray with a null constant marker.

Alternatively, treat it the same as DType.Null and return a NullArray of the right row count (simpler, but loses dtype information for schema-correct output).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions