Skip to content

Nullable ALP flat columns drop row validity — nulls export as invented values (silent corruption) #210

Closed

Description

@dfa1

Found by the Raincloud conformance harness (#205) on palmer-archipelago-antarctica-penguin-data once #206 unblocked the scan.

bill_length_mm / bill_depth_mm are nullable F64 flat columns encoded as vortex.alp (no vortex.masked wrapper in the tree — layout shows vortex.stats → flat with encodings [vortex.struct, vortex.alp]). The file genuinely carries the nulls: the Python bindings (vortex-data==0.69.0) read rows 3 and 271 back as None. vortex-java exports 32.1 / 13.1 for those rows — the decoded ALP values at the null positions — so validity is dropped somewhere between the array node and the scan output when it is not expressed via vortex.masked.

Every prior ok dataset (uci-iris, uci-seeds, …) is null-free, so this path was never exercised: nullable + ALP + real nulls is the trigger. Likely broader than ALP — any nullable flat column whose validity rides on the array node rather than a masked wrapper. Worth checking how the 0.69 writer serializes validity (array-node validity child?) against what our decoders consume.

Repro: hydrate the slug via Raincloud, vortex-cli export …vortex -, rows 4/272 of the CSV vs the parquet sibling.

Silent corruption — same severity class as #208.

Activity

  1. added 4 commits that reference this issue on Jul 6, 2026
    31ec02f
    0bf97d5
    d4c5d11
    422baeb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions