Skip to content

Raincloud conformance oracle: add repeated/list column support (FixedSizeList, List) #261

Closed

Description

@dfa1

Summary

RaincloudConformanceIntegrationTest's Parquet oracle (writeOracleCsv/oracleCell) aborts rather than formats any column with maxRepetitionLevel > 0 — i.e. any LIST/repeated Parquet column, which is how Vortex FixedSizeList and List columns round-trip through Parquet. This blocks conformance verification for any slug that stores columns as lists, even though vortex-java's own CsvExporter now renders them correctly (#257).

Affected slugs (currently gap:257 in the matrix, but that issue is closed — the real blocker is this one)

  • glove-6b-50d, glove-6b-100d, glove-6b-200d — vector column is FixedSizeList[F32, n]
  • osm-germany-relations — members/tags columns are List[Struct]

Confirmed via

Flipping glove-6b-50d to ok and running the conformance test (after #259 fixed the harness hang) reproduces:

TestAbortedException: oracle cannot format repeated list column: vector

from writeOracleCsv (RaincloudConformanceIntegrationTest.java).

Expected behavior

The oracle should recursively read repeated/nested Parquet groups (hardwood's RowReader/ColumnSchema repetition/definition levels) and render them as JSON arrays with the same rules CsvExporter.jsonValue/jsonArray already use, so the two sides can be compared value-for-value:

  • FixedSizeList[F32, 50] → [0.1,0.2,...,0.05]
  • List[Struct] → [{"ref":1,"role":""},...]

Raincloud matrix

Flip the 4 slugs above from gap:257 to gap:<this issue> in the same change that adds this test infra, then to ok once oracle output is verified to actually match CsvExporter's output.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions