Skip to content

feat: [DO NOT MERGE] native Iceberg reads of unshredded Variant columns - #7575

Draft
mbutrovich wants to merge 4 commits into
apache:mainfrom
mbutrovich:iceberg-rust-variant-read
Draft

mbutrovich wants to merge 4 commits into
apache:mainfrom
mbutrovich:iceberg-rust-variant-read

Conversation

@mbutrovich

@mbutrovich mbutrovich commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #4295.

Rationale for this change

This validates apache/iceberg-rust#3683, which adds shredded Variant reads to iceberg-rust, against Comet's Iceberg suites before it merges. apache/iceberg-rust#2880 (unshredded Variant reads), which this PR validated first, has merged.

What changes are included in this PR?

The native dep is temporarily pinned to mbutrovich/iceberg-rust at ae6c7d965572810e6d2273023a1f8eab49c68f13. That commit is the apache/iceberg-rust#3683 head with apache/iceberg-rust#3257 (Arrow 60) reverted, because Comet is still on arrow 59.

  • CometScanRule: allow Variant at any depth in projected Iceberg columns. Empty or unresolved projections still fall back.
  • CometExecRule: let the native Iceberg scan output Variant, as CometScanExec already can.
  • No native change. The scan's schema adapter already converts iceberg-rust's struct<metadata, value> to Spark's struct<value, metadata>.

How are these changes tested?

@mbutrovich mbutrovich self-assigned this Nov 9, 2026
@github-actions github-actions Bot added the area:scan Parquet scan / data reading label Nov 9, 2026
@mbutrovich mbutrovich changed the title [DO NOT MERGE] native Iceberg reads of unshredded Variant columns feat: [DO NOT MERGE] native Iceberg reads of unshredded Variant columns Nov 9, 2026
@mbutrovich
mbutrovich marked this pull request as draft November 9, 2026 18:54
@github-actions github-actions Bot added the enhancement New feature or request label Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg area:scan Parquet scan / data reading enhancement New feature or request run-iceberg-tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant