Repository navigation
chore: prepare for DataFusion 56 - #99
Rich-T-kid wants to merge 2 commits into
Conversation
|
@gabotechs @NoahKusaba @andygrove this is ready for review when you have a chance. It updates the integration to DataFusion 56 and Arrow 60. |
|
Thanks @Rich-T-kid. Should we leave this open until the release is available? |
yeah, sounds good to me |
Update the DataFusion and Arrow dependency graph together, adapt the projection APIs, and carry the current iceberg-rust partition builder API. Refresh plan and result expectations for DataFusion 56.
Update to the current apache/iceberg-rust#3257 head after its rebase and adapt to the latest partition spec API.
6ee8051 to
a88f6cb
Compare
alexanderbianchi
left a comment
There was a problem hiding this comment.
+1 to leaving it open for now, but LGTM once it's time.
|
@CCTY regarding #108 we are going to be blocked on updating the iceberg-rust dep to include apache/iceberg-rust#3704 on bumping this crates arrow library to version 60 which to my understanding is blocked on datafusion 56 which is going to take a few weeks. I am wondering if merging this PR is reasonable in the interim, build off the latest datafusion + iceberg_rust for now while we wait on the official datafusion 56. |
Updates the project to DataFusion 56 and Arrow 60. DataFusion is pinned to the current
branch-56commit, and the Iceberg dependency follows apache/iceberg-rust#3257 until compatible releases are available.The code changes adapt
TableProviderprojections to slices, update partition-spec construction for the current iceberg-rust API, and refresh the affected plan and result snapshots.