Repository navigation
Allow for date -> time promotion in v3 - #14266
rambleraptor wants to merge 11 commits into
Conversation
e610f65 to
28dccba
Compare
There was a problem hiding this comment.
I see few things are missing:
- Add partition field validation logic
- Add test for partition field validation
- Add negative test: v2 tables rejecting date→timestamp
- Add negative test: v3 tables rejecting date→timestamptz
- update formatVersion parameter in JavaDoc
- Please update adding integration test
- Add promotion test on TestSchemaUnionByFieldName.java
- add tests for Parquet another data formats that verifies that data written with old types can be read correctly after promotion.
|
One more thing Did you have a chance to run revapi ? @rambleraptor https://iceberg.apache.org/contribute/#checking-for-api-breakages |
28dccba to
89754a4
Compare
|
@talatuyarer revapi is passing |
|
@talatuyarer changes made! I'm unsure what a Parquet test would look like in practice. Doesn't look like we have an existing test that I can work off of. |
nandorKollar
left a comment
There was a problem hiding this comment.
LGTM, I took a similar approach in my abandoned PR.
|
@nandorKollar thanks so much for the review! |
|
Do we account for metrics written as the pre promotion type? I think this code is already part of metrics evaluation but we should test to make sure a promoted field still filters properly |
|
@RussellSpitzer @talatuyarer I just added a metrics test for Parquet, which I think covers the use case you're describing. Let me know if it doesn't. |
f0d5d23 to
a4ba53a
Compare
a4ba53a to
d108715
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
This adds promotion logic for date->timestamp, as well as proper plumbing to ensure that these promotions only occur for Table Format v3.
d108715 to
92d1079
Compare
emkornfield
left a comment
There was a problem hiding this comment.
Sorry a slightly more thorough review this time. It might pay to get a committer to check how much of the nits are worth addressing, as I'm new to the code base.
| : Lists.newArrayList(); | ||
|
|
||
| boolean isBucketPartitioned = | ||
| partitionFields.stream().anyMatch(pf -> pf.transform().toString().startsWith("bucket[")); |
There was a problem hiding this comment.
is there not better utility methods for detecting this in an already parsed form)?
There was a problem hiding this comment.
Not to my knowledge. If somebody else could verify, that would be great.
There was a problem hiding this comment.
I think blacklisting certain transformations here is very brittle. Is bucket transformation the only one which is problematic, which will produce different value after the promotion? I think currently it is. Is it remain the only one in the future? Probably not. How can we make this future-proof, so that transformations added later on don't break this? My fear is that it very easy to forget to change this line here. TBH, I'd simply not allow date promotion, if it is mentioned in partition field. What do you think? Does it sound too strict?
There was a problem hiding this comment.
If we would like to keep the original idea, and filter for bucket transformation here, then how about this: we introduce a new method to Transform interface, with a default implementation, which we can use to tell, if the given transformation allows a certain type promotion request? The default method allows everything, regardless of source and target types, while the bucket transformation will reject date promotion. If this is a breaking change (due to the default method it shouldn't), then we can introduce a new interface for it, with the same check method, and call it only when the given transformation implements it (of course bucket will). How does this sound?
|
|
||
| Preconditions.checkArgument( | ||
| TypeUtil.isPromotionAllowed(field.type(), newType), | ||
| TypeUtil.isPromotionAllowed(field.type(), newType, formatVersion, isBucketPartitioned), |
There was a problem hiding this comment.
I wonder if a more stable API might be to pass through all transforms that apply to the field (I guess bucket is the riskiest one). As long as this isn't public we probably don't need to concern ourselves too much about it.
There was a problem hiding this comment.
Right, I was thinking in the future there could be more which is why I was thinking about the added flexibility.
| return newType.isPrimitiveType() | ||
| && TypeUtil.isPromotionAllowed(newType, existingType.asPrimitiveType()); | ||
| && TypeUtil.isPromotionAllowed( | ||
| newType, existingType.asPrimitiveType(), formatVersion, false); |
There was a problem hiding this comment.
comment on why/what the false literal is here?
There was a problem hiding this comment.
Added comment to name this parameter.
We haven't had a need to plumb through if this field is referenced by source-id and I didn't think it was necessary. Might've been a poor assumption?
…java Co-authored-by: emkornfield <emkornfield@gmail.com>
…java Co-authored-by: emkornfield <emkornfield@gmail.com>
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
Hello! Could I get a review on this? |

Closes #14265
This adds promotion logic for date->timestamp, as well as proper plumbing to ensure that these promotions only occur for Table Format v3