Skip to content

Allow for date -> time promotion in v3 - #14266

Closed
rambleraptor wants to merge 11 commits into
apache:mainfrom
rambleraptor:date_time_promotion
Closed

rambleraptor wants to merge 11 commits into
apache:mainfrom
rambleraptor:date_time_promotion

Conversation

@rambleraptor

Copy link
Copy Markdown
Contributor

Closes #14265

This adds promotion logic for date->timestamp, as well as proper plumbing to ensure that these promotions only occur for Table Format v3

@talatuyarer talatuyarer left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see few things are missing:

  • Add partition field validation logic
  • Add test for partition field validation
  • Add negative test: v2 tables rejecting date→timestamp
  • Add negative test: v3 tables rejecting date→timestamptz
  • update formatVersion parameter in JavaDoc
  • Please update adding integration test
  • Add promotion test on TestSchemaUnionByFieldName.java
  • add tests for Parquet another data formats that verifies that data written with old types can be read correctly after promotion.

@talatuyarer

Copy link
Copy Markdown
Contributor

One more thing Did you have a chance to run revapi ? @rambleraptor

https://iceberg.apache.org/contribute/#checking-for-api-breakages

@rambleraptor

Copy link
Copy Markdown
Contributor Author

@talatuyarer revapi is passing

@rambleraptor

Copy link
Copy Markdown
Contributor Author

@talatuyarer changes made! I'm unsure what a Parquet test would look like in practice. Doesn't look like we have an existing test that I can work off of.

@nandorKollar nandorKollar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I took a similar approach in my abandoned PR.

@rambleraptor

Copy link
Copy Markdown
Contributor Author

@nandorKollar thanks so much for the review!

@RussellSpitzer

Copy link
Copy Markdown
Member

Do we account for metrics written as the pre promotion type? I think this code is already part of metrics evaluation but we should test to make sure a promoted field still filters properly

@rambleraptor

Copy link
Copy Markdown
Contributor Author

@RussellSpitzer @talatuyarer I just added a metrics test for Parquet, which I think covers the use case you're describing. Let me know if it doesn't.

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Nov 13, 2025
This adds promotion logic for date->timestamp, as well as proper
plumbing to ensure that these promotions only occur for Table Format v3.

@emkornfield emkornfield left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry a slightly more thorough review this time. It might pay to get a committer to check how much of the nits are worth addressing, as I'm new to the code base.

: Lists.newArrayList();

boolean isBucketPartitioned =
partitionFields.stream().anyMatch(pf -> pf.transform().toString().startsWith("bucket["));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there not better utility methods for detecting this in an already parsed form)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not to my knowledge. If somebody else could verify, that would be great.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think blacklisting certain transformations here is very brittle. Is bucket transformation the only one which is problematic, which will produce different value after the promotion? I think currently it is. Is it remain the only one in the future? Probably not. How can we make this future-proof, so that transformations added later on don't break this? My fear is that it very easy to forget to change this line here. TBH, I'd simply not allow date promotion, if it is mentioned in partition field. What do you think? Does it sound too strict?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we would like to keep the original idea, and filter for bucket transformation here, then how about this: we introduce a new method to Transform interface, with a default implementation, which we can use to tell, if the given transformation allows a certain type promotion request? The default method allows everything, regardless of source and target types, while the bucket transformation will reject date promotion. If this is a breaking change (due to the default method it shouldn't), then we can introduce a new interface for it, with the same check method, and call it only when the given transformation implements it (of course bucket will). How does this sound?


Preconditions.checkArgument(
TypeUtil.isPromotionAllowed(field.type(), newType),
TypeUtil.isPromotionAllowed(field.type(), newType, formatVersion, isBucketPartitioned),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if a more stable API might be to pass through all transforms that apply to the field (I guess bucket is the riskiest one). As long as this isn't public we probably don't need to concern ourselves too much about it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bucket is actually called out specifically in the spec for this situation, which is why we need to look just for that one.
Screenshot 2025-11-19 at 12 08 29 PM

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I was thinking in the future there could be more which is why I was thinking about the added flexibility.

return newType.isPrimitiveType()
&& TypeUtil.isPromotionAllowed(newType, existingType.asPrimitiveType());
&& TypeUtil.isPromotionAllowed(
newType, existingType.asPrimitiveType(), formatVersion, false);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment on why/what the false literal is here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comment to name this parameter.

We haven't had a need to plumb through if this field is referenced by source-id and I didn't think it was necessary. Might've been a poor assumption?

@github-actions github-actions Bot removed the stale label Nov 19, 2025
@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Dec 20, 2025
@github-actions

Copy link
Copy Markdown

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Dec 27, 2025
@rambleraptor

Copy link
Copy Markdown
Contributor Author

Hello! Could I get a review on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow for v3 date -> time promotion

6 participants