Skip to content

AddFiles: Wire a destination-table key through the registration chain - #40817

Open
claudevdm wants to merge 2 commits into
apache:masterfrom
claudevdm:addfiles-dynamic-dest3
Open

claudevdm wants to merge 2 commits into
apache:masterfrom
claudevdm:addfiles-dynamic-dest3

Conversation

@claudevdm

Copy link
Copy Markdown
Collaborator

Threads a destination-table key through the AddFiles registration chain, holding the key
constant at the configured table.

stage before after
ReadFooterSchema DoFn<String, SchemaGroup> DoFn<KV<String,String>, KV<String,SchemaGroup>>
ConvertToDataFile DoFn<String, SerializableDataFile> DoFn<KV<String,String>, KV<String,SerializableDataFile>>
CreateManifests keyed by specId keyed by KV<table, specId>
CommitManifestFilesDoFn identifier field batch.getKey()

AddFiles.expand supplies the constant with
WithKeys.<String,String>of(tableIdentifier), and distinctSchemas drops the key again so
its global combine and the commit path downstream are untouched.

Reviewable as six steps, in this order:

  1. key the footer read, add the fileSchemas helper
  2. key its two test helpers
  3. the converter's cached table becomes a ConcurrentMap<String, TableState>, name still constant
  4. key the converter; identifier and its constructor parameters are deleted
  5. the manifest key becomes KV<table, specId>
  6. the committer reads its table from the key

@github-actions github-actions Bot added java io iceberg Iceberg IO connector size/xl XL: >=1000 lines changed labels Oct 26, 2026
@claudevdm
claudevdm force-pushed the addfiles-dynamic-dest3 branch from cea3aea to 782c0c4 Compare October 27, 2026 01:40
@github-actions github-actions Bot added size/m M: 50-249 lines changed and removed size/xl XL: >=1000 lines changed labels Oct 27, 2026
@claudevdm
claudevdm marked this pull request as ready for review October 27, 2026 01:45
@claudevdm
claudevdm requested a review from ahmedabu98 October 27, 2026 01:45
@claudevdm

Copy link
Copy Markdown
Collaborator Author

R: @ahmedabu98

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR cleanly threads a destination-table key (String) through the AddFiles registration pipeline (ReadFooterSchema, ConvertToDataFile, CreateManifests, and CommitManifestFilesDoFn) in preparation for dynamic table destinations. The refactoring preserves existing single-table behavior by attaching a constant table key via WithKeys and dropping it before the global schema combination step.

🔍 General Feedback

  • Thread-safety in ConvertToDataFile is well-maintained by transitioning the single-table state fields into a per-table TableState object managed in a ConcurrentMap, with synchronization and refresh flags scoped per table.
  • Coders and key types are properly configured across all modified PCollection stages and unit test helpers.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

iceberg Iceberg IO connector io java Next Action: Reviewers size/m M: 50-249 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant