Repository navigation
Write gzip metadata files for write.metadata.compression-codec - #4398
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
Metastore catalogs wrote table metadata as .metadata.json even when write.metadata.compression-codec was gzip. Name new metadata files .gz.metadata.json for gzip, as Java does. None stays the default, and other values raise only when metadata is written, so no-op commits on existing tables keep working. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
write.metadata.compression-codechas no effect on the metadata file name. Withwrite.metadata.compression-codec=gzip, PyIceberg still writes00000-<uuid>.metadata.json, while Java writes00000-<uuid>.gz.metadata.jsonand gzips the file. The gzip reader and writer already choose gzip from the.gz.metadata.jsonsuffix (added in apache/iceberg#7984), so only the file name is missing.The Java reference is
BaseMetastoreTableOperations.newTableMetadataFilePath. It readswrite.metadata.compression-codec(defaultnone) from the metadata being written, andTableMetadataParser.getFileExtensionreturns.gz.metadata.jsonfor gzip and.metadata.jsonfor none.Codec.fromNameaccepts the names case-insensitively and throws for any other value.This PR does the same in
LocationProvider.new_table_metadata_file_location, the one place PyIceberg builds table metadata file names. The codec is read from the table properties.none(the default) keeps.metadata.json,gzipgives.gz.metadata.json, and any other value raisesValueErrorwhen a metadata file is written. The metastore catalogs use this for every metadata file they create or commit: SQL, in-memory, Hive, Glue, DynamoDB and BigQuery on create, and SQL, in-memory, Hive and Glue on commit (DynamoDB and BigQuery do not implement commit yet). The codec comes from the properties of the metadata being written, so a commit that sets it applies to the file that commit writes. The newTableProperties.METADATA_COMPRESSIONconstant is documented inmkdocs/docs/configuration.md.REST is unchanged. The REST catalog does not write table metadata files; the server picks the metadata location, so the client does not change the file name.
Are these changes tested?
Yes. Four new tests in
tests/catalog/test_catalog_behaviors.pyrun on the in-memory and SQL catalogs, each with the three identifier shapes:test_table_writes_gzip_metadata_when_compression_codec_is_set: the file name ends with.gz.metadata.jsonand the content starts with the gzip magic bytes. A later append keeps the codec and the version parses from the gzip name (00001-), andload_tablereturns the same metadata.test_table_metadata_compression_codec_default_and_set_on_commit: a table with the default still writes.metadata.json. A commit that setsgzipmakes the next metadata file a.gz.metadata.json.test_table_metadata_invalid_compression_codec_raises:snappyraisesValueErroron create, and no table is created. On commit it raises and the table is unchanged.test_table_metadata_legacy_invalid_compression_codec_on_commit: a table registered withsnappykeeps its metadata location on a commit with no changes. A commit that changes something raisesValueErrorand leaves the table unchanged, and removing the property commits normally.tests/table/test_locations.pyaddstest_new_table_metadata_file_location_compression_codec(none,NONE,gzipandGZIP) andtest_new_table_metadata_file_location_invalid_compression_codec.The added catalog tests cover in-memory, SQL, and SQL without row-count support. Hive, Glue, DynamoDB and BigQuery share the filename generator; this PR adds no codec-specific tests to their existing unit suites, which use mocked services.
These fail on main. Run against main with only the new
TablePropertiesconstant added, the gzip and default tests fail withAssertionError(the file name does not end with.gz.metadata.json), and the invalid-codec and legacy-codec tests fail withDID NOT RAISE ValueError. They pass with this change. The unit suite has no new failures.Are there any user-facing changes?
Yes. With
write.metadata.compression-codec=gzip, new table metadata files written by the SQL, in-memory, Hive, Glue, DynamoDB and BigQuery catalogs are named*.gz.metadata.jsonand gzipped. The default (none) does not change. Any other value now raisesValueErrorwhen PyIceberg writes a metadata file, so a commit that sets an unsupported value fails. For a table that already carries an unsupported value, remove the property or set it tononeorgzipin the next transaction; that corrective transaction can commit normally.Please add the
changeloglabel.This pull request and its description were written by Isaac.