Skip to content

Write gzip metadata files for write.metadata.compression-codec - #4398

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/metadata-compression-codec
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/metadata-compression-codec

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

write.metadata.compression-codec has no effect on the metadata file name. With write.metadata.compression-codec=gzip, PyIceberg still writes 00000-<uuid>.metadata.json, while Java writes 00000-<uuid>.gz.metadata.json and gzips the file. The gzip reader and writer already choose gzip from the .gz.metadata.json suffix (added in apache/iceberg#7984), so only the file name is missing.

The Java reference is BaseMetastoreTableOperations.newTableMetadataFilePath. It reads write.metadata.compression-codec (default none) from the metadata being written, and TableMetadataParser.getFileExtension returns .gz.metadata.json for gzip and .metadata.json for none. Codec.fromName accepts the names case-insensitively and throws for any other value.

This PR does the same in LocationProvider.new_table_metadata_file_location, the one place PyIceberg builds table metadata file names. The codec is read from the table properties. none (the default) keeps .metadata.json, gzip gives .gz.metadata.json, and any other value raises ValueError when a metadata file is written. The metastore catalogs use this for every metadata file they create or commit: SQL, in-memory, Hive, Glue, DynamoDB and BigQuery on create, and SQL, in-memory, Hive and Glue on commit (DynamoDB and BigQuery do not implement commit yet). The codec comes from the properties of the metadata being written, so a commit that sets it applies to the file that commit writes. The new TableProperties.METADATA_COMPRESSION constant is documented in mkdocs/docs/configuration.md.

from pyiceberg.table import TableProperties

table = catalog.create_table(
    "db.events",
    schema=schema,
    properties={TableProperties.METADATA_COMPRESSION: "gzip"},
)
table.metadata_location  # .../metadata/00000-<uuid>.gz.metadata.json

REST is unchanged. The REST catalog does not write table metadata files; the server picks the metadata location, so the client does not change the file name.

Are these changes tested?

Yes. Four new tests in tests/catalog/test_catalog_behaviors.py run on the in-memory and SQL catalogs, each with the three identifier shapes:

  • test_table_writes_gzip_metadata_when_compression_codec_is_set: the file name ends with .gz.metadata.json and the content starts with the gzip magic bytes. A later append keeps the codec and the version parses from the gzip name (00001-), and load_table returns the same metadata.
  • test_table_metadata_compression_codec_default_and_set_on_commit: a table with the default still writes .metadata.json. A commit that sets gzip makes the next metadata file a .gz.metadata.json.
  • test_table_metadata_invalid_compression_codec_raises: snappy raises ValueError on create, and no table is created. On commit it raises and the table is unchanged.
  • test_table_metadata_legacy_invalid_compression_codec_on_commit: a table registered with snappy keeps its metadata location on a commit with no changes. A commit that changes something raises ValueError and leaves the table unchanged, and removing the property commits normally.

tests/table/test_locations.py adds test_new_table_metadata_file_location_compression_codec (none, NONE, gzip and GZIP) and test_new_table_metadata_file_location_invalid_compression_codec.

The added catalog tests cover in-memory, SQL, and SQL without row-count support. Hive, Glue, DynamoDB and BigQuery share the filename generator; this PR adds no codec-specific tests to their existing unit suites, which use mocked services.

These fail on main. Run against main with only the new TableProperties constant added, the gzip and default tests fail with AssertionError (the file name does not end with .gz.metadata.json), and the invalid-codec and legacy-codec tests fail with DID NOT RAISE ValueError. They pass with this change. The unit suite has no new failures.

Are there any user-facing changes?

Yes. With write.metadata.compression-codec=gzip, new table metadata files written by the SQL, in-memory, Hive, Glue, DynamoDB and BigQuery catalogs are named *.gz.metadata.json and gzipped. The default (none) does not change. Any other value now raises ValueError when PyIceberg writes a metadata file, so a commit that sets an unsupported value fails. For a table that already carries an unsupported value, remove the property or set it to none or gzip in the next transaction; that corrective transaction can commit normally.

Please add the changelog label.

This pull request and its description were written by Isaac.

Metastore catalogs wrote table metadata as .metadata.json even when
write.metadata.compression-codec was gzip. Name new metadata files
.gz.metadata.json for gzip, as Java does. None stays the default, and
other values raise only when metadata is written, so no-op commits on
existing tables keep working.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@geruh, you added write.metadata.path in #1642. Could you review choosing the metadata suffix in the existing location provider? This enables write.metadata.compression-codec=gzip through the existing serializers, and rejects unsupported codec values on metadata writes, as Java does.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant