Skip to content

fix: sanitize Avro field names on write, respect iceberg-field-name on read - #2540

Closed
SreeramGarlapati wants to merge 3 commits into
apache:mainfrom
SreeramGarlapati:fix/avro-field-name-sanitization
Closed

SreeramGarlapati wants to merge 3 commits into
apache:mainfrom
SreeramGarlapati:fix/avro-field-name-sanitization

Conversation

@SreeramGarlapati

@SreeramGarlapati SreeramGarlapati commented May 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Iceberg field names can be anything (123column, field.with.dots, etc.) but Avro requires names to match [A-Za-z_][A-Za-z0-9_]*. Java sanitizes invalid names on write and records the original in the iceberg-field-name custom Avro property (TypeToSchema, BuildAvroProjection). On read, Java recovers the authoritative name by field-id projection against the table schema, so in Java that property is write-only provenance rather than something it reads back. iceberg-rust was doing neither half of the write contract — it wrote invalid Avro names directly, and on the standalone Avro→Iceberg schema conversion it had no way to recover the original name.

This causes two interop failures:

  1. Write path: iceberg-rust produces invalid Avro when field names have leading digits or special chars → strict parsers (Java, Python) reject the file
  2. Read path: Avro files carrying sanitized names + iceberg-field-name are converted back to the sanitized name (e.g. _123column) instead of the original

Changes

Write path (Iceberg→Avro conversion in SchemaToAvroSchema::field):

  • Added is_valid_avro_name() — checks [A-Za-z_][A-Za-z0-9_]*
  • Added sanitize_avro_name() — follows Java AvroSchemaUtil.sanitize()'s escape scheme, but is intentionally ASCII-strict so output is always a spec-valid Avro name (Java's unicode Character.isLetter/isLetterOrDigit/isDigit can leave names like café/_٠x that violate Avro's ASCII-only grammar):
    • Leading digit: prefix _ (e.g., 123col → _123col)
    • Anything outside ASCII alnum/_: _x<HEX> (e.g., field.name → field_x2Ename)
    • Operates on UTF-16 code units to match Java's charAt() for supplementary chars
  • When sanitization is needed: stores original name in iceberg-field-name property

Read path (Avro→Iceberg conversion in AvroSchemaToSchema::record):

  • Checks iceberg-field-name custom attribute first, falls back to the Avro field name. Note this is deliberately more faithful than Java's SchemaToType, which uses field.name() directly; iceberg-rust restores the original on the standalone conversion path too, while staying field-id-based for real table reads.

Java reference

Closes #2535

Test plan

  • test_is_valid_avro_name — validates detection of invalid names
  • test_sanitize_avro_name — ASCII edge cases (match Java for the chars both escape)
  • test_sanitize_avro_name_unicode — BMP and supplementary char handling (surrogate pairs)
  • test_sanitization_round_trip — Iceberg→Avro→Iceberg preserves original names
  • test_avro_to_iceberg_uses_iceberg_field_name_property — reads sanitized schemas back to the original name
  • All existing tests pass (no regressions in bidirectional schema conversion tests)

@xanderbailey xanderbailey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice PR, thanks for the contribution, left a couple of questions

// This const may better to maintain in avro-rs.
const LOGICAL_TYPE: &str = "logicalType";

fn is_valid_avro_name(name: &str) -> bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if we could use something from https://github.com/apache/avro-rs/blob/4edb1ce1ae1ab5bd3fafb08ca3f622946c01c9fd/avro/src/validator.rs#L4 but it looks like validate_record_field_name is pub(crate). Is it work filing an issue upstream to see if we could expose something that would allow us to validate against their implementation?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call. i'd keep the local check regardless though — is_valid_avro_name is the exact Avro grammar [A-Za-z_][A-Za-z0-9_]* in ~6 lines with no coupling to avro-rs internals, and the sanitize half is the iceberg iceberg-field-name convention which avro-rs doesn't cover anyway. happy to file an upstream issue to expose validate_record_field_name as good-citizen cleanup and link it here — lmk if you'd like that.

Comment on lines +105 to +107
fn is_ascii_digit_u16(c: u16) -> bool {
matches!(c, 0x30..=0x39)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

    /**
     * Determines if the specified character is a digit.
     * <p>
     * A character is a digit if its general category type, provided
     * by {@code Character.getType(ch)}, is
     * {@code DECIMAL_DIGIT_NUMBER}.
     * <p>
     * Some Unicode character ranges that contain digits:
     * <ul>
     * <li>{@code '\u005Cu0030'} through {@code '\u005Cu0039'},
     *     ISO-LATIN-1 digits ({@code '0'} through {@code '9'})
     * <li>{@code '\u005Cu0660'} through {@code '\u005Cu0669'},
     *     Arabic-Indic digits
     * <li>{@code '\u005Cu06F0'} through {@code '\u005Cu06F9'},
     *     Extended Arabic-Indic digits
     * <li>{@code '\u005Cu0966'} through {@code '\u005Cu096F'},
     *     Devanagari digits
     * <li>{@code '\u005CuFF10'} through {@code '\u005CuFF19'},
     *     Fullwidth digits
     * </ul>
     *
     * Many other character ranges contain digits as well.
     *
     * <p><b>Note:</b> This method cannot handle <a
     * href="#supplementary"> supplementary characters</a>. To support
     * all Unicode characters, including supplementary characters, use
     * the {@link #isDigit(int)} method.
     *
     * @param   ch   the character to be tested.
     * @return  {@code true} if the character is a digit;
     *          {@code false} otherwise.
     * @see     Character#digit(char, int)
     * @see     Character#forDigit(int, int)
     * @see     Character#getType(char)
     */
    public static boolean isDigit(char ch) {
        return isDigit((int)ch);
    }

https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java#L551
Java's isDigit() actually covers more than just ascii digits. This is a pretty niche edge-case.

Trying to reason about this in my head and I don't think it matters for interop that the representations are identical because both will correctly underscore the field and restore it from the map? Can you check my reasoning here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

your conclusion holds — representational parity with Java isn't needed — but let me correct the reasoning (incl. something I'd have gotten wrong).

this impl is deliberately not matching Java's isDigit. Java's validAvroName/sanitize use the unicode Character.isLetter/isLetterOrDigit/isDigit, so Java keeps non-ASCII letters/digits as-is (café, leading ٠ → _٠, …) — which actually violates Avro's ASCII-only name grammar [A-Za-z_][A-Za-z0-9_]*. matching Java's isDigit here would re-introduce the exact invalid-Avro bug this PR fixes. so we stay ASCII-strict: anything outside ASCII alnum/_ is escaped _x<HEX>, output is always spec-valid, original preserved in iceberg-field-name.

correction to the framing — it's not "both restore from the map". I checked the Java source: iceberg-field-name is write-only in Java (written by TypeToSchema + BuildAvroProjection; no read site). Java's SchemaToType just uses field.name(), and the authoritative name is recovered by field-id projection against the table schema. so correctness doesn't depend on the escape representation either way: within iceberg-rust the round-trip is lossless because we write and read the prop; cross-engine, names resolve by field-id. our read path honoring the prop is actually a touch more faithful than Java's SchemaToType on the standalone conversion path.

net: not changing the digit logic. I did fix the doc, which wrongly claimed it "matches Java semantics" — it follows Java's escape scheme but is intentionally ASCII-stricter.

@SreeramGarlapati
SreeramGarlapati force-pushed the fix/avro-field-name-sanitization branch from 68c1e33 to 6e5c644 Compare June 22, 2026 04:10
SreeramGarlapati and others added 3 commits July 20, 2026 06:34
…ld-name

Iceberg field names can be arbitrary (leading digits, dots, spaces, etc.)
but Avro requires names to match [A-Za-z_][A-Za-z0-9_]*. Java handles
this by sanitizing invalid names on write and storing the original in an
"iceberg-field-name" custom property, then checking that property on read.

iceberg-rust was writing unsanitized names directly, which causes Avro
validation failures (or produces files unreadable by strict Avro parsers)
when field names don't conform to Avro's naming rules.

This adds:
- sanitize_avro_name(): matches Java's AvroSchemaUtil.sanitize() logic
  (prefix _ for leading digits, _x<HEX> for special chars)
- Write path: sanitizes the field name and stores the original in
  iceberg-field-name when sanitization was needed
- Read path: checks iceberg-field-name property first, falls back to
  the Avro field name

Closes apache#2535

Co-authored-by: rawataaryan9 <rawataaryan9@users.noreply.github.com>
…tion

Adds test cases for:
- Non-ASCII BMP characters (U+00E9, U+4E2D)
- Supplementary characters (surrogate pair handling, matching Java's UTF-16)
- Empty string edge case
- Read-path: iceberg-field-name property resolution from Java-written schemas
- Verify iceberg-field-name property is set on dotted field names

Co-authored-by: rawataaryan9 <rawataaryan9@users.noreply.github.com>
The previous doc claimed it "matches Java AvroSchemaUtil.sanitize()
semantics". It does not: Java's validAvroName/sanitize use Unicode
Character.isLetter/isLetterOrDigit/isDigit and can emit names that
violate Avro's ASCII-only grammar (e.g. it leaves `café` and `_٠x`
intact). This impl is ASCII-strict so output is always a spec-valid
Avro name, with the exact original preserved via iceberg-field-name.
Document the divergence and that the escape scheme still matches Java
for any character both sides escape.

Co-authored-by: Sreeram Garlapati <sreeramg@salesforce.com>
@SreeramGarlapati
SreeramGarlapati force-pushed the fix/avro-field-name-sanitization branch from 6e5c644 to 6cd6243 Compare July 20, 2026 13:36
@SreeramGarlapati

Copy link
Copy Markdown
Contributor Author

@xanderbailey rebased onto latest main, ci green. i think both your points are covered above:

  • isDigit: staying ascii-strict on purpose — matching Java's unicode isDigit would re-introduce the exact invalid-avro-name case this pr fixes, and cross-engine correctness is by field-id anyway, so representational parity isn't needed.
  • avro-rs validate_record_field_name: kept the local 6-line check since it's zero-coupling and also covers the iceberg-field-name sanitize half that avro-rs doesn't. happy to file an upstream issue to expose their validator as a good-citizen follow-up if u want it.

anything else blocking here?

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Aug 27, 2026
c-thiel added a commit to lakekeeper/iceberg-rust that referenced this pull request Oct 26, 2026
…not allow

Java writes a partition field whose name is not a valid Avro name under a sanitized name (`event.ts_day` becomes `event_x2Ets_day`, `my-col` becomes `my_x2Dcol`, `1st` becomes `_1st`), with the original name in `iceberg-field-name` and the partition field ID in `field-id`. Spark names the field of `days(event.ts)` `event.ts_day` by default. The manifest reader matched partition fields by name only, so it read every partition value of such a manifest as null, without an error. A rewrite then wrote output files and the surviving entries of the rewritten manifests with null partitions, which other engines prune away for partition filters. This is upstream apache#2536; its fix PR apache#2540 was closed as stale and no other PR is open.

The reader now matches each partition field to the writer's Avro field by field ID, then by `iceberg-field-name`, then by name, as Java projects by field ID. It gives the fields of the partition type the writer's names before it decodes the entries, so the existing match by name finds every value.

`apache-avro` rejects a header whose field names are not `[A-Za-z_][A-Za-z0-9_]*`. Java writes names with non-ASCII letters (`größe`) as they are, and iceberg-rust wrote every partition field name as it is, so the fork could not read those manifests at all, including the ones it wrote itself after one commit. On that error the reader now renames such fields in the header, keeping the original name in `iceberg-field-name`, and parses again. Field names do not change how a record is encoded. This joins the existing header fallback for repeated named type definitions; a header with both problems gets both rewrites.

Review: the field-ID-first match in `writer_partition_type` (spec/manifest/mod.rs), and the retry loop in `rewrite_rejected_header`, which replaces the earlier single fallback and keeps its warning text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
c-thiel added a commit to lakekeeper/iceberg-rust that referenced this pull request Oct 26, 2026
The fork wrote partition field names into the manifest's Avro schema as they are, for example `event.ts_day` for `days(event.ts)` or `my-col`. Such a manifest is not valid Avro: `apache-avro` refuses to parse its header, so strict third-party readers fail on it, and so did the fork itself.

Field names in Avro schemas built from Iceberg types are now made valid with the escapes of Java's `AvroSchemaUtil.sanitize` (`event_x2Ets_day`, `my_x2Dcol`, `_1st`, `a_x20b`), with the original name in `iceberg-field-name` and the field ID unchanged, as Java's `TypeToSchema` writes them. The partition values of manifest entries and of `write_data_files_to_avro` are serialized under the same names. Java keeps non-ASCII letters and digits, which Avro does not allow, so the fork escapes them as well (`größe` becomes `gr_xF6_xDFe`); Java readers find the field by its field ID, and the fork's reader by field ID or `iceberg-field-name`.

Upstream: apache#2536; PR apache#2540 did the schema half of this and was closed as stale.

Review: `with_avro_field_names` must give the names `schema_to_avro_schema` writes, because the entry values are resolved against that schema by name. The round-trip test of the previous commit for a manifest the fork writes is replaced by a test that also checks the header.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JanKaul pushed a commit to lakekeeper/iceberg-rust that referenced this pull request Oct 26, 2026
…not allow

Java writes a partition field whose name is not a valid Avro name under a sanitized name (`event.ts_day` becomes `event_x2Ets_day`, `my-col` becomes `my_x2Dcol`, `1st` becomes `_1st`), with the original name in `iceberg-field-name` and the partition field ID in `field-id`. Spark names the field of `days(event.ts)` `event.ts_day` by default. The manifest reader matched partition fields by name only, so it read every partition value of such a manifest as null, without an error. A rewrite then wrote output files and the surviving entries of the rewritten manifests with null partitions, which other engines prune away for partition filters. This is upstream apache#2536; its fix PR apache#2540 was closed as stale and no other PR is open.

The reader now matches each partition field to the writer's Avro field by field ID, then by `iceberg-field-name`, then by name, as Java projects by field ID. It gives the fields of the partition type the writer's names before it decodes the entries, so the existing match by name finds every value.

`apache-avro` rejects a header whose field names are not `[A-Za-z_][A-Za-z0-9_]*`. Java writes names with non-ASCII letters (`größe`) as they are, and iceberg-rust wrote every partition field name as it is, so the fork could not read those manifests at all, including the ones it wrote itself after one commit. On that error the reader now renames such fields in the header, keeping the original name in `iceberg-field-name`, and parses again. Field names do not change how a record is encoded. This joins the existing header fallback for repeated named type definitions; a header with both problems gets both rewrites.

Review: the field-ID-first match in `writer_partition_type` (spec/manifest/mod.rs), and the retry loop in `rewrite_rejected_header`, which replaces the earlier single fallback and keeps its warning text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JanKaul pushed a commit to lakekeeper/iceberg-rust that referenced this pull request Oct 26, 2026
The fork wrote partition field names into the manifest's Avro schema as they are, for example `event.ts_day` for `days(event.ts)` or `my-col`. Such a manifest is not valid Avro: `apache-avro` refuses to parse its header, so strict third-party readers fail on it, and so did the fork itself.

Field names in Avro schemas built from Iceberg types are now made valid with the escapes of Java's `AvroSchemaUtil.sanitize` (`event_x2Ets_day`, `my_x2Dcol`, `_1st`, `a_x20b`), with the original name in `iceberg-field-name` and the field ID unchanged, as Java's `TypeToSchema` writes them. The partition values of manifest entries and of `write_data_files_to_avro` are serialized under the same names. Java keeps non-ASCII letters and digits, which Avro does not allow, so the fork escapes them as well (`größe` becomes `gr_xF6_xDFe`); Java readers find the field by its field ID, and the fork's reader by field ID or `iceberg-field-name`.

Upstream: apache#2536; PR apache#2540 did the schema half of this and was closed as stale.

Review: `with_avro_field_names` must give the names `schema_to_avro_schema` writes, because the entry values are resolved against that schema by name. The round-trip test of the previous commit for a manifest the fork writes is replaced by a test that also checks the header.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

avro schema writer does not sanitize field names that violate avro naming rules

2 participants