Skip to content

[fix](iceberg) Preserve floating-point partition precision - #70613

Draft
hubgeter wants to merge 1 commit into
apache:masterfrom
hubgeter:fix/iceberg-finite-partition-precision
Draft

hubgeter wants to merge 1 commit into
apache:masterfrom
hubgeter:fix/iceberg-finite-partition-precision

Conversation

@hubgeter

Copy link
Copy Markdown
Contributor

Background

The Iceberg writer serializes partition values as strings for writer grouping, partition paths, and FE commit metadata. The FE parses these strings using each partition field's declared type before building Iceberg PartitionData.

What problem does this PR solve?

Issue Number: None
Related PR: #64315 (special-value handling), #65609 (round-trip floating-point formatting)
PR Type: Bugfix

Problem Summary:

Finite FLOAT and DOUBLE identity partition values were serialized with std::to_string, which retains only six digits after the decimal point. For example, DOUBLE values 1.0000001 and 1.0000002 both became 1.000000, producing incorrect partition metadata and potentially merging distinct partitions into one writer. Subsequent partition filtering can silently omit matching rows.

Use the existing fmt shortest-round-trip formatting for finite partition values while preserving the current NaN and infinity spellings. This fixes the shared serialization used by writer keys, paths, and commit values. Previously written incorrect metadata is not repaired by this change.

Details

  • Frequency: Triggered by dynamic FLOAT/DOUBLE identity partition values that cannot round-trip through fixed six-decimal formatting; no concurrency is required.
  • Root cause: floating_point_partition_value_to_string used std::to_string for finite values, losing precision before writer grouping and FE partition parsing.
  • Why existing tests missed it: Existing partition serialization tests cover NaN and infinities, without finite precision or signed-zero bit comparisons.
  • Fix: Use fmt::format with the original FLOAT/DOUBLE type. Add actual transform unit tests for adjacent values, small values, finite extrema, subnormals, signed zeros, commit/human-string consistency, and distinct partition strings.
  • Other paths and branches: Dynamic writer keys, partition paths, and commit values share this helper. No FE parser or Thrift changes are needed. Full static partition fields use a separate precomputed-value path. Apache branch-4.2 at 0a24ed8 has the identical affected production file and should receive this fix and tests; the complete patch applies cleanly, and no additional prerequisite picks are needed.
  • Behavior change: New writes preserve the original finite floating-point partition values. NaN/infinity handling is unchanged. Existing files with incorrect partition metadata still require separate recovery.

Testing

  • Passed clang-format v16, build-support/check-format.sh, build-support/check-build-hygiene.sh, and git diff --check.
  • Independently compiled helpers extracted from the baseline and modified production source, then parsed their output using Java Float.parseFloat/Double.parseDouble. The original implementation lost finite-value precision in 10,006 FLOAT and 10,208 DOUBLE samples; the modified implementation had zero round-trip failures across 40,034 samples, including special values and signed zeros.
  • The typed fast_float parser used by the new unit tests also preserved all 39,923 finite samples. These isolated experiments do not execute the full Doris transform, writer, or commit path.
  • Added PartitionTransformersTest.test_float_partition_value_round_trip and test_double_partition_value_round_trip. Full BE UT was not run: this worktree has no existing BE build, approximately 4.5 GB of free disk space remains, and run-be-ut.sh builds the complete BE test binary even with a GTest filter.
  • No end-to-end Iceberg SQL regression or clang-tidy run: an isolated built cluster and BE compilation database are not available. No regression golden files were handwritten.
  • Verified that the production fix and unit-test patch apply cleanly to the current branch-4.2 files.

Notes for Reviewers

This remains a draft until the full BE tests and external Iceberg SQL checks are completed. Suggested unit-test command: ./run-be-ut.sh --run --filter='PartitionTransformersTest.*' -j 1. The SQL check should write DOUBLE partitions 1.0000001 and 1.0000002, verify distinct accurate typed partitions in the files metadata, and expect equality/range counts of 1, 1, and 2. Recommend picking this PR to branch-4.2 after merge.

Release note

Fix precision loss when writing finite FLOAT/DOUBLE Iceberg identity partition values, preventing incorrect partition metadata and missing rows during partition filtering for newly written data.

Check List (For Author)

  • Test:
    • Regression test
    • Unit Test (added; full BE test binary not built locally)
    • Manual test (isolated serialization/parser experiments and formatting/hygiene checks described above)
  • Behavior changed: Yes (see Details)
  • Does this need documentation: No
  • Pre-submission status: Partially verified: full BE UT, end-to-end Iceberg SQL regression, and clang-tidy remain pending; keep as Draft.

AI Disclosure

  • Tool: Codex
  • Generated: Code, unit tests, validation probes, and this description.
  • Human review: Pending.

### Background

The Iceberg writer serializes partition values as strings for writer grouping, partition paths, and FE commit metadata. The FE parses these strings using each partition field's declared type before building Iceberg PartitionData.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#64315 (special-value handling), apache#65609 (round-trip floating-point formatting)
PR Type: Bugfix

Problem Summary:

Finite FLOAT and DOUBLE identity partition values were serialized with std::to_string, which retains only six digits after the decimal point. For example, DOUBLE values 1.0000001 and 1.0000002 both became 1.000000, producing incorrect partition metadata and potentially merging distinct partitions into one writer. Subsequent partition filtering can silently omit matching rows.

Use the existing fmt shortest-round-trip formatting for finite partition values while preserving the current NaN and infinity spellings. This fixes the shared serialization used by writer keys, paths, and commit values. Previously written incorrect metadata is not repaired by this change.

### Details

- **Frequency:** Triggered by dynamic FLOAT/DOUBLE identity partition values that cannot round-trip through fixed six-decimal formatting; no concurrency is required.
- **Root cause:** floating_point_partition_value_to_string used std::to_string for finite values, losing precision before writer grouping and FE partition parsing.
- **Why existing tests missed it:** Existing partition serialization tests cover NaN and infinities, without finite precision or signed-zero bit comparisons.
- **Fix:** Use fmt::format with the original FLOAT/DOUBLE type. Add actual transform unit tests for adjacent values, small values, finite extrema, subnormals, signed zeros, commit/human-string consistency, and distinct partition strings.
- **Other paths and branches:** Dynamic writer keys, partition paths, and commit values share this helper. No FE parser or Thrift changes are needed. Full static partition fields use a separate precomputed-value path. Apache branch-4.2 at 0a24ed8 has the identical affected production file and should receive this fix and tests; the complete patch applies cleanly, and no additional prerequisite picks are needed.
- **Behavior change:** New writes preserve the original finite floating-point partition values. NaN/infinity handling is unchanged. Existing files with incorrect partition metadata still require separate recovery.

### Testing

- Passed clang-format v16, build-support/check-format.sh, build-support/check-build-hygiene.sh, and git diff --check.
- Independently compiled helpers extracted from the baseline and modified production source, then parsed their output using Java Float.parseFloat/Double.parseDouble. The original implementation lost finite-value precision in 10,006 FLOAT and 10,208 DOUBLE samples; the modified implementation had zero round-trip failures across 40,034 samples, including special values and signed zeros.
- The typed fast_float parser used by the new unit tests also preserved all 39,923 finite samples. These isolated experiments do not execute the full Doris transform, writer, or commit path.
- Added PartitionTransformersTest.test_float_partition_value_round_trip and test_double_partition_value_round_trip. Full BE UT was not run: this worktree has no existing BE build, approximately 4.5 GB of free disk space remains, and run-be-ut.sh builds the complete BE test binary even with a GTest filter.
- No end-to-end Iceberg SQL regression or clang-tidy run: an isolated built cluster and BE compilation database are not available. No regression golden files were handwritten.
- Verified that the production fix and unit-test patch apply cleanly to the current branch-4.2 files.

### Notes for Reviewers

This remains a draft until the full BE tests and external Iceberg SQL checks are completed. Suggested unit-test command: ./run-be-ut.sh --run --filter='PartitionTransformersTest.*' -j 1. The SQL check should write DOUBLE partitions 1.0000001 and 1.0000002, verify distinct accurate typed partitions in the files metadata, and expect equality/range counts of 1, 1, and 2. Recommend picking this PR to branch-4.2 after merge.

### Release note

Fix precision loss when writing finite FLOAT/DOUBLE Iceberg identity partition values, preventing incorrect partition metadata and missing rows during partition filtering for newly written data.

### Check List (For Author)

- Test:
    - [ ] Regression test
    - [ ] Unit Test (added; full BE test binary not built locally)
    - [x] Manual test (isolated serialization/parser experiments and formatting/hygiene checks described above)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Partially verified: full BE UT, end-to-end Iceberg SQL regression, and clang-tidy remain pending; keep as Draft.

### AI Disclosure

- Tool: Codex
- Generated: Code, unit tests, validation probes, and this description.
- Human review: Pending.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant