Repository navigation
[fix](be) Find unlisted nested Iceberg equality keys of old-FE plans by delete-file path - #70093
morningman wants to merge 1 commit into
Conversation
…by delete-file path ### What problem does this PR solve? Issue Number: None Related PR: apache#69723, apache#69821 Problem Summary: Since apache#69723 removed the V1 file scanner, FileScannerV2 also reads the plans of an FE that predates the Iceberg scan semantics versions (Doris 4.1.3 and earlier), which a BE upgraded first executes during a rolling upgrade. Such an FE sends no scan semantics version and no initial defaults, and its schema metadata lists only the projected columns. For an equality-delete key that this metadata does not list, the V2 Iceberg reader falls back to the name the delete file gives the key. It kept only the leaf name and looked it up among the data file's top-level columns. In a data file without field ids (an Iceberg table from migrate or add_files), a nested key such as payload.k was never found there and was read as a missing key, NULL in every row: the tombstone payload.k = 7 kept the rows it deletes, and a NULL tombstone deleted every row, both without an error. Before apache#69723 the V1 reader failed this plan (Missing Iceberg schema metadata for equality-delete field id), and a 4.1.3 BE refuses equality deletes on such files. The filter now keeps the delete file's names from the key's top-level column down to the key, and the fallback walks all of them. A present payload without k is still a missing key and reads as NULL. A key renamed after an ID-less file was written stays unresolved, as before: the old FE sends no name mapping for a column it does not list. It also adds the V2 tests for old-FE plans whose equality keys are missing from the data file, which apache#69821 covered on the V1 reader before V1 was removed. They pass before and after this change. ### Release note Fix equality deletes on a nested field of Iceberg data files without field ids being ignored, or deleting every row, when a BE runs a query planned by an FE of 4.1.3 or earlier during a rolling upgrade. ### Check List (For Author) - Test: Unit Test - be/test/format_v2/table/iceberg_reader_test.cpp: IcebergOldFePlanFindsUnlistedNestedEqualityKeyByDeleteFileNames (16 cases) fails without the fix in the 4 cases with an ID-less file and a schema that does not list payload, and passes with it. IcebergOldFePlanMatchesMissingEqualityKeyAsNull (24 cases) and IcebergOldFePlanMatchesMissingNestedEqualityKeyAsNull (8 cases) pass before and after. - IcebergV2ReaderTest, IcebergTableReaderTest, EqualityDeletePredicateTest, IcebergPositionDeleteSysTableV2*, TableReaderTest, ColumnMapper*, LocalColumnIndexTest and CloneTableExprTreeTest: 411 tests pass (macOS, ASAN). - Behavior changed: Yes. For the plan above, the nested key is read from the data file, so its tombstones delete the rows Iceberg deletes. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 28190 ms |
TPC-DS: Total hot run time: 151151 ms |
ClickBench: Total hot run time: 24.39 s |
|
/review |
There was a problem hiding this comment.
Reviewed PR 70093 at head e35bc1f against base a35707d. The change records the complete ancestor-to-leaf name path for each Iceberg equality-delete key and uses it for legacy old-FE BY_NAME resolution, while preserving field-ID mapping, authoritative aliases, missing-key NULL/default handling, and converted-filter cache propagation. I traced the production path through TableReader and ColumnMapper, checked top-level and nested keys, projected-only and historical schema metadata, ID-present and ID-less Parquet/ORC files, alias and missing-ancestor cases, and reviewed the added old-FE test matrix. No actionable defect was substantiated, so there are no new inline findings and no existing P0/P1 findings to carry forward. User review focus supplied no additional points. Review evidence is static only; builds and tests were not executed. Convergence: true after round 1; all assigned review scopes returned no valuable findings and the final live-head/diff/comment sweep is complete.
Review converged after 1 round(s).
|
Closing this PR because we no longer intend to support scan plans generated by old FEs. The substantive fix extends the fallback guarded by The two missing-key NULL test matrices have some independent coverage value, but they currently construct old-FE plans. That coverage can be added separately using current scan semantics, complete key/ancestor schema metadata, and appropriate name mappings for ID-less files. It does not require retaining this compatibility fix. |
…a files (#70226) ### Background An Iceberg equality delete removes the data rows whose key columns equal one of its rows, its tombstones. A table can add a column after some data files were written and then use it as a key; those files don't store it, so Iceberg reads it there as the column's initial default, or NULL when it has none, and NULL equals only a NULL tombstone. The BE's V2 Iceberg reader builds such a missing key as a literal from the schema metadata that the FE sends with the scan, and matches a data file without field ids, written before `migrate` or `add_files`, through the table's name mapping. ### What problem does this PR solve? Issue Number: None Related PR: #70093 (an earlier version of these tests, built on old-FE plans; closed) PR Type: Test Problem Summary: A missing key without an initial default must read as NULL and match NULL tombstones only; master tests that on Parquet alone, with a single key, almost always in files with field ids. This PR adds two matrices, for a top-level and a nested missing key, on the scan plan the current FE sends, across ORC and Parquet, files with and without field ids, and composite and single-key deletes. It also fixes the ORC fixture writer, which stored child values under NULL parents and so shifted later rows. Only test code changes. ### Details - **Change map:** 338 added lines, all in `be/test/format_v2/table/iceberg_reader_test.cpp`: the two matrices (218), helpers that build the current FE's scan plan and name mappings (41), a writer for the top-level key's delete files (73), and the ORC fixture fix (6). - **Scenarios:** `IcebergMissingEqualityKeyMatchesOnlyNullTombstones`, 24 cases: a file with `id` = 1, 2, 3, written before the table added the optional key `added_key` (INT or STRING, no initial default), in Parquet or ORC, with or without field ids. The delete (id, added_key) = (2, NULL), (3, 0 or '') removes row 2 only; a single NULL tombstone removes every row, with a predicate of NULL literals alone; a single 0 or '' tombstone removes nothing. `IcebergMissingNestedEqualityKeyMatchesOnlyNullTombstones`, 8 cases: the key `payload.k`, added after a file that stores `id` and a nullable `payload {existing}`, NULL in row 2, in Parquet or ORC, with or without field ids, the query reading `id` only or `payload` too. The tombstones (1, NULL payload) and (3, 0) leave rows 2 and 3, and a projected `payload` keeps its values. - **What existing tests miss:** The closest tests use the missing key as the only key, against a NULL tombstone on Parquet only (`IcebergEqualityDeleteMatchesNullForMissingDataColumn`, `IcebergEqualityDeleteMatchesNullMissingKeyUnderUnknownTableType`) or against tombstones far from a NULL slot's value, such as 7 (`IcebergEqualityDeleteMissingKeyDoesNotReadUnsupportedUnprojectedCarrier`, `IcebergMissingWholeStructEqualityKeyMaterializesDefault`). None matches a NULL tombstone in ORC, uses a STRING key, pairs the missing key with a present one, or deletes with a tombstone equal to a NULL slot's value; a key missing under a present ancestor is tested only with a non-NULL default (`IcebergMissingNestedEqualityKeyPreservesNullableParent`). - **Fails when broken:** Two mutations, each run against all of `IcebergV2ReaderTest`. Reading a missing key without a default as its type's default, 0 or '', instead of NULL (`build_missing_equality_delete_key_expr`) fails all 32 new cases. Letting a NULL data key also match, and hash as, the value its nested column holds (`EqualityDeletePredicate`) fails the 24 new cases with a non-NULL tombstone, while the 8 single NULL-tombstone cases pass, as they should. The two mutations also fail 3 and 5 existing tests. ### Testing - `run-be-ut.sh --run --filter='IcebergV2ReaderTest.*:IcebergTableReaderTest.*'`, locally on macOS (arm64, ASAN build) with `be/src/format_v2` and `be/test/format_v2` as on this PR's head: 129 tests pass, the 2 new ones included. ### Notes for Reviewers - `write_nested_equality_orc_file` now marks a child slot NULL under a NULL parent. The ORC writer stores a child's PRESENT bits only for rows whose parent is present but its values for every row the child batch marks present, so `existing` = {10, 20, 30} under a NULL row 2 read back as 10, NULL, 20. The projected-`payload` case exposed it; the helper's other callers never read a child value after a NULL parent and pass unchanged. - The cases without field ids carry an authoritative name mapping on every field, as `migrate` and `add_files` set it and the current FE sends it: under scan semantics v2 such a file is read by name only through one (`IcebergTableReader::_mapping_mode`).
What problem does this PR solve?
Issue Number: None
Related PR: #69723, #69821
Problem Summary:
In short
Since #69723 removed the V1 file scanner, FileScannerV2 also reads the plans of an FE that predates the Iceberg scan semantics versions (Doris 4.1.3 and earlier), which a BE upgraded first executes during a rolling upgrade. For such a plan, the V2 Iceberg reader ignored an equality delete keyed on a nested field of a data file without field ids, or made it delete every row of the file, whenever the query did not read the field's parent struct. Neither case raised an error. This PR resolves such a key by its full path in the delete file. It also adds the V2 tests for old-FE plans whose equality keys are missing from the data file, which #69821 covered on the V1 reader before V1 was removed.
Background
An FE of 4.1.3 or earlier plans an Iceberg scan without
iceberg_scan_semantics_versionand without initial defaults. Its schema metadata (current_schema_id = -1, a single schema-1) lists only the projected columns, or every column when Top-N lazy materialization needs them, and marks every field optional. An equality-delete key is a hidden dependency of the scan, so the old FE usually does not list it.The V2 reader resolves an equality key through that schema metadata. For a key that an old FE does not list, it falls back to the name the delete file gives the key (
IcebergTableReader::_find_equality_delete_data_field). In a data file with field ids the key is found by its field id at any depth. In a data file without them, as in an Iceberg table created bymigrateoradd_files, it is looked up by name.The problem, and what it cost
The fallback kept only the key's leaf name and looked it up among the data file's top-level columns. A nested key was therefore never found and was read as a missing key: NULL in every row of the file.
Take an ID-less data file
id, payload STRUCT<k INT>holding(1, {5}),(2, {7})and(3, NULL), an equality delete onpayload.k, and an old-FE scan that reads onlyid:payload.k77,NULLParquet and ORC behave the same. A data file with field ids, or a scan whose schema lists
payload, was already read correctly. Before #69723 the V1 reader failed this plan (Missing Iceberg schema metadata for equality-delete field id), and a 4.1.3 BE refuses equality deletes on data files without field ids, so during an upgrade this was an error before it became wrong rows. It takes an old FE in front of an upgraded BE, data files without field ids, an equality delete keyed on a nested struct field, and a query that does not read that struct. branch-4.2 reads 4.1.3 plans with the same V2 code.How this PR fixes it
EqualityDeleteFilternow keeps, for each key, the delete file's names from the key's top-level column down to the key (field_name_paths, for example{"payload", "k"}) instead of the leaf name alone. The old-FE fallback builds a table path from these names, so the walk by name findspayloadand thenk, with the same matcher as before. A present parent without the leaf is still a missing key and reads as NULL. The retry with the delete file's leaf name, and the fallback field that builds a missing key's NULL, use the last name of the path, as before.One limit remains: a key renamed after a data file without field ids was written is not found under the delete file's name, because the old FE sends no name mapping for a column it does not list. Such a key reads as NULL, as a top-level key already does.
Results
New cases in
be/test/format_v2/table/iceberg_reader_test.cpp, all on scan parameters built the way a 4.1.3 FE builds them (make_old_fe_iceberg_scan_params):IcebergOldFePlanFindsUnlistedNestedEqualityKeyByDeleteFileNames: the example above, with and without field ids, with and withoutpayloadin the schemapayloadnot listed, Parquet and ORC, both tombstone setsIcebergOldFePlanMatchesMissingEqualityKeyAsNull: an INT or STRING key the file lacks, alone or withid, against NULL and0/''tombstonesIcebergOldFePlanMatchesMissingNestedEqualityKeyAsNull:payload.kthe file lacks, under NULL and present parentsThe last two pin what V2 already did for a key the file lacks: it is NULL in every row and matches only a NULL tombstone, never the
0or empty string stored under a NULL slot.Release note
Fix equality deletes on a nested field of Iceberg data files without field ids being ignored, or deleting every row of the file, when a BE runs a query planned by an FE of 4.1.3 or earlier during a rolling upgrade.
Check List (For Author)
IcebergV2ReaderTest,IcebergTableReaderTest,EqualityDeletePredicateTest,IcebergPositionDeleteSysTableV2*,TableReaderTest,ColumnMapper*,LocalColumnIndexTestandCloneTableExprTreeTestpass locally (411 tests, macOS, ASAN).build-support/check-build-hygiene.shpasses, and clang-tidy reports nothing on the changed lines.