Skip to content

[feature](iceberg) Support Iceberg V3 spatial types on master - #69032

Open
OIiveirra wants to merge 198 commits into
apache:masterfrom
OIiveirra:feature/iceberg-spatial-types
Open

OIiveirra wants to merge 198 commits into
apache:masterfrom
OIiveirra:feature/iceberg-spatial-types

Conversation

@OIiveirra

@OIiveirra OIiveirra commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Background

Iceberg V3 defines GEOMETRY and GEOGRAPHY with a coordinate reference system (CRS); GEOGRAPHY also identifies its edge algorithm. Their Parquet representation carries WKB bytes and spatial annotations. Doris needs to preserve that identity through catalog mapping, query execution and writeback, while keeping its existing STRING-based spatial functions compatible.

What problem does this PR solve?

Issue Number: N/A
Related PR: #67907
PR Type: Feature

Problem Summary:

Users need to query and write Iceberg V3 spatial columns without losing their CRS, edge algorithm or original WKB. Add native spatial types, recursive Parquet read/write and typed spatial functions. This requires consistent type transport, write validation before coercion, and explicit rejection of operations that lack spatial implementations. About 65% of added lines are tests and generated results; existing STRING spatial functions retain their behavior.

Details

  • Change map: 216 files, +22,246/-593: production +7,679/-561; tests +13,680/-31; generated regression output +886/-0; build +1/-1. User documentation is in the separate website PR. Read the production changes in these parts:

    Part Added/deleted lines Purpose and main entry points
    Type identity and binary storage +1,436/-148 Preserve kind/CRS/algorithm through FE/BE descriptors and raw bytes through ColumnSpatial and the shared binary SerDe.
    Iceberg mapping and Parquet I/O +919/-26 Map nested schemas, retain spatial annotations and validate supported writes in the connector and Parquet writer.
    Admission and value execution +1,415/-66 Preserve identity and required-list NULL rules before coercion, and admit only implemented expression/aggregate paths; start with ConnectorSpatialWriteNormalizer and ConnectorSpatialWriteValidator.
    Typed GIS and WKB validation +3,909/-321 Bind typed constructors/accessors, preserve WKB and implement spherical calculations; start with functions_geo.cpp, wkb_parse.cpp and geography_calculation.cpp.
  • Scope: Read/write top-level spatial columns, ARRAY elements, ordinary-key MAP values and STRUCT fields through Parquet. Preserve all seven basic ISO WKB kinds in XY/XYZ/XYM/XYZM, NULL/EMPTY and original bytes. ST_AsBinary preserves Z/M; typed text access and calculations reject non-XY inputs. Geography measurements and relations support default OGC:CRS84/spherical nonempty XY Point/LineString/Polygon, including typed ST_Equals, ST_Overlaps and ST_DWithin. Unsupported shapes/metadata and unresolved conversion-precision cases fail explicitly.
    Spatial Variant values, ORC materialization, native spatial GROUP BY/DISTINCT/set-operation keys, window partition keys and unsupported aggregate combinations are rejected. Preserve UNION ALL, supported spatial value aggregates/windows and ordinary-key/spatial-value operations. Paimon/Lance/GeoArrow, internal spatial storage, indexes, pruning, predicate pushdown, CRS conversion, typed WKT input and additional Geography algorithms remain outside scope.

  • Design: Keep kind/CRS/algorithm in typed metadata because a generic binary or STRING carrier cannot identify them. Raw WKB is canonical so storage round trips retain the input bytes. Binary and spatial Parquet consumers share byte transfer; spatial validation stays in the spatial SerDe. FE write normalization retains ordinary keys/siblings and requiredness before coercion. Spherical predicates use original-coordinate checks to reject unresolved S2 boundary-rounding ambiguity. Scope decisions are recorded in this PR's review discussion; no separate design issue is linked.

  • Usage: The typed constructor accepts hexadecimal WKB. For example, create a Point at (1, 2):

    SELECT ST_AsText(ST_GeometryFromWKBTyped(
        '0101000000000000000000F03F0000000000000040'));

    For an imported Iceberg spatial column, use ST_AsText(geom) or hex(ST_AsBinary(geom)) to inspect it; matching target schemas support INSERT SELECT while preserving spatial identity.

  • Default behavior: No new runtime flag is introduced. Eligible Iceberg fields map to native spatial types automatically. Spatial writes require Iceberg V3 and Parquet, with an explicit connector capability for nested leaves. Nondefault CRS/algorithm metadata is retained without conversion; property-referenced projjson: writes are rejected before file creation.

  • Compatibility: Participating FEs/BEs must support the new types; mixed 4.x/5.0 execution and downgrade support for spatial metadata are outside scope. Connector plugins must match API 33.0, distinct from the reviewed base API 32.0. Existing VARBINARY Block layout and STRING constructors remain unchanged. Typed MySQL values use binary metadata; generic Flight clients need text/binary projections because direct spatial Flight schemas are unsupported. Retain the BE constant-folding exclusion because FE cannot decode spatial PValues.

  • Behavior change: Iceberg spatial fields are exposed as native types with binary MySQL results. Connector plugins must be rebuilt for API 33.0, including plugins that only expose ordinary columns.

  • Persistence and failure handling: Preserve spatial parameters in schema/type transport and Parquet annotations. Validate spatial identity and supported schemas before coercion or file creation. Reuse Iceberg's existing commit/conflict and cleanup lifecycle; DDL preflight checks SDK-observed schemas, without claiming a server-wide invariant for REST rebases the client cannot observe.

  • Follow-up plan: Nullable V2 batching and general dense-Polygon optimization remain deferred.

Testing

  • Local validation at d89b8a795bc: 337 FE tests passed, including the complete Connector SPI suite and selected Iceberg mapping/DDL/write-plan tests.
  • Local validation at e993d2e9d03 and c3c75caa110: 45 binary/spatial SerDe BE ASAN tests and 74 FE validator/normalizer/converter tests passed. These cover explicit ORC rejection and nested required-list NULL handling.
  • Earlier recorded integration validation at 8a774c249e29: 360 FE tests, 132 Geography BE tests, and the GIS/Iceberg suites passed locally, with 70 GIS and 131 Iceberg result sections. These results predate subsequent changes.
  • On current head e27809effd8, selected BE tests did not execute: local Lance/Rust dependency setup blocked the build. The new BSON-isolation regression and latest required-list external SQL cases remain unexecuted locally. The external suite requires enableIcebergTest and working Spark/MinIO fixtures. Latest-head full community CI has not been verified here; earlier passes are not a current-head CI verdict.

Notes for Reviewers

  • The latest merge keeps BSON decoding on concrete VARBINARY SerDe only; spatial columns retain the default BSON rejection. It adds no spatial MongoDB support.
  • Detailed precision/error cases are retained in the Geography tests and the spatial regression suites. Unsupported operations are deliberate boundaries, not promises of bytewise spatial SQL equality.

Release note

Support native Iceberg V3 GEOMETRY/GEOGRAPHY schema mapping and Parquet read/write, including ARRAY elements, ordinary-key MAP values and STRUCT fields. Preserve CRS, edge algorithm and raw ISO WKB including Z/M, with explicit XY calculation limits and unsupported-operation errors. Connector plugins must match API 33.0.

Check List (For Author)

  • Behavior changed: Yes (see Details)
  • Does this need documentation: Yes (apache/doris-website#4239)
  • Pre-submission status: Partially verified: latest-head BE unit execution is blocked by local dependencies; the new required-list external cases and latest full community CI have not been verified here.

AI Disclosure

  • Tool: Codex (model: not stated)
  • Generated: Assisted with implementation, tests, review-driven fixes and this description.
  • Human review: not stated

### What problem does this PR solve?\n\nIssue Number: close apache#28228\n\nRelated PR: apache#67907\n\nProblem Summary: Preserve parameterized geometry and geography types through Nereids function binding and add the geography WKB constructor on master.\n\n### Release note\n\nAdd spatial function type binding for geometry and geography values.\n\n### Check List (For Author)\n\n- Test: FE focused test pending; BE build pending\n- Behavior changed: Yes (spatial function return and input types)\n- Does this need documentation: No
Align generated Nereids spatial constructors with the native Geometry return type and preserve spatial inputs for ST_ASTEXT.
Record the master-port decisions, focused FE test result, and remaining BE and format gaps.
Add Geometry and Geography to the BE Iceberg schema model, preserve their Arrow binary metadata, and reject spatial values in internal-table DDL including nested containers.
Allow Arrow and Parquet binary materialization to operate on the spatial column container while preserving existing VARBINARY behavior.

### Check List (For Author)\n\n- Test: FE DDL test pending; BE build pending\n- Behavior changed: Yes (spatial binary read/write)\n- Does this need documentation: No
Allow spatial functions to consume raw WKB columns and preserve canonical WKB bytes in geometry and geography constructors.

### Check List (For Author)\n\n- Test: BE unit test pending; FE tests unaffected\n- Behavior changed: Yes (spatial WKB function inputs)\n- Does this need documentation: No
Reject malformed spatial payloads before geo functions execute and expose the validator for Iceberg write paths.

### Check List (For Author)\n\n- Test: BE unit test pending; dependency setup still blocks BE build\n- Behavior changed: Yes (invalid spatial input errors)\n- Does this need documentation: No
Emit top-level Geometry and Geography logical annotations in Iceberg Parquet files, validate WKB inputs before conversion, and reject nested spatial columns that the writer cannot represent.

### Check List (For Author)\n\n- Test: C++ writer test added; BE build pending dependency setup\n- Behavior changed: Yes (Iceberg Parquet schema and input validation)\n- Does this need documentation: No
### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: apache#67907\n\nProblem Summary: Validate raw spatial WKB inputs before spatial functions and Iceberg writes while preserving Geography SRID compatibility.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: FE focused tests passed; BE build blocked during third-party submodule preparation\n- Behavior changed: Yes (invalid Geometry WKB is rejected earlier)\n- Does this need documentation: No
### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: apache#67907\n\nProblem Summary: Remove stale comments that described explicitly supported Iceberg spatial types as unsupported and record the current validation commit in the work log.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: No need to test; comment and worklog only\n- Behavior changed: No\n- Does this need documentation: No
### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: apache#67907\n\nProblem Summary: The mapping implementation handles Geometry and Geography explicitly, but its fallback comment still listed them as unsupported.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: No need to test; comment only\n- Behavior changed: No\n- Does this need documentation: No
### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: apache#67907\n\nProblem Summary: Add an external Iceberg V3 regression that creates a Geometry table, writes raw WKB through Doris, reads it back from Parquet, and checks ST_AsText output.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: Added gated external Iceberg regression; not run because the external Docker environment is unavailable\n- Behavior changed: No\n- Does this need documentation: No
### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: apache#67907\n\nProblem Summary: Generic connector sink coercion could hide Geometry/Geography kind, CRS, and edge-algorithm mismatches before Iceberg writes. Validate spatial source expressions against target metadata before the common coercion path, while preserving NULL writes.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: ./build.sh --fe; FE_UT_PARALLEL=1 ./run-fe-ut.sh --run org.apache.doris.datasource.connector.converter.ConnectorSpatialWriteValidatorTest\n- Behavior changed: Yes (invalid spatial write conversions fail during analysis)\n- Does this need documentation: No
### What problem does this PR solve?\n\nIssue Number: N/A\n\nRelated PR: apache#67907\n\nProblem Summary: Adding external-only spatial primitive types left the exhaustive storage-field mapping without explicit cases, causing BE compilation to fail under -Werror.\n\n### Release note\n\nNone\n\n### Check List (For Author)\n\n- Test: BE build resumed after this compile fix\n- Behavior changed: No; spatial types still reject internal segment storage\n- Does this need documentation: No
### What problem does this PR solve?

Related PR: apache#67907

Problem Summary: Align spatial serialization with current master APIs. Put optional spatial metadata on PScalarType, adapt generic binary SerDe to StringRef and Arrow offsets, remove obsolete column overrides and version constants, and verify Parquet CRS through concrete logical types.

### Release note

None

### Check List (For Author)

- Test: ./build.sh --be passed; BE spatial unit tests compiled but ASAN linking failed due toolchain duplicate glibc symbols; release retry pending
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Related PR: apache#67907

Problem Summary: Connector changelog UPDATE and MERGE plans returned before the generic sink validator, allowing spatial metadata to be coerced without checking kind, CRS, or geography algorithm. Validate each changelog expression before casting and cover both paths with plan-builder tests.

### Release note

None

### Check List (For Author)

- Test: FE connector spatial and changelog plan tests, 13 tests passed
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Related PR: apache#67907

Problem Summary: Geography values created without explicit metadata are valid under the Iceberg V3 default contract, but the BE type constructor rejected an empty algorithm. Normalize the default to spherical while preserving Geometry and explicit metadata semantics.

### Release note

None

### Check List (For Author)

- Test: focused BE gtest exposed and isolated the default metadata case; follow-up spatial suite in progress
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Related PR: apache#67907

Problem Summary: Keep projected Geometry accessors in raw WKB space, validate S2 Geography metadata before execution, preserve legacy encoded WKB plans and current master geo functions, fix spatial serialized-size accounting, and use the supported Parquet group annotation API. Cover projected Geometry and legacy EWKB with focused tests.

### Release note

Spatial metadata mismatches fail explicitly rather than using incorrect spherical semantics.

### Check List (For Author)

- Test: run-be-ut.sh, 86 tests passed with ASAN and local GLIBC_COMPATIBILITY=OFF; clang-format 16.0.6; build hygiene passed; clang-tidy in progress
- Behavior changed: Yes, reject unsupported spatial metadata
- Does this need documentation: Yes, pending spatial feature documentation
### What problem does this PR solve?

Related PR: apache#67907

Problem Summary: Apply clang-format 16 to the spatial feature scope and resolve new auto and integer sign comparison diagnostics while retaining verified geo behavior.

### Release note

None

### Check List (For Author)

- Test: run-be-ut.sh, 86/86 passed with ASAN and local GLIBC_COMPATIBILITY=OFF; check-format.sh passed; whitespace and build hygiene passed
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Related PR: apache#67907

Problem Summary: Resolve qualified-auto diagnostics in the new spatial column implementation and record verified personal test service health.

### Release note

None

### Check List (For Author)

- Test: full feature clang-tidy in progress; existing spatial BE suite passed 86/86 before these qualifier-only changes
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Related PR: apache#67907

Problem Summary: ST_AsBinary copied Geometry WKB directly while ignoring the nullable column null map. Propagate NULL before copying raw bytes and test accessors with valid hidden payloads. Also resolve the fixed binary width integer-comparison diagnostic.

### Release note

ST_AsBinary returns NULL for nullable Geometry input rows.

### Check List (For Author)

- Test: run-be-ut.sh, 87/87 spatial and geo tests passed with ASAN and local GLIBC_COMPATIBILITY=OFF
- Behavior changed: Yes, preserve NULL results
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#67907

Problem Summary: Iceberg spatial schemas have native Doris types, but SQL CREATE and ALTER cannot declare them. Add Geometry and Geography grammar with CRS and edge algorithm parameters, preserve their use as non-reserved column names, and normalize supported Geography algorithms in the shared type factory. Dispatch column types through the parser visitor so CREATE and ALTER use the same type parsing path.

### Release note

Allow GEOMETRY and GEOGRAPHY declarations with CRS and Geography edge algorithm parameters for Iceberg external tables.

### Check List (For Author)

- Test: Unit Test; standard run-fe-ut.sh, DataTypeTest 38/38 and ColumnDefinitionTest 8/8 passed, fe-spatial-ddl-ut.log BUILD SUCCESS
- Behavior changed: Yes; SQL recognizes parameterized spatial types and rejects unsupported Geography algorithms
- Does this need documentation: Yes; spatial external-table documentation pending
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#67907

Problem Summary: Iceberg CREATE and column evolution can introduce spatial columns on V2 or non-Parquet tables, or introduce nested spatial values that Doris cannot write. Validate these constraints before remote schema mutations. CREATE uses effective catalog override, request, and default properties; ADD and MODIFY use actual table metadata and the writer's existing format resolution, including the write-format alias. Ordinary column changes retain their existing path.

### Release note

Reject Iceberg spatial schema changes that require unsupported table versions, write formats, or nested spatial writes.

### Check List (For Author)

- Test: Unit Test; standard run-fe-ut.sh via personal wrapper, IcebergSpatialDdlTest 8/8, metadata DDL 37/37 and column evolution 23/23 passed, fe-spatial-spi-ddl-ut.log BUILD SUCCESS
- Behavior changed: Yes; unsupported spatial schema changes fail before remote mutation
- Does this need documentation: Yes; spatial external-table limitations documentation pending
### Background

The MAP aggregate functions validate their key types before coercion and again
after rewriting. Aggregate states and combinators share that validation.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (follow-up)
PR Type: Bugfix

Problem Summary:

The spatial MAP key guard missed spatial types inside ARRAY, MAP and STRUCT
keys. The planner could accept a key type that the execution path cannot use.

### Details

- **Frequency:** A MAP aggregate or state uses a key with a nested GEOMETRY or GEOGRAPHY leaf.
- **Root cause:** The legality check tested only the outer key type.
- **Why existing tests missed it:** The MAP aggregate key tests only covered direct spatial keys.
- **Fix:** Reuse the recursive spatial type check on the MAP key and extend the existing aggregate, state and combinator cases.
- **Other paths and branches:** Value and DISTINCT rules continue to apply independently.
- **Behavior change:** The planner rejects nested spatial MAP keys with the existing spatial-key analysis error.

### Testing

- Confirmed the new FE check assertion fails on the previous implementation.
- Ran SpatialAggregateLegalityTest, SpatialPredicateLegalityTest and MapLambdaFunctionsTest.
- Passed Checkstyle and git diff --check.

### Release note

MAP aggregate functions reject keys that contain nested spatial values during
analysis.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Code, test updates and commit message
- Human review: Pending

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One new P2 correctness finding: the spatial BINARY override can undo a timestamp sibling rewrite in nested Iceberg equality-delete files, causing an otherwise valid scan to fail. It is distinct from the earlier missing-BINARY-override thread. The inline comment contains the trigger and fix direction.

Static coverage: all 234 changed paths (149 FE/regression, 70 BE I/O/type/protocol, 15 GIS), complete changed test bodies and regression results, plus an independent risk review. No additional user-provided focus was supplied.

Critical checkpoint conclusions:

  1. Goal and proof: native spatial identity/WKB, Iceberg v3 Parquet I/O and deliberately bounded Geography calculations are wired end to end; the reported delete-reader interaction remains incorrect. Relevant positive/negative tests are present and were inspected.
  2. Scope and reuse: the large change follows one type feature through its consumers; shared binary serialization, validation and schema helpers reuse existing mechanisms.
  3. Concurrency and locks: no new worker or lock order found. DDL candidate validation retains existing Iceberg CAS/retry behavior; reader overrides are split-local and separate from cached metadata.
  4. Lifecycle: column/aggregate payloads own their bytes; window erasure reclaims dead arenas. Writer failure cleanup closes/destroys upload owners before deletion. No new cross-TU initialization dependency found.
  5. Configuration: no new runtime setting; existing timestamp mapping is the reported trigger. SQL-mode rendering and internal byte-mode cache identity were checked.
  6. Compatibility: legacy STRING GIS and ordinary VARBINARY behavior remain separate; the VARBINARY Block layout is unchanged. Connector API31 and its version gate/baseline agree. Mixed 4.x/5.0 spatial execution and downgrade remain explicitly unsupported under the existing maintainer scope decision; this review does not certify them.
  7. Thrift/protobuf: all touched definitions compared with branch-3.1/4.0/4.1/4.2 and master refs. Spatial enums avoid FILE's existing IDs; new metadata fields are optional and no new collision was found.
  8. Parallel consumers: checked typed/legacy, scalar/nested, plain/dictionary, NULL/constant, projected/hidden delete keys, constructors/aggregates/state/combinators, INSERT/UPDATE/MERGE/changelog, and data/delete-only writers. The combined BINARY/timestamp tombstone case is the verified omission.
  9. Conditions: metadata/format/dimension/EMPTY restrictions and NULL-provenance checks have concrete paths and negative controls. Ordinary sibling casts, input scope, ExprIds, UNION common types and selected MERGE-branch evaluation were traced.
  10. Tests: inspected unit, actual-file and external regression coverage. Missing combined nested BINARY/timestamptz delete-key coverage is requested inline.
  11. Results: changed golden rows match query/fixture shapes on inspection; no additional result mismatch found. Generation provenance and runtime numeric results were not verified.
  12. Observability: unsupported operations/WKB failures retain analysis or Status context; cleanup logs identify files. No new per-row INFO logging or concrete additional metric defect found.
  13. Persistence and failover: CRS/algorithm use existing type persistence and descriptor paths. No new EditLog, visible-version or MoW logic; Iceberg retains existing metadata commit ownership.
  14. Writes and failure atomicity: full-schema admission and pre-file CRS validation were checked, along with existing commit/rollback and failed-open cleanup. No separate new partial-commit, leak or deadlock found.
  15. FE/BE transport: scalar/column Thrift, protobuf and connector copies retain spatial parameters. Constant folding excludes unsupported spatial PValues; MySQL and typed ST_AsBinary use binary carriers.
  16. Performance: reviewed owned filtering/erasure, partial TopN, raw-coordinate calculations and indexed admission. Already-reported nullable V2 batching and general dense-polygon/long-arc work remain known limits and are not reposted; no universal linear bound or benchmark result is claimed.
  17. Final challenge and prior reviews: rechecked candidates, callers, alternate paths and anchors. Historical P1 COUNT, FILE IDs, Apple ARM arithmetic, dimensional validation, ORC and nested MAP keys are repaired; bearing, polygon-interior distance and identity-partition allegations are contradicted by current code/library contracts and tests. No in-scope existing P0/P1 remains independently confirmed; the mixed-version scope exception is stated above.

Validation was static inspection only. No builds, tests, benchmarks, cluster execution, source changes or submodule initialization were performed. Runtime/platform/ASAN/external interoperability validation remains unverified.

Review converged after 2 round(s).

### Background

Iceberg equality-delete readers can apply timestamp semantics and raw BINARY
overrides to different leaves of one nested Parquet STRUCT.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (follow-up)
PR Type: Bugfix

Problem Summary:

Synchronizing an annotated BINARY leaf rebuilt its parent STRUCT from the
native footer and restored a timestamp sibling to DATETIMEV2. The file block
then disagreed with the reader output when timestamp mapping was disabled.

### Details

- **Frequency:** A nested equality-delete key combines an annotated BINARY leaf with a TIMESTAMPTZ sibling.
- **Root cause:** The raw-byte override rebuilt the native parent from stale child types.
- **Why existing tests missed it:** BINARY and timestamp equality-delete tests covered separate nested paths.
- **Fix:** Synchronize the current split-local child types into the native schema before rebuilding the parent.
- **Other paths and branches:** Existing timestamp and spatial decoding rules remain in place.

### Testing

- Ran 213 targeted Iceberg and Parquet BE ASAN tests.
- Added a regression for nested BINARY plus TIMESTAMPTZ schema synchronization.
- Passed changed-file Clang checks, build hygiene, clang-format and git diff --check.

### Release note

Nested Iceberg equality-delete schemas preserve timestamp siblings when a
spatially annotated BINARY key is read as raw bytes.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Code, regression test and commit message
- Human review: Pending
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of PR #69032 at 4d7ce45ce69a02ab2ff547496eac9dfc505a7522: no new actionable finding was verified in the supported feature scope. No additional user review focus was supplied. Existing findings were not reposted.

Static review covered all 234 paths in the supplied aggregate diff: 82 BE, 143 FE, three shared protocol definitions and six regression SQL/output files. Frontend/backend complete reviews and a separate risk review examined related invocation chains and tests. No builds, tests, benchmarks, source edits or submodule initialization were performed. Author-reported test passes were treated as prior reports, not validation executed by this review.

Critical checkpoint conclusions:

  • Goal and proof: spatial kind/CRS/algorithm and raw WKB are carried through Iceberg mapping, Parquet I/O and typed execution. Positive/negative tests were inspected; source inspection does not establish a current-head runtime pass.
  • Scope and reuse: the broad change follows the necessary type, connector, planner, reader/writer and function boundaries. Binary payload transfer is shared while spatial validation remains separate. No additional scope/refactoring defect identified.
  • Concurrency and locks: function scratch, normalization state and semantic reader trees are query/split owned; cached footer metadata stays immutable and cache keys include annotation interpretation. Iceberg DDL validates the SDK-observed commit candidate across retries. No new thread-entry, lock-order or heavy-under-lock defect identified.
  • Lifecycle and initialization: column copies and aggregate states own WKB; mutable filtering preserves arena ownership and erasure reclaims discarded storage. Writer error cleanup closes pending uploads before deletion. Wide arithmetic has a portable fallback for narrow long double. No additional ownership/static-initialization defect identified.
  • Configuration: no new runtime configuration or dynamic-update obligation.
  • Compatibility and API/storage changes: existing VARBINARY block layout is retained; connector API is deliberately bumped to 31.0, with current31/prior30 probes aligned. Participating nodes must support spatial types. Mixed 4.x/5.0 spatial execution is an explicit maintainer-approved scope exclusion (decision, existing disposition); it is not reposted or counted as an in-scope P1.
  • Thrift/protobuf IDs: complete touched definitions were compared with current live-matching branch-3.1, branch-4.0, branch-4.1 and branch-4.2 definitions. New metadata fields are optional; spatial enum allocations avoid branch-4.1 FILE's protobuf45/Thrift47 identities. No new collision or ID reuse found.
  • Parallel paths: scalar/nested, plain/dictionary, nullable/constant, data/equality-delete, typed/opaque-BINARY and insert/update/merge paths were inspected. ORC spatial paths explicitly reject before invalid casts; Hive/load BINARY consumers retain opaque bytes. Sort/spill and position-delete writer entry points were also traced.
  • Conditional and error behavior: unsupported metadata, dimensions, kernels and unresolved calculation ambiguity have explicit rejection paths. Actual NULL required container values are rejected recursively by VIcebergTableWriter::_write_prepared_block -> iceberg::validate_write_block before partition dispatch or Arrow conversion; optional parent NULLs correctly mask children. No reachable required-container write defect remains.
  • Test coverage: inspected unit/regression additions cover transport, ownership, NULLs, complex schemas, dimensions, negative admission, BINARY overrides, equality deletes, numerical cases and work counts. Existing runtime write-validator tests also cover required elements/children and rejection before writer creation. Runtime/platform/cross-writer execution remains unverified here.
  • Expected results: changed regression SQL and generated output sections were statically compared for shape, row counts and expectations, including corrected Z/M projections. No additional mismatch was substantiated. Current-head output generation and independently computed numerical truth for every spherical fixture were not verified by execution.
  • Observability: errors retain type/row/file context at existing boundaries and reader profile paths remain intact; no new high-volume INFO logging defect identified. Work-count hooks are present, but logs, metrics and performance counters were not measured.
  • Persistence and failover: spatial parameters use the existing catalog/schema/type serialization paths. Internal spatial storage is rejected; no internal rowset, visible-version, delete-bitmap or EditLog transaction protocol changes. Iceberg metadata commit/retry and cleanup retain their existing ownership.
  • Writes and atomicity: data-file writes use the pinned target schema and existing commit/rollback lifecycle. Recursive target-schema value validation precedes all production partition-writer routes; sorted/spilled data has already passed that validation. The direct alternate Parquet-writer consumer is restricted to fixed position-delete fields. No new write-atomicity or value-loss defect identified.
  • FE/BE transport: connector/catalog types, Thrift, PTypeDesc, PColumnMeta, MySQL binary metadata and recursive spatial constant-fold exclusion were inspected. Direct Flight spatial schemas remain unsupported as documented; text/binary projections are the supported client route.
  • Performance and memory: in-place filtering, periodic arena compaction, byte-oriented WKB parsing and indexed/grouped noding address earlier redundant work. Query-owned CTE memoization avoids repeated producer expansion. Nullable-selection batching and general dense-polygon optimization remain known existing discussion limits; no new distinct performance defect or benchmark claim.
  • Additional risks and prior findings: current timestamp-sibling/BINARY overrides synchronize split-local sibling types before rebuilding parents; UNION normalization reconciles branch types and preserves output identities; recursive spatial MAP-key rejection and API31 wiring are present. Earlier in-scope P1 reports are fixed or excluded by current source/dependency guards. No existing in-scope blocking comment ID remains to carry forward, and no concrete candidate or initial risk item remains unresolved.

The conclusion is bounded to static inspection of this head. Runtime/interoperability, exhaustive numerical-oracle and benchmark validation remain limits of this review, not claims of completed execution.

Review converged after 3 round(s).

…sync

### Background

Iceberg uses file column definitions to carry logical types into Parquet reads.
Parquet keeps physical field nullability in its reader schema.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (follow-up)
PR Type: Bugfix

Problem Summary:

Synchronizing logical types after a nested BINARY override made required reader
fields nullable. Keep each Parquet reader node's original nullability when
copying its logical type.

### Details

- **Frequency:** A nested BINARY override touches a Parquet STRUCT with required descendants.
- **Root cause:** File column definitions add nullable wrappers for table mapping, and the sync copied those wrappers into the reader schema.
- **Why existing tests missed it:** The schema test used nullable leaves. Existing file tests exercised BINARY and timestamp equality keys separately.
- **Fix:** Preserve the reader node's nullability and test the combined key on both sides of an equality delete.
- **Other paths and branches:** This follows up on the split-local Parquet schema change in apache#69032.

### Testing

- The new required-leaf assertions fail on 4d7ce45 and pass with the fix.
- With timestamp mapping disabled, both equality-key orders retain IDs 2 and 3 from real Parquet data and delete files.
- Ran 214 Iceberg and Parquet BE ASAN tests and checked the changed lines with clang-format and clang-tidy.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Code, tests and commit message
- Human review: Pending
englefly
englefly previously approved these changes Nov 13, 2026

@englefly englefly left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

HappenLee
HappenLee previously approved these changes Nov 14, 2026

@HappenLee HappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

### Background

The spatial-type branch adds typed Geography evaluation and connector metadata.
Current master adds relation-function constant caching and connector API 31.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (follow-up), apache#70302 (constant shape caching), apache#70372 (connector API 31)
PR Type: Build/CI/Chore

Problem Summary:

Merge current master and reconcile its constant shape cache with typed WKB
admission and NULL evaluation. Keep the spatial connector surface on API 32
after upstream API 31.

### Details

- **Affected:** The spatial relation implementation, geometry state, conflict-local tests, and connector API version pins.
- **Developer and release impact:** The merged branch retains constant caching and the spatial calculation contracts. Connector plugins use API 32.
- **License and third-party:** This merge introduces no extra dependency changes.
- **Rollback:** Revert this merge while retaining the branch's existing commits.

### Testing

- Ran 399 BE ASAN tests across VGeoFunctionsTest, GeoTypesTest, IcebergV2ReaderTest and NewParquetReaderTest.
- Ran 155 connector SPI tests and 14 PluginApiVersionWiringTest tests.
- Checked changed C++ lines with clang-format 16 and clang-tidy, plus build hygiene and FE Checkstyle.

### Release note

None

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Merge resolution, focused regression coverage and commit message
- Human review: Pending
### Background

Typed Geography computes relative bearings from original WKB ordinates.
Connector writes retag spatial NULLs before coercing nested target types.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (follow-up fixes)
PR Type: Bugfix

Problem Summary:
Near-zero tangent cancellation can turn a representable non-equatorial angle
into zero. Spatial constructors alongside typed NULLs in STRUCT writes are
rejected because the same-producer proof only admits slots. Use a stable
difference-angle expansion and admit unchanged bound expressions with exactly
matching spatial identity.

### Details

- Frequency: Precision-boundary angles and nested spatial constructor writes.
- Root cause: An excessively coarse fallback error bound and a slot-only proof.
- Why existing tests missed it: They covered equatorial endpoints and slots.
- Fix: Certify the relative angle with weighted product error bounds and exact
  great-circle proof; return an explicit ambiguity error when unresolved.
  Keep matching bound values in their producer while retagging NULL siblings.
- Other paths and branches: Preserve degenerate NULL angles, legacy STRING
  behavior, Cast validation, and spatial identity checks.
- Behavior change: Correct supported tiny angles and constructor/CASE writes.

### Testing

- New non-equatorial angle and constructor regressions fail on 7953ffc.
- 402 BE ASAN tests passed across VGeoFunctionsTest, GeoTypesTest,
  IcebergV2ReaderTest and NewParquetReaderTest.
- 46 FE tests passed across ConnectorSpatialWriteNormalizerTest and
  ConnectorSpatialWriteValidatorTest; CheckStyle passed.
- Changed-line clang-format 16 and clang-tidy, build hygiene, and diff checks
  passed. External SQL integration and full community CI were not run.

### Release note

Preserve representable tiny Geography angles and permit matching bound
spatial values alongside spatial NULLs in nested writes.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready for this local fix scope

### AI Disclosure

- Tool: Codex
- Generated: Fixes, regression tests, and commit description.
- Human review: Not stated.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43823157694

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43823667186

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

### Background

Typed Geography computes relative bearings from original WKB coordinates.
Spatial writes retag proven NULL values in their owning producer before sink
coercion, while preserving ordinary keys and ordering.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (review follow-up)
PR Type: Bugfix

Problem Summary:
The angle fast path ignores cancellation inside its tangent inputs and can
return a wrong tiny turn before reaching the stable fallback. Ordinary
keys/count sorting expressions still block proven spatial MAP NULL writes.
Account for tangent input error and recursively retain safe MAP observations.

### Details

- Frequency: Always for the reported native-wide micro-angle and affected
  same-producer MAP ordering shapes.
- Root cause: The fast bound covered only final products, and the Sort gate
  recognized only a root CARDINALITY with no composed consumer rebinding.
- Why existing tests missed it: Angle tests reached the later residual;
  MAP tests covered direct CARDINALITY but no composed keys/count consumers.
- Fix: Carry the north construction scale into the fast decision and use the
  existing raw fallback when the estimate does not fit DOUBLE precision.
  Recognize existing keys/count observations recursively and rebind changed
  slots and function metadata inside ordinary ordering expressions.
- Other paths and branches: Native and portable-wide paths are tested.
  Preserve lookup value dependencies, original casts, producer scope, volatile
  key identity, sorting, and limits. These defects are in this PR's new types.
- Behavior change: Correct supported tiny angles and nested MAP NULL writes.

### Testing

- New regressions fail on 83a676b because of the target defects.
- 403 BE ASAN tests and 48 FE normalizer/validator tests passed.
- CheckStyle, changed-line clang-format 16 and clang-tidy, build hygiene,
  and diff checks passed.
- External SQL integration and complete community CI were not run locally.

### Release note

Correct tiny Geography turns affected by tangent cancellation and allow
existing keys/count sorting consumers during spatial MAP NULL writes.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready for these local fixes

### AI Disclosure

- Tool: Codex
- Generated: Fixes, regression tests, and commit description.
- Human review: Not stated.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

englefly
englefly previously approved these changes Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43852398429

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

### Background

The Iceberg spatial branch extends connector type mappings and schema carriers.
Upstream master now supports read-only Iceberg UNKNOWN columns in the same paths.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032 (spatial support), apache#70112 (upstream UNKNOWN support)
PR Type: Build/CI/Chore

Problem Summary:

Merge master at 13cff7f into the spatial branch. Preserve GEOMETRY and
GEOGRAPHY metadata alongside upstream UNKNOWN read semantics and write guards.
Reconcile the mapping tests and pass the authoritative scan schema to the spatial
equality-delete carrier test after upstream changed the encoding method signature.

### Details

- **Change map:** Synchronize nine upstream commits across 203 paths; manual
  resolutions are limited to one mapping file and two existing Iceberg test files.
  Other upstream content remains the automatic merge result.
- **Affected:** Iceberg primitive mapping and mapping/schema carrier tests.
- **Developer and release impact:** Preserve both branches' behavior and restore
  compilation of the spatial schema carrier test; no additional feature scope.
- **License and third-party:** No new dependencies or license changes.
- **Rollback:** Revert the merge commit using the spatial branch parent as mainline.

### Testing

- Local Maven reactor: 141 tests passed across IcebergTypeMappingReadTest (15),
  IcebergSchemaUtilsTest (40), ConnectorColumnConverterTest (38),
  ConnectorSpatialWriteValidatorTest (17), and ConnectorSpatialWriteNormalizerTest (31).
- Checkstyle: zero violations in the tested reactor.
- Connector forbidden-import and BE build/header hygiene gates passed.
- Source whitespace checks passed. Upstream generated regression outputs retain
  their existing whitespace.
- Original datasketches submodule modifications retained with identical binary-diff hash.
- Full CI, BE unit tests, and a new full PR review were not run for this merge.

### Release note

None

### Check List (For Author)

- Behavior changed: No additional behavior beyond the two merged branches.
- Does this need documentation: No.
- Pre-submission status: Partially verified: focused FE tests and static checks passed;
  full CI and BE tests remain unverified for this merge.

### AI Disclosure

- Tool: Codex.
- Generated: Conflict resolution, test call-site adjustment, and this commit message.
- Human review: Not performed within this agent session.
… lint

### Background

The spatial PR supports typed Geometry and Geography values through SQL functions and Iceberg reads and writes. Existing coverage includes geometric calculations and nested spatial schema conversion.

### What problem does this PR solve?

Issue Number: None
Related PR: None
PR Type: Test

Problem Summary:

Non-equatorial tiny angles need SQL-level coverage, and NULL-only spatial MAP casts need an end-to-end write check when key/count observers share the producer used by SORT/LIMIT. Add those scenarios and remove the changed-line clang-tidy diagnostics without changing production behavior.

### Details

- **Scenarios:** Four non-equatorial tiny angles with an independent analytic oracle; Geometry and Geography MAP writes with cardinality, size, map_keys and map_contains_key sort expressions, checking the actual LIMIT winner and persisted NULL value.
- **What existing tests miss:** SQL binding/execution for tangent cancellation, and the written row when NULL map values acquire a destination spatial identity while other expressions observe the producer.
- **Fails when broken:** Tiny positive angles collapse to zero, inverse angles round to 2*pi, or the map producer yields a wrong winner, key, cardinality or value.
- **Behavior change:** None. Production edits remove a redundant declaration and use designated initializers, auto and ranges in existing spatial code. Test-only helpers preserve all assertions and intentional reverse-argument cases.

### Testing

- Local standard build.sh --be --fe passed with ASAN, GLIBC compatibility, UI, Java extensions and CDC enabled.
- Nine standard regression suites passed: six spatial suites, two VARBINARY suites and Iceberg spatial V3.
- Golden output was generated by the standard runner; all 70 existing tags were unchanged and all four new rows were independently checked.
- Persisted MAP rows were verified through a new connection: IDs 2304 and 2404, key 204, cardinality one, NULL value.
- Full FE selection: 22,680 tests, one Variant fixture failure, 59 skipped. Supplemental Java-extension coverage: 482 tests, no failures/errors, 18 skipped after using a private Java temporary directory. A temporary debug-flag probe confirmed the Variant fixture precondition and was restored.
- Segmented BE ASAN coverage (GLIBC_COMPATIBILITY=OFF for the local UT dependency setup; the complete product used ON) accounted for all 16,916 enabled tests: 16,847 passed, 66 skipped and three unresolved CDC process tests failed; 48 disabled tests were not run. Original cache-state ASAN failures were retained and isolated/configuration controls were recorded. The final segment had no assertion failures but exited on a 224-byte OpenMP allocation leak.
- Clang-format, changed-line clang-tidy convergence, CheckStyle, architecture gates, license headers and git diff checks passed.

### Notes for Reviewers

No unrelated Variant, CDC or cache implementation changes are included. Full local unit-test execution is not entirely green, and this does not replace the remaining community CI configurations.

### Release note

None.

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Partially verified: the local unit-test failures and LSan exit above remain; targeted regressions and the complete product build passed.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6).
- Generated: The lint adjustments, regression additions, generated golden output and this message.
- Human review: Pending author review.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43926053832

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39

Copy link
Copy Markdown
Contributor

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43999643506

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

### Background

The spatial PR extends the connector SPI. Upstream master independently added secret-property declarations to that SPI and advanced the connector API to the same major.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032, apache#70338 (upstream connector API change)
PR Type: Build/CI/Chore

Problem Summary:

Merge master 9b5352b into the spatial branch while retaining both SPI surfaces. Resolve the API-32 version collision by advancing the merged spatial contract to API 33 and checking that pre-spatial API-32 plugins are rejected.

### Details

- **Change map:** Merge upstream master; manually reconcile only the connector parent POM, its recorded-surface version test, and the engine plugin-version wiring test. Other changes retain the upstream merge result.
- **Affected:** Connector plugin API version declaration and its two version tests.
- **Developer and release impact:** In-tree connectors share API 33; third-party connector plugins need the spatial API contract. Preserve both the spatial and upstream secret-property surface baselines.
- **License and third-party:** No additional dependency or license change beyond the upstream merge.
- **Rollback:** Revert this merge while retaining its first parent if necessary.

### Testing

- The personal standard FE UT wrapper passed all 291 tests in 39 classes: the complete connector SPI suite, IcebergWritePlanProviderTest and PluginApiVersionWiringTest; zero failures, errors or skips.
- The personal standard BE UT wrapper compiled the merged ASAN sources and passed all 292 tests in eight Iceberg, table-reader and cache suites; zero failures or skips. Local UT GLIBC_COMPATIBILITY=OFF; task-owned dependency adapter restored.
- The initial BE run had three fixture-setup failures from inaccessible shared temporary directories. The same selection passed with a private TMPDIR, without source changes or changing those directories.
- Clang-format16 checked 5145 files; build hygiene and full FE reactor CheckStyle passed. Code whitespace checks passed; generated regression golden formatting was preserved.
- Complete product rebuild, SQL regression, full unit-test matrix and a new full PR review were not run for this merge.

### Release note

None.

### Check List (For Author)

- Behavior changed: Yes, connector API advances to 33 to distinguish the merged spatial contract from upstream API 32.
- Does this need documentation: No
- Pre-submission status: Partially verified: targeted merge checks passed; complete community CI remains separate.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6).
- Generated: The merge conflict resolution, API-version test adjustments and this message.
- Human review: Pending author review.
### Background

The spatial branch passes its Parquet spatial-annotation policy through data and delete readers. Upstream master added independent file-encryption descriptors to those same reader constructors and a generic table-reader hook.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#69032, apache#70321 (independent delete keys), apache#70314 (encrypted indexes)
PR Type: Build/CI/Chore

Problem Summary:

Merge master 43963ce into the spatial branch. Preserve each data/delete file's own encryption descriptor together with the existing spatial-annotation option, and retain both protected table-reader hooks. Keep upstream encrypted-cache isolation and the spatial read behavior intact.

### Details

- **Change map:** Manually reconcile three files: iceberg_reader.cpp, table_reader.cpp and table_reader.h. Other changes retain the upstream merge result. The earlier merge b6c42b5 already reconciles the connector API as major 33.
- **Affected:** Two Parquet reader construction sites and the table-reader hook declarations.
- **Developer and release impact:** Spatial readers retain their annotation policy while data and delete readers keep independent encryption keys. Existing encrypted-cache bypass, metadata-aggregate rejection and plaintext caching remain in place.
- **License and third-party:** No additional dependency or license change beyond upstream.
- **Rollback:** Revert this merge while retaining its first parent if necessary.

### Testing

- Final merged candidate: 291 FE tests in 39 classes passed with zero failures/errors/skips, including the complete connector SPI, IcebergWritePlanProviderTest and PluginApiVersionWiringTest.
- Final merged candidate: BE ASAN compiled and ran 895 tests in 55 suites: 893 passed, zero failed, two existing opt-in Parquet performance microbenchmarks skipped. Includes 17 executed encryption-related cases and the spatial/Iceberg/Parquet reader coverage.
- BE used the standard personal clang21 wrapper, GLIBC_COMPATIBILITY=OFF and a private TMPDIR. The task-owned dependency adapter was restored.
- Clang-format16, build hygiene, explicitly enabled full FE reactor CheckStyle and code whitespace checks passed. Changed-line clang-tidy validates the two implementations against the merged source; the compiler resource include path is provided by a task-private adapter.
- Complete product packaging, SQL regression, full unit-test matrix and a new full PR review were not run for this merge.

### Release note

None.

### Check List (For Author)

- Behavior changed: No additional behavior beyond the existing spatial branch and upstream changes.
- Does this need documentation: No
- Pre-submission status: Partially verified: targeted merge validation passed; complete community CI remains separate.

### AI Disclosure

- Tool: OpenAI Codex (GPT-6).
- Generated: The three conflict resolutions and this message.
- Human review: Pending author review.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#69032
commit: b9be379cce13471a6615c802eeb8dea3e3e05f9e
base: 43963ce4c9271826121ae1fdf5dde04e3c00e11a
reviewed_at: 2026-11-17T17:10+08:00
reviewer: OIiveirra
model: gpt-6.1-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 1, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • F-01 (Minor, wording; inherited, Regression:no) fe/fe-connector/pom.xml:58 — The PR description still says connector API32/upstream31; implementation and loader consistently use API33, against reviewed-base API32. Update the three description paragraphs; no runtime gate defect found.
  • Complete static review of234 changed files, with27 inherited dismissals reconfirmed. Both83a676b Major repairs remain fixed. No builds, tests, SQL or cluster execution were run during this review.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants