Repository navigation
Conversation
### What problem does this PR solve? Issue Number: N/A Related PR: apache#65851, apache#69488, apache#69499 Problem Summary: ADD COLUMN rejects required fields even when a format-version 3 table can supply a non-null initial and write default. Nested additions also reject defaults in the parser and metadata layer. Reuse full column-definition parsing for nested paths and stage optional or required additions through one SDK helper for top-level, batch, STRUCT, ARRAY element and MAP value additions. Check the loaded table format before committing defaults and preserve existing parent, collision, position and write-schema validation. Correct Boolean and SQL temporal literal conversion on the same default path. Keep the existing SHOW CREATE and catalog default presentation. ### Release note Iceberg format-version 3 tables support adding top-level and nested NOT NULL fields with non-null defaults. Boolean 0/1 and SQL date/timestamp default literals are converted correctly. Older formats continue to reject non-null column defaults. ### Check List (For Author) - Test: Targeted run-fe-ut.sh connector, parser and command suites passed (313 tests); additional typed default and real SDK schema tests passed (30 tests, including repeated coverage). Full build.sh --fe and architecture gates passed before the final typed conversion update. Manual SQL DDL positive and negative cases passed. Parquet/ORC regression suite added; local read/write execution is pending preparation of the native runtime. - Behavior changed: Yes; required v3 additions and nested scalar defaults are supported, and invalid Boolean defaults fail instead of silently becoming false. - Does this need documentation: Yes; the connector contract and examples are included under fe/fe-connector/docs/iceberg-add-column-defaults.md.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#65851, apache#69488 Problem Summary: External ADD COLUMN accepts compact SQL dates and timestamps such as 20240102 and 20240102030405, but their raw numeric strings can be mistaken for connector epoch values. Extend the existing civil TIMESTAMP_NS default normalization to DATE and DATETIME with the same parser used by validation. Preserve fractional precision and keep TIMESTAMPTZ and OLAP behavior unchanged. Add focused normalization tests and top-level/nested regression cases. ### Release note External DATE and DATETIME column defaults preserve the date/time represented by compact SQL literals. ### Check List (For Author) - Test: run-fe-ut.sh passed ColumnDefinitionTest, AlterTableCommandTest and IcebergNestedSchemaEvolutionParserTest, including existing nanosecond normalization coverage. The previous full FE build passed; the final rebuild and Parquet/ORC regression are pending this normalization update. - Behavior changed: Yes; civil external DDL defaults are normalized before connector conversion. - Does this need documentation: Yes; the connector contract is included in this branch.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#65851, apache#69488 Problem Summary: Persisted Iceberg defaults must preserve the instant of a TIMESTAMPTZ SQL literal and the fractional precision of its target type. Resolve zoned literals with the existing session-time-zone conversion helper and convert other parsed temporal literals through the typed literal bridge before passing their canonical value to connectors. Extend real SDK tests and the Parquet/ORC regression with zoned, compact and rounded defaults. Prepare local Hadoop test data directories and check NULL rejection independently of the validation layer that reports it. ### Release note External temporal column defaults preserve compact SQL dates, target fractional precision and the DDL session's time zone for TIMESTAMPTZ values without an explicit offset. ### Check List (For Author) - Test: run-fe-ut.sh passed 81 focused connector, parser, ALTER command and column-definition tests. Manual Parquet/ORC old/new file read-write and required-NULL rejection tests passed. A complete preliminary regression generated outputs successfully; final output generation and comparison await deployment of this last temporal update. Previous full FE builds and architecture gates passed; the final FE rebuild follows this commit. - Behavior changed: Yes; external temporal defaults are canonicalized with their target type and session-zone semantics before connector conversion. - Does this need documentation: Yes; the connector contract is updated in this branch.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#65851, apache#69488 Problem Summary: Record script-generated expected results for required top-level and nested column additions, including old/new Parquet and ORC files, NULL parents, typed and compact literals, session-zone defaults, fractional rounding, negative writes, older formats and batch atomicity. Clarify literal defaults and the optional local filesystem test fixture. ### Release note None ### Check List (For Author) - Test: run-regression-test.sh --run -d external_table_p0/iceberg -s test_iceberg_add_column_defaults -genOut passed, followed by a successful normal comparison. Final build.sh --fe and Checkstyle passed, as did 81 focused FE tests and both connector architecture gates. Local read/write testing used the branch-built FE and an ASAN BE cache; no BE code changed. - Behavior changed: No; this commit adds generated test results and documentation for the feature in this branch. - Does this need documentation: Yes; the connector contract is included and clarified.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28714 ms |
Contributor
TPC-DS: Total hot run time: 151707 ms |
Contributor
ClickBench: Total hot run time: 24.45 s |
…load ### What problem does this PR solve? Issue Number: N/A Related PR: apache#65851, apache#69488, apache#69499 Problem Summary: Required and nested ADD COLUMN defaults must retain their SQL values when converted to Iceberg metadata. Normalize decimal scale, valid Boolean whitespace, compact UUIDs and escaped strings; preserve nanosecond gap semantics and temporal precision while keeping unqualified zoned values compatible with existing connector inputs. Resolve Iceberg zoned defaults through the session timezone and reject non-finite floating defaults before metadata publication. Verify real Hadoop metadata commits/reloads, extend read/write regression coverage and update both existing v2 rejection assertions. ### Release note Typed literal defaults for required and nested Iceberg column additions preserve their values through metadata serialization and reads/writes. Non-finite floating defaults fail before commit. ### Check List (For Author) - Test: 89 focused FE tests passed; full build.sh --fe, Checkstyle and both connector architecture gates passed. The preset regression runner generated the new expected output and then successfully compared all four affected external suites against isolated REST/MinIO services. Local read/write validation used the branch-built FE and an ASAN BE cache; BE code is unchanged. - Behavior changed: Yes; correct typed default conversion and prevent publication of unreadable floating defaults. Existing SHOW CREATE default presentation remains unchanged. - Does this need documentation: Yes; the connector contract and fixture requirements are updated in this commit.
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28466 ms |
Contributor
TPC-DS: Total hot run time: 151615 ms |
Contributor
ClickBench: Total hot run time: 24.18 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#65851, apache#69488 Problem Summary: Equivalent explicit and session-zone TIMESTAMPTZ defaults can differ by one hour when fractional rounding crosses a DST overlap. A valid civil timestamp can also resolve outside the UTC range supported by Doris and become an unreadable default after schema commit. Preserve civil precision until the existing Iceberg session boundary, round the resolved instant at the original declared precision, and reject unsupported final UTC years before staging. Share the conversion across single, batch and nested additions and retain unparameterized SPI epoch microseconds. ### Release note Iceberg zoned column defaults consistently interpret explicit and implicit time zones across DST rounding and reject unsupported UTC values before committing schema changes. ### Check List (For Author) - Test: 91 focused FE tests passed; full build.sh --fe, Checkstyle and both connector architecture gates passed. The preset regression runner generated expected results and passed the four affected external suites, including old/new files, explicit/session CET equality, UTC upper/lower rejection and batch atomicity. Local native validation uses an existing ASAN BE cache; C++ code is unchanged. - Behavior changed: Yes; zoned ADD COLUMN defaults resolve time zones before target rounding and validate the final UTC range. Existing default presentation is preserved. - Does this need documentation: Yes; the connector contract describes the precision and UTC-range policy.
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28772 ms |
Contributor
TPC-DS: Total hot run time: 151587 ms |
Contributor
ClickBench: Total hot run time: 24.33 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
…bounds ### What problem does this PR solve? Issue Number: N/A Related PR: apache#70220 Problem Summary: Compact TIMESTAMPTZ defaults such as 20240102030405.123456 lose their fractional seconds when civil precision is inferred as zero. Parse unqualified civil defaults explicitly at microsecond precision before session-zone binding and target rounding. Zone conversion or rounding can also produce year-zero dates that schema metadata accepts but physical Parquet timestamp reads turn into NULL. Validate DATETIME and final UTC TIMESTAMPTZ values against the readable 0001-9999 range before staging single, batch or nested additions. ### Release note Iceberg column defaults preserve compact timestamp fractions and reject DATETIME and final UTC TIMESTAMPTZ values outside years 0001 through 9999 before committing schema changes. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 92 focused FE tests passed; full FE build and Checkstyle passed. - Generated defaults-suite output and normal comparison of four Iceberg suites passed, including Parquet/ORC old/new files and minimum/maximum civil and UTC timestamp defaults. - Both connector architecture gates passed. Read/write verification uses the branch-built FE and an ASAN BE cache; a full local BE build remains blocked by a required submodule clone. - Behavior changed: Yes; preserve compact fractions and reject unsupported final UTC dates before metadata publication. - Does this need documentation: Yes; update the included connector contract.
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28769 ms |
Contributor
TPC-DS: Total hot run time: 151480 ms |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#70220 Problem Summary: The existing DEFAULT validator accepts compact fractional timestamps without a decimal point, including forms with or without T and explicit offsets or named zones. External DATETIME and TIMESTAMPTZ normalization reparses the raw spelling with a formatter that requires the decimal point, rejecting valid ADD COLUMN statements. Insert the fractional separator lexically before timezone detection while retaining all validated digits, the original zone and declared precision. Keep existing zone-first rounding and timestamp range checks. Add equivalent-spelling unit controls and persistent read/write cases across single, batch and nested additions; copy non-null nested defaults into new physical files to verify their values survive. ### Release note External DATETIME and TIMESTAMPTZ column defaults accept validated compact fractional timestamps without a decimal point. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 93 focused FE tests passed; full FE build and Checkstyle passed. - Generated defaults-suite output and normal comparison of four Iceberg suites passed, including Parquet/ORC old/new files and non-null nested defaults copied into physical files. - Both connector architecture gates passed. Read/write checks use the branch-built FE and an ASAN BE cache; a full local BE build remains blocked by a required submodule clone. - Behavior changed: Yes; retain accepted compact timestamp DEFAULT spellings and their precision/timezone semantics. - Does this need documentation: Yes; update the included connector contract.
Contributor
Author
|
run buildall |
Contributor
ClickBench: Total hot run time: 24.52 s |
Contributor
Author
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#70220
commit: 431539613dc21f83e330776d146c3345ce471108
base: 380912cc7344e92375c3de58ec37c99602cad013
reviewed_at: 2026-11-05T05:22+00:00
reviewer: hubgeter
model: gpt-6.1-sol
effort: max
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
Contributor
TPC-H: Total hot run time: 28897 ms |
Contributor
TPC-DS: Total hot run time: 152234 ms |
Contributor
ClickBench: Total hot run time: 24.35 s |
hubgeter
marked this pull request as ready for review
November 9, 2026 09:54
hubgeter
requested review from
924060929,
CalvinKirs,
englefly,
morrySnow and
starocean999
as code owners
November 9, 2026 09:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: #65851, #69488, #69499
Problem Summary:
ALTER TABLE ... ADD COLUMN added INT NOT NULL DEFAULT 7is rejected even on an Iceberg format-version 3 table. Nested additions also reject DEFAULT in their separate grammar and metadata validation.Reuse the full column definition for nested paths and share SDK staging for top-level single/batch additions, STRUCT children, ARRAY element children and MAP value children. Preserve requiredness and set both initial and write defaults. Validate the loaded table's format version before committing; retain batch atomicity, path/position validation and statement-bound write-schema checks. NULL parents remain NULL, and explicit NULL in a supplied required field remains an error.
Normalize defaults to their typed metadata representation: trim valid Boolean spellings, exactly rescale decimals, canonicalize compact UUIDs, decode and quote strings according to SQL mode, and preserve SQL temporal precision and the existing nanosecond gap policy. Unqualified TIMESTAMPTZ values retain civil text across the generic connector boundary; Iceberg resolves them using the existing session timezone API, then applies the requested precision so equivalent explicit/session zones agree across DST carries. Compact fractional timestamps retain all six digits before zone binding. Validated compact spellings with or without T or a decimal point are normalized lexically before zone detection, retaining the original offset or named zone and declared precision. DATETIME defaults and final UTC TIMESTAMPTZ defaults must fit years 0001 through 9999 before staging, matching physical Parquet timestamp reads as well as historical default materialization. Year-zero physical timestamps would otherwise read as NULL even when the historical default is valid. Reject non-finite floating defaults before they can publish schema JSON that cannot reload. Iceberg SHOW CREATE, DESC and information_schema default presentation retain their current behavior.
Release note
Iceberg format-version 3 tables support adding top-level and nested NOT NULL fields with non-null literal defaults. Typed defaults preserve their SQL values through metadata serialization and reads/writes. Compact DATETIME and TIMESTAMPTZ defaults accept fractional seconds without a decimal point. Format-version 1 and 2 tables continue to reject non-null column defaults. Non-finite floating defaults are rejected before commit, as are DATETIME defaults or final UTC TIMESTAMPTZ defaults outside years 0001 through 9999.
Check List (For Author)
Validation:
Final focused
run-fe-ut.shrun: 93 tests passed (37 connector, 56 core/parser/command), with no failures or skips. HadoopCatalog tests commit and reload real metadata, including decimal/string/UUID/session-zone defaults, DST carries, negative epoch rounding, UTC bounds, rejection of year-zero values and failure atomicity.Final
build.sh --fe, Checkstyle and both connector architecture gates: passed (0 Checkstyle violations).run-regression-test.sh --run -d external_table_p0/iceberg: all four suites passed in normal comparison: the new defaults suite and the three existing suites listed below. The new 146-line output was generated by the preset runner against the existing newer REST/MinIO fixture; Parquet/ORC old/new files, nested defaults, compact fractional timestamps with and without a decimal point, copied non-null nested defaults in physical new files, nanosecond DEFAULT versus CAST, explicit/implicit DST equality, UTC/date rejection, minimum/maximum civil and UTC timestamp reads/writes, SQL modes, string HEX values and failure atomicity were verified.Existing REST suites
iceberg_schema_change_ddl,test_iceberg_nested_schema_evolution_ddlandtest_iceberg_schema_metadata_atomicity_matrixpassed.Local read/write validation uses the branch-built FE with an ASAN BE cache (BE code is unchanged). A full local BE build was blocked while cloning a required submodule.
Behavior changed:
Does this need documentation?
fe/fe-connector/docs/iceberg-add-column-defaults.md.Check List (For Reviewer who merge this PR)