Skip to content

[fix](variant) Make default_variant_* session variables test-only and fix CREATE TABLE LIKE of VARIANT - #70501

Merged
csun5285 merged 1 commit into
apache:masterfrom
csun5285:variant-like-session-defaults
Nov 16, 2026
Merged

csun5285 merged 1 commit into
apache:masterfrom
csun5285:variant-like-session-defaults

Conversation

@csun5285

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #59183

Problem Summary:

CREATE TABLE LIKE prints the source table as DDL text (the SHOW CREATE TABLE text) and parses it again. The text of a VARIANT column omits variant_enable_doc_mode and variant_enable_nested_group when they are false, and the parser fills every property missing from a DDL from the default_variant_* session variables. So the copy depended on the session that ran the statement:

CREATE TABLE src (id INT, v VARIANT<'a.b':INT, PROPERTIES("variant_max_subcolumns_count"="3")>)
DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num"="1");
SET default_variant_enable_doc_mode = true;
CREATE TABLE dst LIKE src;
SHOW CREATE TABLE dst;
-- before: v variant<MATCH_NAME 'a.b':int,PROPERTIES ("variant_enable_doc_mode" = "true",
--         "variant_doc_materialization_min_rows" = "0","variant_doc_hash_shard_count" = "64")>
--         (doc mode, the subcolumn count 3 is gone)
-- after:  the SET fails; with the default session v is as in src

The default_variant_* session variables exist for the fuzzy tests only. Fix:

  1. The parser reads the default_variant_* session variables only when the FE runs with enable_debug_points = true, which every regression pipeline and docker test cluster sets. Otherwise, and without a session, the properties a DDL leaves out take the built-in values (a parse without a session used its own constants before, such as subcolumn count 0 and nested group on). It does not use the global values either: a value set with SET GLOBAL on an older version still loads from an image. So a DDL text, such as the one CREATE TABLE LIKE prints and parses, creates the same column in every session.
  2. The variables are invisible in SHOW VARIABLES, and SET / SET GLOBAL / SET_VAR of them fails unless debug points are enabled. Users state VARIANT properties in the column definition.
  3. A nested group VARIANT type prints only variant_enable_nested_group, as a doc mode type prints only the doc mode properties.

With debug points enabled, CREATE TABLE LIKE still follows the session; a test that copies a VARIANT table sets the variables first, as the existing suites do.

Release note

The default_variant_* session variables (default_variant_max_subcolumns_count, default_variant_enable_typed_paths_to_sparse, default_variant_max_sparse_column_statistics_size, default_variant_sparse_hash_shard_count, default_variant_enable_doc_mode, default_variant_doc_materialization_min_rows, default_variant_doc_hash_shard_count, default_variant_enable_nested_group) can no longer be set and are not shown by SHOW VARIABLES; set the properties in the VARIANT column definition instead. A value set with SET GLOBAL before the upgrade no longer takes effect. CREATE TABLE LIKE copies the VARIANT properties of the source table regardless of the session.

Check List (For Author)

  • Test: Unit Test / Regression test / Manual test
    • FE UT: NereidsParserTest (session values used only with debug points), CreateTableLikeTest (LIKE under doc mode and nested group session defaults), VariableMgrTest (SET fails, hidden, forwarded values, replay), TypeTest (nested group text, defaults without a session), PropertyAnalyzerTest
    • Regression test: variant_p0/test_variant_create_table_like
    • Manual test: the DDL above on a local cluster; with enable_debug_points = false, SET, SET GLOBAL and SET_VAR of default_variant_* fail, SHOW VARIABLES does not list them, and with use_fuzzy_session_variable = true the fuzzed session values do not reach new VARIANT columns
  • Behavior changed: Yes. SET of the default_variant_* session variables fails and their values are ignored unless enable_debug_points is on; SHOW CREATE TABLE of a nested group VARIANT column prints only variant_enable_nested_group.
  • Does this need documentation: Yes, VARIANT.md lists the session variables (doc PR to follow)

Background

What problem does this PR solve?

Issue Number:
Related PR:
PR Type:

Problem Summary:

Details

Testing

Notes for Reviewers

Release note

Check List (For Author)

  • Behavior changed:
  • Does this need documentation:
  • Pre-submission status:

AI Disclosure

  • Tool:
  • Generated:
  • Human review:

… fix CREATE TABLE LIKE of VARIANT

### What problem does this PR solve?

Issue Number: None

Related PR: apache#59183

Problem Summary:

CREATE TABLE LIKE prints the source table as DDL text (the SHOW CREATE TABLE text) and parses it again.
The text of a VARIANT column omits `variant_enable_doc_mode` and `variant_enable_nested_group` when they
are false, and the parser fills every property missing from a DDL from the `default_variant_*` session
variables. So the copy depended on the session that ran the statement:

```sql
CREATE TABLE src (id INT, v VARIANT<'a.b':INT, PROPERTIES("variant_max_subcolumns_count"="3")>)
DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES("replication_num"="1");
SET default_variant_enable_doc_mode = true;
CREATE TABLE dst LIKE src;
SHOW CREATE TABLE dst;
-- before: v variant<MATCH_NAME 'a.b':int,PROPERTIES ("variant_enable_doc_mode" = "true",
--         "variant_doc_materialization_min_rows" = "0","variant_doc_hash_shard_count" = "64")>
--         (doc mode, the subcolumn count 3 is gone)
-- after:  the SET fails; with the default session v is as in src
```

`default_variant_enable_nested_group = true` turned an ordinary or a doc mode column into a nested
group column the same way. The fuzzy regression runs set `default_variant_enable_doc_mode` at random
per connection, so CREATE TABLE LIKE of a VARIANT table was random there too.

Separately, SHOW CREATE TABLE printed a nested group VARIANT column (master only, apache#64680) with the
subcolumn properties nested group clears, including `variant_max_sparse_column_statistics_size = 0`,
which does not parse: CREATE TABLE LIKE of such a table always failed with
"variant_max_sparse_column_statistics_size must between 1 and 50000".

The `default_variant_*` session variables exist for the fuzzy tests only. Fix:
1. The parser reads the `default_variant_*` session variables only when the FE runs with
   `enable_debug_points = true`, which every regression pipeline and docker test cluster sets.
   Otherwise, and without a session, the properties a DDL leaves out take the built-in values (a parse
   without a session used its own constants before, such as subcolumn count 0 and nested group on). It
   does not use the global values either: a value set with SET GLOBAL on an older version still loads
   from an image. So a DDL text, such as the one CREATE TABLE LIKE prints and parses, creates the same
   column in every session.
2. The variables are invisible in SHOW VARIABLES, and SET / SET GLOBAL / SET_VAR of them fails unless
   debug points are enabled. Users state VARIANT properties in the column definition.
3. A nested group VARIANT type prints only `variant_enable_nested_group`, as a doc mode type prints
   only the doc mode properties.

With debug points enabled, CREATE TABLE LIKE still follows the session; a test that copies a VARIANT
table sets the variables first, as the existing suites do.

### Release note

The `default_variant_*` session variables (`default_variant_max_subcolumns_count`,
`default_variant_enable_typed_paths_to_sparse`, `default_variant_max_sparse_column_statistics_size`,
`default_variant_sparse_hash_shard_count`, `default_variant_enable_doc_mode`,
`default_variant_doc_materialization_min_rows`, `default_variant_doc_hash_shard_count`,
`default_variant_enable_nested_group`) can no longer be set and are not shown by SHOW VARIABLES; set the
properties in the VARIANT column definition instead. A value set with SET GLOBAL before the upgrade no
longer takes effect. CREATE TABLE LIKE copies the VARIANT properties of the source table regardless of
the session.

### Check List (For Author)

- Test: Unit Test / Regression test / Manual test
    - FE UT: NereidsParserTest (session values used only with debug points), CreateTableLikeTest
      (LIKE under doc mode and nested group session defaults), VariableMgrTest (SET fails, hidden,
      forwarded values, replay), TypeTest (nested group text, defaults without a session),
      PropertyAnalyzerTest
    - Regression test: variant_p0/test_variant_create_table_like
    - Manual test: the DDL above on a local cluster; with enable_debug_points = false, SET, SET GLOBAL
      and SET_VAR of default_variant_* fail, SHOW VARIABLES does not list them, and with
      use_fuzzy_session_variable = true the fuzzed session values do not reach new VARIANT columns
- Behavior changed: Yes. SET of the default_variant_* session variables fails and their values are
  ignored unless enable_debug_points is on; SHOW CREATE TABLE of a nested group VARIANT column prints
  only variant_enable_nested_group.
- Does this need documentation: Yes, VARIANT.md lists the session variables (doc PR to follow)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@csun5285

Copy link
Copy Markdown
Contributor Author

/review

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #70501 at 9bc256576a1ed99a6f775d4dd2062f6cc2586636: one P2 finding in the shared session-variable checker. The parser and both VARIANT renderers implement the intended production defaults and nested-group SQL normalization, but the new SET restriction also rejects internal reset/rollback operations. See the inline finding for the upgrade and hint-cleanup paths.

Reviewed all 11 changed paths, related callers and consumers, added FE tests, and the regression suite/output. There were no existing inline findings to carry forward and no additional user-provided focus points.

The relevant schema flow is:

CREATE TABLE LIKE
  -> render source schema under table read lock
  -> release lock
  -> NereidsParser -> visitVariantPredefinedFields
  -> built-in defaults + explicit properties -> mode normalization
  -> normal CREATE

The variable lifecycle has two distinct loading paths: image loading assigns annotated fields directly, whereas journal replay invokes checkers and catches failures per variable. New sessions clone the image-loaded defaults. This distinction exposes the inline issue.

Required checkpoint Conclusion from static inspection
Goal and evidence Isolates production VARIANT defaults from sessions and fixes nested-group LIKE rendering. Parser, LIKE and type tests exercise the intended behavior; internal restoration remains defective.
Scope and clarity Focused changes to parser default selection, two corresponding renderers, eight existing variable annotations/checkers, and tests.
Concurrency and locks No new threads or lock ordering. Built-in defaults are privately held and read only. LIKE releases the source table lock before reparsing and creating.
Lifecycle and static initialization Java safely publishes the built-in SessionVariable; its relevant fields have literal defaults and the object is never mutated. Internal session restoration is the reported issue.
Configuration and dynamic updates No config added. Existing enable_debug_points is immutable; production ignores session defaults, while debug-enabled tests intentionally retain them.
Compatibility and rolling upgrades No storage/wire field layout changes. The intentional SET/SHOW behavior change is documented, but legacy image values expose the reset regression.
Thrift/protobuf IDs No gensrc changes or added IDs/enums.
Parallel paths Both catalog and Nereids VARIANT renderers updated consistently. Checked bare/property-bearing VARIANT, SQL type parsing, forwarding, fuzzy initialization, SET, SET GLOBAL, SET_VAR, reset, image and journal paths.
Conditional checks Comments explain the production/debug branch; nested-group omission agrees with existing parser normalization. The shared validation guard is too broad for internal restoration.
Test coverage Added tests cover production/debug defaults, hostile session modes during LIKE, nested-group rendering, all eight variables, forwarding and journal replay. Missing image/reset and rejected multi-hint cleanup coverage relates to the inline finding.
Changed expected results DESC output follows fixed schema order and matches ordinary, doc-mode and nested-group properties. Output contents inspected; generation provenance and runtime results were not independently verified.
Observability Existing user-facing exceptions/logging are sufficient for this local change; no new operational metrics needed.
Transactions and persistence Metadata/edit-log schemas unchanged. Reviewed image versus journal replay and checkpoint separation; the reset issue comes from retained image-loaded values. No new transaction/failover protocol.
Data writes and atomicity No row-write/publish changes; partition visibility and delete-bitmap invariants are unaffected. Internal reset/rollback can stop partway through, as reported.
FE/BE variable passing No new transmitted variables. Existing needForward fields and catalog/type conversion remain; production parsing ignores forwarded test defaults.
Performance One built-in object allocated at class initialization; no per-column SessionVariable allocation or new row-level work. No material performance issue identified.
Other issues No additional verified defect after reviewing failure cleanup and special/parallel paths; related restoration manifestations are consolidated in one inline finding.

Review converged in two rounds after independent coverage and a final challenge of the frozen finding set. No unresolved candidates or uncovered changed-file scope remain. Validation was static only: no builds, tests, benchmarks, or runtime checks were executed.

Review converged after 2 round(s).

@csun5285

Copy link
Copy Markdown
Contributor Author

run cloudut

@csun5285

Copy link
Copy Markdown
Contributor Author

run feut

1 similar comment
@csun5285

Copy link
Copy Markdown
Contributor Author

run feut

@csun5285

Copy link
Copy Markdown
Contributor Author

run complie

@csun5285

Copy link
Copy Markdown
Contributor Author

run buildall

@csun5285

Copy link
Copy Markdown
Contributor Author

/reveiw

@csun5285

Copy link
Copy Markdown
Contributor Author

run external

@csun5285
csun5285 merged commit 0e8b56e into apache:master Nov 16, 2026
36 checks passed
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 16, 2026
…ANT properties

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812, apache#70501, apache#70488

Problem Summary:

After rebasing onto master, three tests that master added meet the changes of this PR, and two tests of
this PR meet apache#70501:

- VariableMgrTest (apache#70501) lists every `default_variant_*` session variable, including
  `default_variant_doc_materialization_min_rows` and `default_variant_doc_hash_shard_count`, which this PR
  renames and removes, so it no longer compiles. It now lists `default_variant_subcolumn_extraction_min_rows`.
- test_variant_create_table_like (apache#70501) creates a doc mode column with
  `variant_doc_materialization_min_rows` and `variant_doc_hash_shard_count`, which this PR no longer
  accepts. It uses `variant_subcolumn_extraction_min_rows` and `variant_sparse_hash_shard_count`, and pins
  the subcolumn count and the row threshold, which DESC of a doc mode column now prints and fuzzy mode
  picks at random. Its .out is regenerated by the framework: the new names, the count of the doc mode
  column, and no `variant_enable_typed_paths_to_sparse`; the copy still prints the same as the source.
- test_search_bkd_skip_threshold (apache#70488) needs `v.n` and `v.p` extracted with their index in a 10-row
  data file. Fuzzy mode now also picks the row threshold without doc mode, and a threshold above 10
  extracts no path, so SEARCH fails with "SEARCH index unavailable for Variant field 'v.n'". It pins the
  threshold to 0, as the other suites that need an index on every path do.
- VariantSerializeTypeDdlTest and VariantLoadParseInjectionTest create a column with
  `variant_enable_typed_paths_to_sparse = true` through the session variable, the only way to get it
  now. Since apache#70501 the parser reads the `default_variant_*` session variables only with debug points
  enabled, which FE UT does not set, so the column got false: the first test failed, and the second
  passed without covering the property. Both enable debug points while they create the table.

### Release note

None

### Check List (For Author)

- Test: Unit Test, Regression test
    - FE UT: the 11 classes that the VARIANT properties touch plus VariableMgrTest and
      CreateTableLikeTest, 335 tests, 1 failure (testModifyColumnKeepsTypedPathsToSparse, fixed above);
      after the fix VariantSerializeTypeDdlTest and VariantLoadParseInjectionTest run again, 10 tests,
      0 failures.
    - Regression: test_variant_create_table_like (.out regenerated with -forceGenOut, then run again) and
      test_search_bkd_skip_threshold pass. With the global `default_variant_subcolumn_extraction_min_rows`
      set to 15, test_search_bkd_skip_threshold fails with the error above without the pin and passes
      with it.
- Behavior changed: No (tests only).
- Does this need documentation: No.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hello-stephen pushed a commit that referenced this pull request Nov 16, 2026
…70639)

### Background

VARIANT tables with escaped dotted keys reject NestedGroup columns. The
parser only applies the test-only `default_variant_*` session variables
when debug points are enabled.

### What problem does this PR solve?

Issue Number: None
Related PR: #70501 (added the debug-point gate for VARIANT session
defaults)
PR Type: Test

Problem Summary:

`VariantEscapeTableTest.testNestedGroupFinalTypeIsForbidden` sets the
NestedGroup session default without enabling debug points. The parser
therefore uses the built-in default, creates the table, and the
expected-exception assertion fails. Enable debug points for the
session-default case and restore both the original configuration and
session value in `finally`.

### Details

- **Failure:** The original test fails at the `nested_default` CREATE
TABLE assertion with "Expected java.lang.Exception to be thrown, but
nothing was thrown."
- **Cause:** The test did not enable the debug-point gate introduced by
#70501.
- **Stability evidence:** The failure was reproduced on master before
the change. After the change, the complete test class and the adjacent
parser and CREATE TABLE LIKE test classes pass. Both modified values are
restored even if the assertion fails.

### Testing

On the remote Linux development worktree:

- Before the fix: `./run-fe-ut.sh --run
org.apache.doris.catalog.VariantEscapeTableTest#testNestedGroupFinalTypeIsForbidden`
fails at the expected-exception assertion.
- After the fix: `./run-fe-ut.sh --run
org.apache.doris.catalog.VariantEscapeTableTest,org.apache.doris.catalog.CreateTableLikeTest,org.apache.doris.nereids.parser.NereidsParserTest`
passes all 115 tests (9 + 14 + 92), with no failures, errors, or skips.
- `PATH=~/agent-bin:$PATH build-support/clang-format.sh` completes
successfully.
- `mvn checkstyle:check -pl fe-core` reports 0 violations.

### Release note

None.

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: OpenAI Codex
- Generated: Test fix and PR description.
- Human review: Not yet performed.
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 16, 2026
…ning

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

This is the VARIANT part of apache#69812, split out so that it can be picked to branch-4.2 on its own, and it gives the VARIANT properties the meaning that the VARIANT tuning guide describes. Every user-visible change below shows 4.1 and this PR, each as one SQL script with its result in comments. The results of this PR come from running the scripts; those of 4.1 come from the code of branch-4.1. `SET describe_extend_variant_column = true` makes DESC list the subcolumns that the data files extracted. Changes that one script cannot show are listed at the end.

#### 1. `variant_max_subcolumns_count = 0` on a new column extracts no dynamic path

4.1: 0 means no limit, so every path is extracted.

```sql
CREATE TABLE t1 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_max_subcolumns_count" = "0")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": "x"}'));
SET describe_extend_variant_column = true;
DESC t1;
-- v, v.a, v.b, v.t: every path is a subcolumn
```

This PR: 0 keeps every dynamic path in the sparse column. Typed paths are still extracted.

```sql
CREATE TABLE t1 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_max_subcolumns_count" = "0")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": "x"}'));
SET describe_extend_variant_column = true;
DESC t1;
-- v, v.t: only the typed path is a subcolumn; a and b stay in the sparse column
SELECT v FROM t1;
-- {"a":1,"b":"x","t":1}
```

#### 2. `variant_doc_materialization_min_rows` is renamed to `variant_subcolumn_extraction_min_rows`

4.1:

```sql
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_doc_materialization_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t2;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "100",
--                        "variant_doc_hash_shard_count" = "64")> NULL
```

This PR: the old name is refused.

```sql
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_doc_materialization_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_max_subcolumns_count and
-- variant_max_sparse_column_statistics_size and variant_sparse_hash_shard_count and variant_enable_doc_mode
-- and variant_subcolumn_extraction_min_rows and variant_enable_nested_group
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_subcolumn_extraction_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t2;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "100","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 3. `variant_doc_hash_shard_count` is replaced by `variant_sparse_hash_shard_count`

4.1: doc mode has its own shard count property and refuses `variant_sparse_hash_shard_count`.

```sql
CREATE TABLE t3 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_doc_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t3;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "0",
--                        "variant_doc_hash_shard_count" = "8")> NULL
CREATE TABLE t3b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_sparse_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_sparse_hash_shard_count and
-- variant_enable_doc_mode cannot be set together
```

This PR: `variant_sparse_hash_shard_count` sets the shard count in both modes; the doc mode name is refused.

```sql
CREATE TABLE t3 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_doc_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_max_subcolumns_count and ... (as in 2)
CREATE TABLE t3b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_sparse_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t3b;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "8")> NULL
```

#### 4. The row threshold works without doc mode

4.1: the threshold exists only in doc mode, as `variant_doc_materialization_min_rows`. Without doc mode the property is dropped.

```sql
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_subcolumn_extraction_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_enable_typed_paths_to_sparse and
-- variant_max_subcolumns_count and variant_max_sparse_column_statistics_size and variant_sparse_hash_shard_count
-- and variant_enable_doc_mode and variant_doc_materialization_min_rows and variant_doc_hash_shard_count
-- and variant_enable_nested_group
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_doc_materialization_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- accepted, but the threshold is dropped: SHOW CREATE TABLE does not show it, and every load extracts its paths
```

This PR: `variant_subcolumn_extraction_min_rows` works with and without doc mode. A data file with fewer rows extracts no dynamic path.

```sql
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_subcolumn_extraction_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t4 VALUES (1, parse_to_variant('{"t": 1, "a": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t4;
-- v, v.t: the 2-row data file is below the threshold and extracts no dynamic path
INSERT INTO t4 VALUES (3, parse_to_variant('{"b": 3}')), (4, parse_to_variant('{"b": 4}')),
                      (5, parse_to_variant('{"b": 5}'));
DESC t4;
-- v, v.b, v.t: the 3-row data file extracts b
```

#### 5. Doc mode takes the subcolumn count and the shard count, and extracts up to the count

4.1: doc mode cannot be combined with the subcolumn count or the shard count, even with `"variant_enable_doc_mode" = "false"`, and it extracts every path once a data file reaches the threshold.

```sql
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true",
                                                        "variant_max_subcolumns_count" = "1")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_max_subcolumns_count and
-- variant_enable_doc_mode cannot be set together
CREATE TABLE t5 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "false", "variant_max_subcolumns_count" = "1")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- the same error; variant_sparse_hash_shard_count is refused the same way
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t5 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t5;
-- v, v.a, v.b, v.t: every path is a subcolumn
```

This PR: doc mode is a separate switch. It extracts the typed paths and up to the count of dynamic paths, the most frequent first; the count does not include typed paths, and 0 extracts none. The document keeps every path.

```sql
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true",
                                                        "variant_max_subcolumns_count" = "1",
                                                        "variant_sparse_hash_shard_count" = "4")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t5 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t5;
-- v, v.a, v.t: the typed path and the most frequent dynamic path (a: 2 rows, b: 1 row)
-- with "variant_max_subcolumns_count" = "0": v, v.t
SELECT v FROM t5 ORDER BY k;
-- {"a":1,"b":1,"t":1}
-- {"a":2,"t":2}
CREATE TABLE t5b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "false", "variant_max_subcolumns_count" = "10")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- accepted
```

#### 6. Doc mode: the default shard count is 1, and SHOW CREATE TABLE prints the count

4.1: the document is split into 64 columns by default.

```sql
CREATE TABLE t6 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t6;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "0",
--                        "variant_doc_hash_shard_count" = "64")> NULL
```

This PR: 1 by default, as without doc mode.

```sql
CREATE TABLE t6 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t6;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 7. `variant_enable_typed_paths_to_sparse` cannot be set to `true` and is not printed

4.1:

```sql
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t7;
-- `v` variant<'t':int,PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_enable_typed_paths_to_sparse" = "true",
--                                "variant_max_sparse_column_statistics_size" = "10000","variant_sparse_hash_shard_count" = "1")> NULL
-- typed paths compete with dynamic paths for variant_max_subcolumns_count
```

This PR: the property is internal. It still takes `false`, which released versions print for every column without doc mode, so their DDL still runs.

```sql
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_enable_typed_paths_to_sparse = true is not supported
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "false")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t7;
-- `v` variant<'t':int,PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_max_sparse_column_statistics_size" = "10000",
--                                "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 8. MODIFY COLUMN can change the row threshold without doc mode

4.1: only the threshold of a doc mode column can be changed (apache#61100); a column without doc mode has none.

```sql
CREATE TABLE t8 (k INT, v VARIANT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
ALTER TABLE t8 MODIFY COLUMN v VARIANT<PROPERTIES("variant_subcolumn_extraction_min_rows" = "100")>;
-- ERROR 1235 (42000): Not Supported: only support for variant_enable_typed_paths_to_sparse and ... (as in 4)
```

This PR:

```sql
CREATE TABLE t8 (k INT, v VARIANT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
ALTER TABLE t8 MODIFY COLUMN v VARIANT<PROPERTIES("variant_subcolumn_extraction_min_rows" = "100")>;
-- the job finishes (SHOW ALTER TABLE COLUMN)
SHOW CREATE TABLE t8;
-- `v` variant<PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_max_sparse_column_statistics_size" = "10000",
--                        "variant_subcolumn_extraction_min_rows" = "100","variant_sparse_hash_shard_count" = "1")> NULL
INSERT INTO t8 VALUES (1, parse_to_variant('{"a": 1}'));
SET describe_extend_variant_column = true;
DESC t8;
-- v: the 1-row data file is below the threshold
```

#### 9. CTAS of a VARIANT expression creates a column that extracts no dynamic path

4.1: the type of `parse_to_variant(...)` has a count of 0, which extracts every path, and the new column takes it.

```sql
CREATE TABLE t10 PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, parse_to_variant('{"a": 1, "b": 2}') AS v;
SET describe_extend_variant_column = true;
DESC t10;
-- v, v.a, v.b: every path is a subcolumn
```

This PR: the new column is V2 and keeps the count of 0, so it extracts no dynamic path. Casting the expression to `VARIANT` gives it the default count of a declared column, 2048.

```sql
CREATE TABLE t10 PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, parse_to_variant('{"a": 1, "b": 2}') AS v;
SET describe_extend_variant_column = true;
DESC t10;
-- v: variant<PROPERTIES ("variant_max_subcolumns_count" = "0",...)>, no subcolumn
CREATE TABLE t10b PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, CAST(parse_to_variant('{"a": 1, "b": 2}') AS VARIANT) AS v;
DESC t10b;
-- v: variant<PROPERTIES ("variant_max_subcolumns_count" = "2048",...)>, v.a, v.b
```

#### Changes that one SQL script cannot show

- **Upgrade.** The whole cluster has to be stopped; FE and BE of different versions must not run together. An older BE does not know the serialize type below and extracts every path of a new column with a count of 0, and the new reader fails an assertion on such a data file. Downgrading is not supported: an older version reads that count as "extract every path", and its compaction drops the paths that only the sparse column holds.
- **Columns created before the upgrade keep their behavior.** A hidden serialize type tells new columns (V2) from older ones (V1); DESC and SHOW CREATE TABLE do not print it. It is sent as `TColumn.variant_serialize_type` (31) and stored as `ColumnPB.variant_serialize_type` (36), both optional with default V1, so every existing column is V1. On a V1 column a count of 0 still extracts every path, a doc mode column created by 4.1 still extracts every path once a data file reaches its threshold, and a column with `variant_enable_typed_paths_to_sparse = true` keeps it (MODIFY COLUMN keeps it too). Every column created after the upgrade is V2, also with ADD COLUMN, CTAS and CREATE TABLE LIKE, and also when the SHOW CREATE TABLE output of an old table is run again: a stated count of 0 then extracts no dynamic path, and `variant_enable_typed_paths_to_sparse = true` is not carried over. CTAS copies the count of the query's type, so a count of 0 of an old column extracts no dynamic path in the new column (9 shows the same for a VARIANT expression).
- **MODIFY COLUMN on an older column without doc mode whose count is 0** refuses a non-zero threshold ("Can not set variant_subcolumn_extraction_min_rows on a column of an older version whose variant_max_subcolumns_count is 0"): there 0 extracts every path, and that compaction branch only builds the subcolumns its inputs extracted, so the paths that a data file below the threshold left in the sparse column would be lost. An older column with a count above 0 takes a threshold like a new one.
- **Compaction.** It applies the threshold to the rows of its inputs. Below it, the subcolumns that inputs extracted go back to the sparse column instead of being dropped; this happens after an ALTER raises the threshold, or when an earlier compaction extracted them from more rows than it kept. At or above it, it extracts the most frequent paths of all inputs. In doc mode every doc bucket of the output has its own writer, so compaction chooses the dynamic paths once from the doc statistics of all inputs; in 4.1 every bucket extracted every path.
- **DDL between versions.** The SHOW CREATE TABLE output of a 4.1 doc mode table has to be renamed (2, 3) before it runs. The output of this version uses `variant_subcolumn_extraction_min_rows`, which an older FE, for example the target of CCR, does not accept.
- **SEARCH on recent data.** SEARCH needs an index on the subcolumn of the path. A data file below the threshold has no subcolumns, so SEARCH on it fails with "SEARCH index unavailable for Variant field" until compaction extracts the path, as it already does for a path beyond the count. MATCH falls back to reading the values unless `enable_match_without_inverted_index = false`.
- **Fuzzy tests.** Fuzzy mode now also randomizes the row threshold without doc mode and `default_variant_enable_typed_paths_to_sparse` (never with doc mode, which refuses it). The suites that print the subcolumn layout, or need an index on every path, pin them.
- **Test-only session variables.** Since apache#70501 the `default_variant_*` session variables can only be set with `enable_debug_points = true`, as in the tests. This PR renames `default_variant_doc_materialization_min_rows` to `default_variant_subcolumn_extraction_min_rows` and removes `default_variant_doc_hash_shard_count`; `default_variant_sparse_hash_shard_count` sets the shard count of both modes.

### Release note

On a new VARIANT column, `variant_max_subcolumns_count = 0` keeps every dynamic path in the sparse encoding instead of extracting it; existing columns keep the old meaning. `variant_subcolumn_extraction_min_rows` replaces `variant_doc_materialization_min_rows`, and `variant_sparse_hash_shard_count` now also sets the shard count of doc mode in place of `variant_doc_hash_shard_count`. The two doc mode names are not accepted any more, so the SHOW CREATE TABLE output of a 4.1 doc mode table has to be renamed before it runs. Doc mode can be combined with the subcolumn count and the row threshold, and the row threshold also works without doc mode. New doc mode columns extract up to the subcolumn count instead of every path, doc mode columns created by 4.1 keep extracting every path, and the default shard count of doc mode is 1 instead of 64. `variant_enable_typed_paths_to_sparse` cannot be set to `true` any more and is no longer printed; columns that have it keep it. MODIFY COLUMN can change the row threshold, but not on a column without doc mode created before the upgrade whose subcolumn count is 0. CTAS creates new columns with the count of the query's type, so CTAS of a VARIANT expression such as `parse_to_variant(...)` extracts no dynamic path unless the expression is cast to `VARIANT`. Upgrading needs the whole cluster stopped, and downgrading is not supported. SHOW CREATE TABLE prints the new names, which an older FE (for example a CCR target) does not accept.

### Check List (For Author)

- Test: Unit Test, Regression test, Manual test
    - FE UT: the classes that the VARIANT properties touch plus those that apache#69995 and apache#70501 added or changed,
      446 tests, 0 failures after two test fixes (VariantEscapeTableTest needs debug points to read the
      session defaults since apache#70501; ExpressionTranslatorTest passes the serialize type).
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1082 run, 1072 passed, 9 skipped
      (5 pre-existing GTEST_SKIPs, 4 nested-group cases), 0 failed after removing the assertion of
      TabletSchemaTest.MasterPersistedEncodingGoldenBytes that ColumnPB has no field 36 (no branch uses it).
    - Regression: every variant_p0 suite (215) with the default session; only outfile_csv_variant_doc_value
      fails because it needs S3 credentials. With fuzz-like global defaults (subcolumn count 4, row
      threshold 19, typed paths to sparse, shard count 3) the pinned suites pass; with their pins removed,
      their query results equal the default run except index scores, which need an extracted path, and
      whole VARIANT values with an empty array, which a subcolumn drops and the sparse column keeps (also on
      master).
    - Manual test: an upgrade from 4.1 with 35 tables that cover every VARIANT property 4.1 accepts,
      followed by loads, compaction, DDL and a restart; every table reads back the same data, except a
      Decimal256 typed path, which master cannot read before this PR either.
- Behavior changed: Yes. See the release note.
- Does this need documentation: Yes, the VARIANT documentation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 16, 2026
…ning

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

This is the VARIANT part of apache#69812, split out so that it can be picked to branch-4.2 on its own, and it gives the VARIANT properties the meaning that the VARIANT tuning guide describes. Every user-visible change below shows 4.1 and this PR, each as one SQL script with its result in comments. The results of this PR come from running the scripts; those of 4.1 come from the code of branch-4.1. `SET describe_extend_variant_column = true` makes DESC list the subcolumns that the data files extracted. Changes that one script cannot show are listed at the end.

#### 1. `variant_max_subcolumns_count = 0` on a new column extracts no dynamic path

4.1: 0 means no limit, so every path is extracted.

```sql
CREATE TABLE t1 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_max_subcolumns_count" = "0")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": "x"}'));
SET describe_extend_variant_column = true;
DESC t1;
-- v, v.a, v.b, v.t: every path is a subcolumn
```

This PR: 0 keeps every dynamic path in the sparse column. Typed paths are still extracted.

```sql
CREATE TABLE t1 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_max_subcolumns_count" = "0")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": "x"}'));
SET describe_extend_variant_column = true;
DESC t1;
-- v, v.t: only the typed path is a subcolumn; a and b stay in the sparse column
SELECT v FROM t1;
-- {"a":1,"b":"x","t":1}
```

#### 2. `variant_doc_materialization_min_rows` is renamed to `variant_subcolumn_extraction_min_rows`

4.1:

```sql
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_doc_materialization_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t2;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "100",
--                        "variant_doc_hash_shard_count" = "64")> NULL
```

This PR: the old name is refused.

```sql
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_doc_materialization_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_max_subcolumns_count and
-- variant_max_sparse_column_statistics_size and variant_sparse_hash_shard_count and variant_enable_doc_mode
-- and variant_subcolumn_extraction_min_rows and variant_enable_nested_group
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_subcolumn_extraction_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t2;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "100","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 3. `variant_doc_hash_shard_count` is replaced by `variant_sparse_hash_shard_count`

4.1: doc mode has its own shard count property and refuses `variant_sparse_hash_shard_count`.

```sql
CREATE TABLE t3 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_doc_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t3;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "0",
--                        "variant_doc_hash_shard_count" = "8")> NULL
CREATE TABLE t3b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_sparse_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_sparse_hash_shard_count and
-- variant_enable_doc_mode cannot be set together
```

This PR: `variant_sparse_hash_shard_count` sets the shard count in both modes; the doc mode name is refused.

```sql
CREATE TABLE t3 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_doc_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_max_subcolumns_count and ... (as in 2)
CREATE TABLE t3b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_sparse_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t3b;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "8")> NULL
```

#### 4. The row threshold works without doc mode

4.1: the threshold exists only in doc mode, as `variant_doc_materialization_min_rows`. Without doc mode the property is dropped.

```sql
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_subcolumn_extraction_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_enable_typed_paths_to_sparse and
-- variant_max_subcolumns_count and variant_max_sparse_column_statistics_size and variant_sparse_hash_shard_count
-- and variant_enable_doc_mode and variant_doc_materialization_min_rows and variant_doc_hash_shard_count
-- and variant_enable_nested_group
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_doc_materialization_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- accepted, but the threshold is dropped: SHOW CREATE TABLE does not show it, and every load extracts its paths
```

This PR: `variant_subcolumn_extraction_min_rows` works with and without doc mode. A data file with fewer rows extracts no dynamic path.

```sql
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_subcolumn_extraction_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t4 VALUES (1, parse_to_variant('{"t": 1, "a": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t4;
-- v, v.t: the 2-row data file is below the threshold and extracts no dynamic path
INSERT INTO t4 VALUES (3, parse_to_variant('{"b": 3}')), (4, parse_to_variant('{"b": 4}')),
                      (5, parse_to_variant('{"b": 5}'));
DESC t4;
-- v, v.b, v.t: the 3-row data file extracts b
```

#### 5. Doc mode takes the subcolumn count and the shard count, and extracts up to the count

4.1: doc mode cannot be combined with the subcolumn count or the shard count, even with `"variant_enable_doc_mode" = "false"`, and it extracts every path once a data file reaches the threshold.

```sql
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true",
                                                        "variant_max_subcolumns_count" = "1")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_max_subcolumns_count and
-- variant_enable_doc_mode cannot be set together
CREATE TABLE t5 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "false", "variant_max_subcolumns_count" = "1")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- the same error; variant_sparse_hash_shard_count is refused the same way
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t5 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t5;
-- v, v.a, v.b, v.t: every path is a subcolumn
```

This PR: doc mode is a separate switch. It extracts the typed paths and up to the count of dynamic paths, the most frequent first; the count does not include typed paths, and 0 extracts none. The document keeps every path.

```sql
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true",
                                                        "variant_max_subcolumns_count" = "1",
                                                        "variant_sparse_hash_shard_count" = "4")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t5 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t5;
-- v, v.a, v.t: the typed path and the most frequent dynamic path (a: 2 rows, b: 1 row)
-- with "variant_max_subcolumns_count" = "0": v, v.t
SELECT v FROM t5 ORDER BY k;
-- {"a":1,"b":1,"t":1}
-- {"a":2,"t":2}
CREATE TABLE t5b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "false", "variant_max_subcolumns_count" = "10")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- accepted
```

#### 6. Doc mode: the default shard count is 1, and SHOW CREATE TABLE prints the count

4.1: the document is split into 64 columns by default.

```sql
CREATE TABLE t6 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t6;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "0",
--                        "variant_doc_hash_shard_count" = "64")> NULL
```

This PR: 1 by default, as without doc mode.

```sql
CREATE TABLE t6 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t6;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 7. `variant_enable_typed_paths_to_sparse` cannot be set to `true` and is not printed

4.1:

```sql
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t7;
-- `v` variant<'t':int,PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_enable_typed_paths_to_sparse" = "true",
--                                "variant_max_sparse_column_statistics_size" = "10000","variant_sparse_hash_shard_count" = "1")> NULL
-- typed paths compete with dynamic paths for variant_max_subcolumns_count
```

This PR: the property is internal. It still takes `false`, which released versions print for every column without doc mode, so their DDL still runs.

```sql
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_enable_typed_paths_to_sparse = true is not supported
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "false")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t7;
-- `v` variant<'t':int,PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_max_sparse_column_statistics_size" = "10000",
--                                "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 8. MODIFY COLUMN can change the row threshold without doc mode

4.1: only the threshold of a doc mode column can be changed (apache#61100); a column without doc mode has none.

```sql
CREATE TABLE t8 (k INT, v VARIANT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
ALTER TABLE t8 MODIFY COLUMN v VARIANT<PROPERTIES("variant_subcolumn_extraction_min_rows" = "100")>;
-- ERROR 1235 (42000): Not Supported: only support for variant_enable_typed_paths_to_sparse and ... (as in 4)
```

This PR:

```sql
CREATE TABLE t8 (k INT, v VARIANT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
ALTER TABLE t8 MODIFY COLUMN v VARIANT<PROPERTIES("variant_subcolumn_extraction_min_rows" = "100")>;
-- the job finishes (SHOW ALTER TABLE COLUMN)
SHOW CREATE TABLE t8;
-- `v` variant<PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_max_sparse_column_statistics_size" = "10000",
--                        "variant_subcolumn_extraction_min_rows" = "100","variant_sparse_hash_shard_count" = "1")> NULL
INSERT INTO t8 VALUES (1, parse_to_variant('{"a": 1}'));
SET describe_extend_variant_column = true;
DESC t8;
-- v: the 1-row data file is below the threshold
```

#### 9. CTAS of a VARIANT expression creates a column that extracts no dynamic path

4.1: the type of `parse_to_variant(...)` has a count of 0, which extracts every path, and the new column takes it.

```sql
CREATE TABLE t10 PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, parse_to_variant('{"a": 1, "b": 2}') AS v;
SET describe_extend_variant_column = true;
DESC t10;
-- v, v.a, v.b: every path is a subcolumn
```

This PR: the new column is V2 and keeps the count of 0, so it extracts no dynamic path. Casting the expression to `VARIANT` gives it the default count of a declared column, 2048.

```sql
CREATE TABLE t10 PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, parse_to_variant('{"a": 1, "b": 2}') AS v;
SET describe_extend_variant_column = true;
DESC t10;
-- v: variant<PROPERTIES ("variant_max_subcolumns_count" = "0",...)>, no subcolumn
CREATE TABLE t10b PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, CAST(parse_to_variant('{"a": 1, "b": 2}') AS VARIANT) AS v;
DESC t10b;
-- v: variant<PROPERTIES ("variant_max_subcolumns_count" = "2048",...)>, v.a, v.b
```

#### Changes that one SQL script cannot show

- **Upgrade.** The whole cluster has to be stopped; FE and BE of different versions must not run together. An older BE does not know the serialize type below and extracts every path of a new column with a count of 0, and the new reader fails an assertion on such a data file. Downgrading is not supported: an older version reads that count as "extract every path", and its compaction drops the paths that only the sparse column holds.
- **Columns created before the upgrade keep their behavior.** A hidden serialize type tells new columns (V2) from older ones (V1); DESC and SHOW CREATE TABLE do not print it. It is sent as `TColumn.variant_serialize_type` (31) and stored as `ColumnPB.variant_serialize_type` (36), both optional with default V1, so every existing column is V1. On a V1 column a count of 0 still extracts every path, a doc mode column created by 4.1 still extracts every path once a data file reaches its threshold, and a column with `variant_enable_typed_paths_to_sparse = true` keeps it (MODIFY COLUMN keeps it too). Every column created after the upgrade is V2, also with ADD COLUMN, CTAS and CREATE TABLE LIKE, and also when the SHOW CREATE TABLE output of an old table is run again: a stated count of 0 then extracts no dynamic path, and `variant_enable_typed_paths_to_sparse = true` is not carried over. CTAS copies the count of the query's type, so a count of 0 of an old column extracts no dynamic path in the new column (9 shows the same for a VARIANT expression).
- **MODIFY COLUMN on an older column without doc mode whose count is 0** refuses a non-zero threshold ("Can not set variant_subcolumn_extraction_min_rows on a column of an older version whose variant_max_subcolumns_count is 0"): there 0 extracts every path, and that compaction branch only builds the subcolumns its inputs extracted, so the paths that a data file below the threshold left in the sparse column would be lost. An older column with a count above 0 takes a threshold like a new one.
- **Compaction.** It applies the threshold to the rows of its inputs. Below it, the subcolumns that inputs extracted go back to the sparse column instead of being dropped; this happens after an ALTER raises the threshold, or when an earlier compaction extracted them from more rows than it kept. At or above it, it extracts the most frequent paths of all inputs. In doc mode every doc bucket of the output has its own writer, so compaction chooses the dynamic paths once from the doc statistics of all inputs; in 4.1 every bucket extracted every path.
- **DDL between versions.** The SHOW CREATE TABLE output of a 4.1 doc mode table has to be renamed (2, 3) before it runs. The output of this version uses `variant_subcolumn_extraction_min_rows`, which an older FE, for example the target of CCR, does not accept.
- **SEARCH on recent data.** SEARCH needs an index on the subcolumn of the path. A data file below the threshold has no subcolumns, so SEARCH on it fails with "SEARCH index unavailable for Variant field" until compaction extracts the path, as it already does for a path beyond the count. MATCH falls back to reading the values unless `enable_match_without_inverted_index = false`.
- **Fuzzy tests.** Fuzzy mode now also randomizes the row threshold without doc mode and `default_variant_enable_typed_paths_to_sparse` (never with doc mode, which refuses it). The suites that print the subcolumn layout, or need an index on every path, pin them.
- **Test-only session variables.** Since apache#70501 the `default_variant_*` session variables can only be set with `enable_debug_points = true`, as in the tests. This PR renames `default_variant_doc_materialization_min_rows` to `default_variant_subcolumn_extraction_min_rows` and removes `default_variant_doc_hash_shard_count`; `default_variant_sparse_hash_shard_count` sets the shard count of both modes.

### Release note

On a new VARIANT column, `variant_max_subcolumns_count = 0` keeps every dynamic path in the sparse encoding instead of extracting it; existing columns keep the old meaning. `variant_subcolumn_extraction_min_rows` replaces `variant_doc_materialization_min_rows`, and `variant_sparse_hash_shard_count` now also sets the shard count of doc mode in place of `variant_doc_hash_shard_count`. The two doc mode names are not accepted any more, so the SHOW CREATE TABLE output of a 4.1 doc mode table has to be renamed before it runs. Doc mode can be combined with the subcolumn count and the row threshold, and the row threshold also works without doc mode. New doc mode columns extract up to the subcolumn count instead of every path, doc mode columns created by 4.1 keep extracting every path, and the default shard count of doc mode is 1 instead of 64. `variant_enable_typed_paths_to_sparse` cannot be set to `true` any more and is no longer printed; columns that have it keep it. MODIFY COLUMN can change the row threshold, but not on a column without doc mode created before the upgrade whose subcolumn count is 0. CTAS creates new columns with the count of the query's type, so CTAS of a VARIANT expression such as `parse_to_variant(...)` extracts no dynamic path unless the expression is cast to `VARIANT`. Upgrading needs the whole cluster stopped, and downgrading is not supported. SHOW CREATE TABLE prints the new names, which an older FE (for example a CCR target) does not accept.

### Check List (For Author)

- Test: Unit Test, Regression test, Manual test
    - FE UT: the classes that the VARIANT properties touch plus those that apache#69995 and apache#70501 added or changed,
      446 tests, 0 failures after two test fixes (VariantEscapeTableTest needs debug points to read the
      session defaults since apache#70501; ExpressionTranslatorTest passes the serialize type).
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1082 run, 1072 passed, 9 skipped
      (5 pre-existing GTEST_SKIPs, 4 nested-group cases), 0 failed after removing the assertion of
      TabletSchemaTest.MasterPersistedEncodingGoldenBytes that ColumnPB has no field 36 (no branch uses it).
    - Regression: every variant_p0 suite (215) with the default session; only outfile_csv_variant_doc_value
      fails because it needs S3 credentials. With fuzz-like global defaults (subcolumn count 4, row
      threshold 19, typed paths to sparse, shard count 3) the pinned suites pass; with their pins removed,
      their query results equal the default run except index scores, which need an extracted path, and
      whole VARIANT values with an empty array, which a subcolumn drops and the sparse column keeps (also on
      master).
    - Manual test: an upgrade from 4.1 with 35 tables that cover every VARIANT property 4.1 accepts,
      followed by loads, compaction, DDL and a restart; every table reads back the same data, except a
      Decimal256 typed path, which master cannot read before this PR either.
- Behavior changed: Yes. See the release note.
- Does this need documentation: Yes, the VARIANT documentation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 17, 2026
…ning

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

This is the VARIANT part of apache#69812, split out so that it can be picked to branch-4.2 on its own, and it gives the VARIANT properties the meaning that the VARIANT tuning guide describes. Every user-visible change below shows 4.1 and this PR, each as one SQL script with its result in comments. The results of this PR come from running the scripts; those of 4.1 come from the code of branch-4.1. `SET describe_extend_variant_column = true` makes DESC list the subcolumns that the data files extracted. Changes that one script cannot show are listed at the end.

#### 1. `variant_max_subcolumns_count = 0` on a new column extracts no dynamic path

4.1: 0 means no limit, so every path is extracted.

```sql
CREATE TABLE t1 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_max_subcolumns_count" = "0")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": "x"}'));
SET describe_extend_variant_column = true;
DESC t1;
-- v, v.a, v.b, v.t: every path is a subcolumn
```

This PR: 0 keeps every dynamic path in the sparse column. Typed paths are still extracted.

```sql
CREATE TABLE t1 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_max_subcolumns_count" = "0")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": "x"}'));
SET describe_extend_variant_column = true;
DESC t1;
-- v, v.t: only the typed path is a subcolumn; a and b stay in the sparse column
SELECT v FROM t1;
-- {"a":1,"b":"x","t":1}
```

#### 2. `variant_doc_materialization_min_rows` is renamed to `variant_subcolumn_extraction_min_rows`

4.1:

```sql
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_doc_materialization_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t2;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "100",
--                        "variant_doc_hash_shard_count" = "64")> NULL
```

This PR: the old name is refused.

```sql
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_doc_materialization_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_max_subcolumns_count and
-- variant_max_sparse_column_statistics_size and variant_sparse_hash_shard_count and variant_enable_doc_mode
-- and variant_subcolumn_extraction_min_rows and variant_enable_nested_group
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_subcolumn_extraction_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t2;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "100","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 3. `variant_doc_hash_shard_count` is replaced by `variant_sparse_hash_shard_count`

4.1: doc mode has its own shard count property and refuses `variant_sparse_hash_shard_count`.

```sql
CREATE TABLE t3 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_doc_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t3;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "0",
--                        "variant_doc_hash_shard_count" = "8")> NULL
CREATE TABLE t3b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_sparse_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_sparse_hash_shard_count and
-- variant_enable_doc_mode cannot be set together
```

This PR: `variant_sparse_hash_shard_count` sets the shard count in both modes; the doc mode name is refused.

```sql
CREATE TABLE t3 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_doc_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_max_subcolumns_count and ... (as in 2)
CREATE TABLE t3b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_sparse_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t3b;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "8")> NULL
```

#### 4. The row threshold works without doc mode

4.1: the threshold exists only in doc mode, as `variant_doc_materialization_min_rows`. Without doc mode the property is dropped.

```sql
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_subcolumn_extraction_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_enable_typed_paths_to_sparse and
-- variant_max_subcolumns_count and variant_max_sparse_column_statistics_size and variant_sparse_hash_shard_count
-- and variant_enable_doc_mode and variant_doc_materialization_min_rows and variant_doc_hash_shard_count
-- and variant_enable_nested_group
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_doc_materialization_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- accepted, but the threshold is dropped: SHOW CREATE TABLE does not show it, and every load extracts its paths
```

This PR: `variant_subcolumn_extraction_min_rows` works with and without doc mode. A data file with fewer rows extracts no dynamic path.

```sql
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_subcolumn_extraction_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t4 VALUES (1, parse_to_variant('{"t": 1, "a": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t4;
-- v, v.t: the 2-row data file is below the threshold and extracts no dynamic path
INSERT INTO t4 VALUES (3, parse_to_variant('{"b": 3}')), (4, parse_to_variant('{"b": 4}')),
                      (5, parse_to_variant('{"b": 5}'));
DESC t4;
-- v, v.b, v.t: the 3-row data file extracts b
```

#### 5. Doc mode takes the subcolumn count and the shard count, and extracts up to the count

4.1: doc mode cannot be combined with the subcolumn count or the shard count, even with `"variant_enable_doc_mode" = "false"`, and it extracts every path once a data file reaches the threshold.

```sql
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true",
                                                        "variant_max_subcolumns_count" = "1")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_max_subcolumns_count and
-- variant_enable_doc_mode cannot be set together
CREATE TABLE t5 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "false", "variant_max_subcolumns_count" = "1")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- the same error; variant_sparse_hash_shard_count is refused the same way
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t5 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t5;
-- v, v.a, v.b, v.t: every path is a subcolumn
```

This PR: doc mode is a separate switch. It extracts the typed paths and up to the count of dynamic paths, the most frequent first; the count does not include typed paths, and 0 extracts none. The document keeps every path.

```sql
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true",
                                                        "variant_max_subcolumns_count" = "1",
                                                        "variant_sparse_hash_shard_count" = "4")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t5 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t5;
-- v, v.a, v.t: the typed path and the most frequent dynamic path (a: 2 rows, b: 1 row)
-- with "variant_max_subcolumns_count" = "0": v, v.t
SELECT v FROM t5 ORDER BY k;
-- {"a":1,"b":1,"t":1}
-- {"a":2,"t":2}
CREATE TABLE t5b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "false", "variant_max_subcolumns_count" = "10")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- accepted
```

#### 6. Doc mode: the default shard count is 1, and SHOW CREATE TABLE prints the count

4.1: the document is split into 64 columns by default.

```sql
CREATE TABLE t6 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t6;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "0",
--                        "variant_doc_hash_shard_count" = "64")> NULL
```

This PR: 1 by default, as without doc mode.

```sql
CREATE TABLE t6 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t6;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 7. `variant_enable_typed_paths_to_sparse` cannot be set to `true` and is not printed

4.1:

```sql
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t7;
-- `v` variant<'t':int,PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_enable_typed_paths_to_sparse" = "true",
--                                "variant_max_sparse_column_statistics_size" = "10000","variant_sparse_hash_shard_count" = "1")> NULL
-- typed paths compete with dynamic paths for variant_max_subcolumns_count
```

This PR: the property is internal. It still takes `false`, which released versions print for every column without doc mode, so their DDL still runs.

```sql
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_enable_typed_paths_to_sparse = true is not supported
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "false")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t7;
-- `v` variant<'t':int,PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_max_sparse_column_statistics_size" = "10000",
--                                "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 8. MODIFY COLUMN can change the row threshold without doc mode

4.1: only the threshold of a doc mode column can be changed (apache#61100); a column without doc mode has none.

```sql
CREATE TABLE t8 (k INT, v VARIANT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
ALTER TABLE t8 MODIFY COLUMN v VARIANT<PROPERTIES("variant_subcolumn_extraction_min_rows" = "100")>;
-- ERROR 1235 (42000): Not Supported: only support for variant_enable_typed_paths_to_sparse and ... (as in 4)
```

This PR:

```sql
CREATE TABLE t8 (k INT, v VARIANT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
ALTER TABLE t8 MODIFY COLUMN v VARIANT<PROPERTIES("variant_subcolumn_extraction_min_rows" = "100")>;
-- the job finishes (SHOW ALTER TABLE COLUMN)
SHOW CREATE TABLE t8;
-- `v` variant<PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_max_sparse_column_statistics_size" = "10000",
--                        "variant_subcolumn_extraction_min_rows" = "100","variant_sparse_hash_shard_count" = "1")> NULL
INSERT INTO t8 VALUES (1, parse_to_variant('{"a": 1}'));
SET describe_extend_variant_column = true;
DESC t8;
-- v: the 1-row data file is below the threshold
```

#### 9. CTAS of a VARIANT expression creates a column that extracts no dynamic path

4.1: the type of `parse_to_variant(...)` has a count of 0, which extracts every path, and the new column takes it.

```sql
CREATE TABLE t10 PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, parse_to_variant('{"a": 1, "b": 2}') AS v;
SET describe_extend_variant_column = true;
DESC t10;
-- v, v.a, v.b: every path is a subcolumn
```

This PR: the new column is V2 and keeps the count of 0, so it extracts no dynamic path. Casting the expression to `VARIANT` gives it the default count of a declared column, 2048.

```sql
CREATE TABLE t10 PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, parse_to_variant('{"a": 1, "b": 2}') AS v;
SET describe_extend_variant_column = true;
DESC t10;
-- v: variant<PROPERTIES ("variant_max_subcolumns_count" = "0",...)>, no subcolumn
CREATE TABLE t10b PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, CAST(parse_to_variant('{"a": 1, "b": 2}') AS VARIANT) AS v;
DESC t10b;
-- v: variant<PROPERTIES ("variant_max_subcolumns_count" = "2048",...)>, v.a, v.b
```

#### Changes that one SQL script cannot show

- **Upgrade.** The whole cluster has to be stopped; FE and BE of different versions must not run together. An older BE does not know the serialize type below and extracts every path of a new column with a count of 0, and the new reader fails an assertion on such a data file. Downgrading is not supported: an older version reads that count as "extract every path", and its compaction drops the paths that only the sparse column holds.
- **Columns created before the upgrade keep their behavior.** A hidden serialize type tells new columns (V2) from older ones (V1); DESC and SHOW CREATE TABLE do not print it. It is sent as `TColumn.variant_serialize_type` (31) and stored as `ColumnPB.variant_serialize_type` (36), both optional with default V1, so every existing column is V1. On a V1 column a count of 0 still extracts every path, a doc mode column created by 4.1 still extracts every path once a data file reaches its threshold, and a column with `variant_enable_typed_paths_to_sparse = true` keeps it (MODIFY COLUMN keeps it too). Every column created after the upgrade is V2, also with ADD COLUMN, CTAS and CREATE TABLE LIKE, and also when the SHOW CREATE TABLE output of an old table is run again: a stated count of 0 then extracts no dynamic path, and `variant_enable_typed_paths_to_sparse = true` is not carried over. CTAS copies the count of the query's type, so a count of 0 of an old column extracts no dynamic path in the new column (9 shows the same for a VARIANT expression).
- **MODIFY COLUMN on an older column without doc mode whose count is 0** refuses a non-zero threshold ("Can not set variant_subcolumn_extraction_min_rows on a column of an older version whose variant_max_subcolumns_count is 0"): there 0 extracts every path, and that compaction branch only builds the subcolumns its inputs extracted, so the paths that a data file below the threshold left in the sparse column would be lost. An older column with a count above 0 takes a threshold like a new one.
- **Compaction.** It applies the threshold to the rows of its inputs. Below it, the subcolumns that inputs extracted go back to the sparse column instead of being dropped; this happens after an ALTER raises the threshold, or when an earlier compaction extracted them from more rows than it kept. At or above it, it extracts the most frequent paths of all inputs. In doc mode every doc bucket of the output has its own writer, so compaction chooses the dynamic paths once from the doc statistics of all inputs; in 4.1 every bucket extracted every path.
- **DDL between versions.** The SHOW CREATE TABLE output of a 4.1 doc mode table has to be renamed (2, 3) before it runs. The output of this version uses `variant_subcolumn_extraction_min_rows`, which an older FE, for example the target of CCR, does not accept.
- **SEARCH on recent data.** SEARCH needs an index on the subcolumn of the path. A data file below the threshold has no subcolumns, so SEARCH on it fails with "SEARCH index unavailable for Variant field" until compaction extracts the path, as it already does for a path beyond the count. MATCH falls back to reading the values unless `enable_match_without_inverted_index = false`.
- **Fuzzy tests.** Fuzzy mode now also randomizes the row threshold without doc mode and `default_variant_enable_typed_paths_to_sparse` (never with doc mode, which refuses it). The suites that print the subcolumn layout, or need an index on every path, pin them.
- **Test-only session variables.** Since apache#70501 the `default_variant_*` session variables can only be set with `enable_debug_points = true`, as in the tests. This PR renames `default_variant_doc_materialization_min_rows` to `default_variant_subcolumn_extraction_min_rows` and removes `default_variant_doc_hash_shard_count`; `default_variant_sparse_hash_shard_count` sets the shard count of both modes.

### Release note

On a new VARIANT column, `variant_max_subcolumns_count = 0` keeps every dynamic path in the sparse encoding instead of extracting it; existing columns keep the old meaning. `variant_subcolumn_extraction_min_rows` replaces `variant_doc_materialization_min_rows`, and `variant_sparse_hash_shard_count` now also sets the shard count of doc mode in place of `variant_doc_hash_shard_count`. The two doc mode names are not accepted any more, so the SHOW CREATE TABLE output of a 4.1 doc mode table has to be renamed before it runs. Doc mode can be combined with the subcolumn count and the row threshold, and the row threshold also works without doc mode. New doc mode columns extract up to the subcolumn count instead of every path, doc mode columns created by 4.1 keep extracting every path, and the default shard count of doc mode is 1 instead of 64. `variant_enable_typed_paths_to_sparse` cannot be set to `true` any more and is no longer printed; columns that have it keep it. MODIFY COLUMN can change the row threshold, but not on a column without doc mode created before the upgrade whose subcolumn count is 0. CTAS creates new columns with the count of the query's type, so CTAS of a VARIANT expression such as `parse_to_variant(...)` extracts no dynamic path unless the expression is cast to `VARIANT`. Upgrading needs the whole cluster stopped, and downgrading is not supported. SHOW CREATE TABLE prints the new names, which an older FE (for example a CCR target) does not accept.

### Check List (For Author)

- Test: Unit Test, Regression test, Manual test
    - FE UT: the classes that the VARIANT properties touch plus those that apache#69995 and apache#70501 added or changed,
      446 tests, 0 failures after two test fixes (VariantEscapeTableTest needs debug points to read the
      session defaults since apache#70501; ExpressionTranslatorTest passes the serialize type).
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1082 run, 1072 passed, 9 skipped
      (5 pre-existing GTEST_SKIPs, 4 nested-group cases), 0 failed after removing the assertion of
      TabletSchemaTest.MasterPersistedEncodingGoldenBytes that ColumnPB has no field 36 (no branch uses it).
    - Regression: every variant_p0 suite (215) with the default session; only outfile_csv_variant_doc_value
      fails because it needs S3 credentials. With fuzz-like global defaults (subcolumn count 4, row
      threshold 19, typed paths to sparse, shard count 3) the pinned suites pass; with their pins removed,
      their query results equal the default run except index scores, which need an extracted path, and
      whole VARIANT values with an empty array, which a subcolumn drops and the sparse column keeps (also on
      master). With a global subcolumn count of 0, with and without doc mode, every variant_p0 suite passes
      except the S3 one. variant_p1 and the p2 suites that run locally pass with a count of 0 without doc
      mode, except the ones that set enable_variant_flatten_nested, which master's V2 writer rejects.
    - Manual test: an upgrade from 4.1 with 35 tables that cover every VARIANT property 4.1 accepts,
      followed by loads, compaction, DDL and a restart; every table reads back the same data, except a
      Decimal256 typed path, which master cannot read before this PR either.
- Behavior changed: Yes. See the release note.
- Does this need documentation: Yes, the VARIANT documentation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
csun5285 added a commit to csun5285/doris that referenced this pull request Nov 17, 2026
…ning

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69812

Problem Summary:

This is the VARIANT part of apache#69812, split out so that it can be picked to branch-4.2 on its own, and it gives the VARIANT properties the meaning that the VARIANT tuning guide describes. Every user-visible change below shows 4.1 and this PR, each as one SQL script with its result in comments. The results of this PR come from running the scripts; those of 4.1 come from the code of branch-4.1. `SET describe_extend_variant_column = true` makes DESC list the subcolumns that the data files extracted. Changes that one script cannot show are listed at the end.

#### 1. `variant_max_subcolumns_count = 0` on a new column extracts no dynamic path

4.1: 0 means no limit, so every path is extracted.

```sql
CREATE TABLE t1 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_max_subcolumns_count" = "0")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": "x"}'));
SET describe_extend_variant_column = true;
DESC t1;
-- v, v.a, v.b, v.t: every path is a subcolumn
```

This PR: 0 keeps every dynamic path in the sparse column. Typed paths are still extracted.

```sql
CREATE TABLE t1 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_max_subcolumns_count" = "0")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t1 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": "x"}'));
SET describe_extend_variant_column = true;
DESC t1;
-- v, v.t: only the typed path is a subcolumn; a and b stay in the sparse column
SELECT v FROM t1;
-- {"a":1,"b":"x","t":1}
```

#### 2. `variant_doc_materialization_min_rows` is renamed to `variant_subcolumn_extraction_min_rows`

4.1:

```sql
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_doc_materialization_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t2;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "100",
--                        "variant_doc_hash_shard_count" = "64")> NULL
```

This PR: the old name is refused.

```sql
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_doc_materialization_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_max_subcolumns_count and
-- variant_max_sparse_column_statistics_size and variant_sparse_hash_shard_count and variant_enable_doc_mode
-- and variant_subcolumn_extraction_min_rows and variant_enable_nested_group
CREATE TABLE t2 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true",
                                             "variant_subcolumn_extraction_min_rows" = "100")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t2;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "100","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 3. `variant_doc_hash_shard_count` is replaced by `variant_sparse_hash_shard_count`

4.1: doc mode has its own shard count property and refuses `variant_sparse_hash_shard_count`.

```sql
CREATE TABLE t3 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_doc_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t3;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "0",
--                        "variant_doc_hash_shard_count" = "8")> NULL
CREATE TABLE t3b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_sparse_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_sparse_hash_shard_count and
-- variant_enable_doc_mode cannot be set together
```

This PR: `variant_sparse_hash_shard_count` sets the shard count in both modes; the doc mode name is refused.

```sql
CREATE TABLE t3 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_doc_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_max_subcolumns_count and ... (as in 2)
CREATE TABLE t3b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true", "variant_sparse_hash_shard_count" = "8")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t3b;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "8")> NULL
```

#### 4. The row threshold works without doc mode

4.1: the threshold exists only in doc mode, as `variant_doc_materialization_min_rows`. Without doc mode the property is dropped.

```sql
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_subcolumn_extraction_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: only support for variant_enable_typed_paths_to_sparse and
-- variant_max_subcolumns_count and variant_max_sparse_column_statistics_size and variant_sparse_hash_shard_count
-- and variant_enable_doc_mode and variant_doc_materialization_min_rows and variant_doc_hash_shard_count
-- and variant_enable_nested_group
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_doc_materialization_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- accepted, but the threshold is dropped: SHOW CREATE TABLE does not show it, and every load extracts its paths
```

This PR: `variant_subcolumn_extraction_min_rows` works with and without doc mode. A data file with fewer rows extracts no dynamic path.

```sql
CREATE TABLE t4 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_subcolumn_extraction_min_rows" = "3")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t4 VALUES (1, parse_to_variant('{"t": 1, "a": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t4;
-- v, v.t: the 2-row data file is below the threshold and extracts no dynamic path
INSERT INTO t4 VALUES (3, parse_to_variant('{"b": 3}')), (4, parse_to_variant('{"b": 4}')),
                      (5, parse_to_variant('{"b": 5}'));
DESC t4;
-- v, v.b, v.t: the 3-row data file extracts b
```

#### 5. Doc mode takes the subcolumn count and the shard count, and extracts up to the count

4.1: doc mode cannot be combined with the subcolumn count or the shard count, even with `"variant_enable_doc_mode" = "false"`, and it extracts every path once a data file reaches the threshold.

```sql
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true",
                                                        "variant_max_subcolumns_count" = "1")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_max_subcolumns_count and
-- variant_enable_doc_mode cannot be set together
CREATE TABLE t5 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "false", "variant_max_subcolumns_count" = "1")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- the same error; variant_sparse_hash_shard_count is refused the same way
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t5 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t5;
-- v, v.a, v.b, v.t: every path is a subcolumn
```

This PR: doc mode is a separate switch. It extracts the typed paths and up to the count of dynamic paths, the most frequent first; the count does not include typed paths, and 0 extracts none. The document keeps every path.

```sql
CREATE TABLE t5 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_doc_mode" = "true",
                                                        "variant_max_subcolumns_count" = "1",
                                                        "variant_sparse_hash_shard_count" = "4")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
INSERT INTO t5 VALUES (1, parse_to_variant('{"t": 1, "a": 1, "b": 1}')), (2, parse_to_variant('{"t": 2, "a": 2}'));
SET describe_extend_variant_column = true;
DESC t5;
-- v, v.a, v.t: the typed path and the most frequent dynamic path (a: 2 rows, b: 1 row)
-- with "variant_max_subcolumns_count" = "0": v, v.t
SELECT v FROM t5 ORDER BY k;
-- {"a":1,"b":1,"t":1}
-- {"a":2,"t":2}
CREATE TABLE t5b (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "false", "variant_max_subcolumns_count" = "10")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- accepted
```

#### 6. Doc mode: the default shard count is 1, and SHOW CREATE TABLE prints the count

4.1: the document is split into 64 columns by default.

```sql
CREATE TABLE t6 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t6;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_doc_materialization_min_rows" = "0",
--                        "variant_doc_hash_shard_count" = "64")> NULL
```

This PR: 1 by default, as without doc mode.

```sql
CREATE TABLE t6 (k INT, v VARIANT<PROPERTIES("variant_enable_doc_mode" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t6;
-- `v` variant<PROPERTIES ("variant_enable_doc_mode" = "true","variant_max_subcolumns_count" = "2048",
--                        "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 7. `variant_enable_typed_paths_to_sparse` cannot be set to `true` and is not printed

4.1:

```sql
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t7;
-- `v` variant<'t':int,PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_enable_typed_paths_to_sparse" = "true",
--                                "variant_max_sparse_column_statistics_size" = "10000","variant_sparse_hash_shard_count" = "1")> NULL
-- typed paths compete with dynamic paths for variant_max_subcolumns_count
```

This PR: the property is internal. It still takes `false`, which released versions print for every column without doc mode, so their DDL still runs.

```sql
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "true")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
-- ERROR 1235 (42000): Not Supported: errCode = 2, detailMessage = variant_enable_typed_paths_to_sparse = true is not supported
CREATE TABLE t7 (k INT, v VARIANT<'t' : INT, PROPERTIES("variant_enable_typed_paths_to_sparse" = "false")>)
DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
SHOW CREATE TABLE t7;
-- `v` variant<'t':int,PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_max_sparse_column_statistics_size" = "10000",
--                                "variant_subcolumn_extraction_min_rows" = "0","variant_sparse_hash_shard_count" = "1")> NULL
```

#### 8. MODIFY COLUMN can change the row threshold without doc mode

4.1: only the threshold of a doc mode column can be changed (apache#61100); a column without doc mode has none.

```sql
CREATE TABLE t8 (k INT, v VARIANT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
ALTER TABLE t8 MODIFY COLUMN v VARIANT<PROPERTIES("variant_subcolumn_extraction_min_rows" = "100")>;
-- ERROR 1235 (42000): Not Supported: only support for variant_enable_typed_paths_to_sparse and ... (as in 4)
```

This PR:

```sql
CREATE TABLE t8 (k INT, v VARIANT) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");
ALTER TABLE t8 MODIFY COLUMN v VARIANT<PROPERTIES("variant_subcolumn_extraction_min_rows" = "100")>;
-- the job finishes (SHOW ALTER TABLE COLUMN)
SHOW CREATE TABLE t8;
-- `v` variant<PROPERTIES ("variant_max_subcolumns_count" = "2048","variant_max_sparse_column_statistics_size" = "10000",
--                        "variant_subcolumn_extraction_min_rows" = "100","variant_sparse_hash_shard_count" = "1")> NULL
INSERT INTO t8 VALUES (1, parse_to_variant('{"a": 1}'));
SET describe_extend_variant_column = true;
DESC t8;
-- v: the 1-row data file is below the threshold
```

#### 9. CTAS of a VARIANT expression creates a column that extracts no dynamic path

4.1: the type of `parse_to_variant(...)` has a count of 0, which extracts every path, and the new column takes it.

```sql
CREATE TABLE t10 PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, parse_to_variant('{"a": 1, "b": 2}') AS v;
SET describe_extend_variant_column = true;
DESC t10;
-- v, v.a, v.b: every path is a subcolumn
```

This PR: the new column is V2 and keeps the count of 0, so it extracts no dynamic path. Casting the expression to `VARIANT` gives it the default count of a declared column, 2048.

```sql
CREATE TABLE t10 PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, parse_to_variant('{"a": 1, "b": 2}') AS v;
SET describe_extend_variant_column = true;
DESC t10;
-- v: variant<PROPERTIES ("variant_max_subcolumns_count" = "0",...)>, no subcolumn
CREATE TABLE t10b PROPERTIES ("replication_num" = "1")
AS SELECT 1 AS k, CAST(parse_to_variant('{"a": 1, "b": 2}') AS VARIANT) AS v;
DESC t10b;
-- v: variant<PROPERTIES ("variant_max_subcolumns_count" = "2048",...)>, v.a, v.b
```

#### Changes that one SQL script cannot show

- **Upgrade.** The whole cluster has to be stopped; FE and BE of different versions must not run together. An older BE does not know the serialize type below: it stores a new column with a count of 0 as one of an older version and extracts every path. When such a data file is later compacted together with data that the upgraded BE wrote, the compaction fails with an error and keeps its input rowsets instead of dropping those paths. Downgrading is not supported: an older version reads that count as "extract every path", and its compaction drops the paths that only the sparse column holds.
- **Columns created before the upgrade keep their behavior.** A hidden serialize type tells new columns (V2) from older ones (V1); DESC and SHOW CREATE TABLE do not print it. It is sent as `TColumn.variant_serialize_type` (31) and stored as `ColumnPB.variant_serialize_type` (36), both optional with default V1, so every existing column is V1. On a V1 column a count of 0 still extracts every path, a doc mode column created by 4.1 still extracts every path once a data file reaches its threshold, and a column with `variant_enable_typed_paths_to_sparse = true` keeps it (MODIFY COLUMN keeps it too). Every column created after the upgrade is V2, also with ADD COLUMN, CTAS and CREATE TABLE LIKE, and also when the SHOW CREATE TABLE output of an old table is run again: a stated count of 0 then extracts no dynamic path, and `variant_enable_typed_paths_to_sparse = true` is not carried over. CTAS copies the count of the query's type, so a count of 0 of an old column extracts no dynamic path in the new column (9 shows the same for a VARIANT expression).
- **MODIFY COLUMN on an older column without doc mode whose count is 0** refuses a non-zero threshold ("Can not set variant_subcolumn_extraction_min_rows on a column of an older version whose variant_max_subcolumns_count is 0"): there 0 extracts every path, and that compaction branch only builds the subcolumns its inputs extracted, so the paths that a data file below the threshold left in the sparse column would be lost. An older column with a count above 0 takes a threshold like a new one.
- **Compaction.** It applies the threshold to the rows of its inputs. Below it, the subcolumns that inputs extracted go back to the sparse column instead of being dropped; this happens after an ALTER raises the threshold, or when an earlier compaction extracted them from more rows than it kept. At or above it, it extracts the most frequent paths of all inputs. In doc mode every doc bucket of the output has its own writer, so compaction chooses the dynamic paths once from the doc statistics of all inputs; in 4.1 every bucket extracted every path.
- **DDL between versions.** The SHOW CREATE TABLE output of a 4.1 doc mode table has to be renamed (2, 3) before it runs. The output of this version uses `variant_subcolumn_extraction_min_rows`, which an older FE, for example the target of CCR, does not accept.
- **SEARCH on recent data.** SEARCH needs an index on the subcolumn of the path. A data file below the threshold has no subcolumns, so SEARCH on it fails with "SEARCH index unavailable for Variant field" until compaction extracts the path, as it already does for a path beyond the count. MATCH falls back to reading the values unless `enable_match_without_inverted_index = false`.
- **Fuzzy tests.** Fuzzy mode now also randomizes the row threshold without doc mode and `default_variant_enable_typed_paths_to_sparse` (never with doc mode, which refuses it). The suites that print the subcolumn layout, or need an index on every path, pin them.
- **Test-only session variables.** Since apache#70501 the `default_variant_*` session variables can only be set with `enable_debug_points = true`, as in the tests. This PR renames `default_variant_doc_materialization_min_rows` to `default_variant_subcolumn_extraction_min_rows` and removes `default_variant_doc_hash_shard_count`; `default_variant_sparse_hash_shard_count` sets the shard count of both modes.

### Release note

On a new VARIANT column, `variant_max_subcolumns_count = 0` keeps every dynamic path in the sparse encoding instead of extracting it; existing columns keep the old meaning. `variant_subcolumn_extraction_min_rows` replaces `variant_doc_materialization_min_rows`, and `variant_sparse_hash_shard_count` now also sets the shard count of doc mode in place of `variant_doc_hash_shard_count`. The two doc mode names are not accepted any more, so the SHOW CREATE TABLE output of a 4.1 doc mode table has to be renamed before it runs. Doc mode can be combined with the subcolumn count and the row threshold, and the row threshold also works without doc mode. New doc mode columns extract up to the subcolumn count instead of every path, doc mode columns created by 4.1 keep extracting every path, and the default shard count of doc mode is 1 instead of 64. `variant_enable_typed_paths_to_sparse` cannot be set to `true` any more and is no longer printed; columns that have it keep it. MODIFY COLUMN can change the row threshold, but not on a column without doc mode created before the upgrade whose subcolumn count is 0. CTAS creates new columns with the count of the query's type, so CTAS of a VARIANT expression such as `parse_to_variant(...)` extracts no dynamic path unless the expression is cast to `VARIANT`. Upgrading needs the whole cluster stopped, and downgrading is not supported. SHOW CREATE TABLE prints the new names, which an older FE (for example a CCR target) does not accept.

### Check List (For Author)

- Test: Unit Test, Regression test, Manual test
    - FE UT: the classes that the VARIANT properties touch plus those that apache#69995 and apache#70501 added or changed,
      446 tests, 0 failures after two test fixes (VariantEscapeTableTest needs debug points to read the
      session defaults since apache#70501; ExpressionTranslatorTest passes the serialize type).
    - BE UT: `*Variant*:*SchemaUtil*:TabletSchemaTest.*:IndexStorage*`, 1083 run, 1074 passed, 9 skipped
      (5 pre-existing GTEST_SKIPs, 4 nested-group cases), 0 failed, with the assertion of
      TabletSchemaTest.MasterPersistedEncodingGoldenBytes that ColumnPB has no field 36 removed (no branch uses
      it). VariantV2CountZeroTargetRefusesExtractedInputs compacts a V1 rowset with a count of 0 into the V2
      schema and expects the compaction to fail.
    - Regression: every variant_p0 suite (215) with the default session; only outfile_csv_variant_doc_value
      fails because it needs S3 credentials. With fuzz-like global defaults (subcolumn count 4, row
      threshold 19, typed paths to sparse, shard count 3) the pinned suites pass; with their pins removed,
      their query results equal the default run except index scores, which need an extracted path, and
      whole VARIANT values with an empty array, which a subcolumn drops and the sparse column keeps (also on
      master). With a global subcolumn count of 0, with and without doc mode, every variant_p0 suite passes
      except the S3 one. variant_p1 and the p2 suites that run locally pass with a count of 0 without doc
      mode, except the ones that set enable_variant_flatten_nested, which master's V2 writer rejects.
    - Manual test: an upgrade from 4.1 with 35 tables that cover every VARIANT property 4.1 accepts,
      followed by loads, compaction, DDL and a restart; every table reads back the same data, except a
      Decimal256 typed path, which master cannot read before this PR either.
- Behavior changed: Yes. See the release note.
- Does this need documentation: Yes, the VARIANT documentation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants