Skip to content

branch-4.2: [feature](variant) Add variant_escape_dots_in_keys to keep dotted keys distinct - #70643

Open
eldenmoon wants to merge 7 commits into
apache:branch-4.2from
eldenmoon:branch-pick-69995-4.2-20261116
Open

eldenmoon wants to merge 7 commits into
apache:branch-4.2from
eldenmoon:branch-pick-69995-4.2-20261116

Conversation

@eldenmoon

@eldenmoon eldenmoon commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

cherry-pick #69995

Preserve literal dotted and percent-containing Variant keys on branch-4.2. The format stays behind the default-off enable_variant_escape_dots_in_keys activation gate, and escaped tables execute with the V2 kernel.

Compatibility and activation

  • Retain actual-execution-type checks before scan schema promotion, including cached point queries and both RowId fetch protocols. Ordinary-column reads remain allowed.
  • Keep escaped storage behind the activation gate for creation, inheritance, load planning and RESTORE. This is operator attestation that all participating FE/BE and Cloud meta-services support the format, not automatic node capability detection. Old-node rejoin/downgrade with escaped data remains unsupported; see docs/variant-escaped-key-compatibility.md.
  • Maintenance uses the persisted schema property: switching the FE query implementation to V1 must not select a V1 maintenance writer for an escaped tablet. MV/rollup creation still requires V2.
  • Retain per-segment historical decoding, V2 flat-root compaction, typed marker-0 special-key projection, mixed-encoding rewrites and canonical statistics without relabelling old storage. No V1 escaped-key codec is added.

V1 Variant use-after-free (fixed in this head)

ColumnVariant::get() returns an empty Field for an empty object (and a NULL value). Field::destroy() frees the owned VariantField but leaves its pointer in the Field union storage, so the returned Field has no live VariantField.

Aggregation readers (ReaderReplaceFunction::insert_result_into), insert_from()/insert_indices_from() (RowId fetch) and compaction call ColumnVariant::try_insert() with that Field and dereference get<TYPE_VARIANT>(), reading the freed path map. ASAN reproduced the same heap-use-after-free in vertical compaction, query aggregation and RowId fetch on unescaped V1 tables.

try_insert() now handles a NULL field as a default Variant value, matching ColumnVariantV2::insert(). A unit test inserts the Field returned by get() for a default row. The ASAN UT case and the four neighbouring insert_from cases pass; the three previously crashing SQL suites pass in the full P0 round below.

Test fixtures

V2 execution keeps a JSON string as a string Variant value; the V1 sink parsed it. Seven suites inserted JSON strings into VARIANT columns and then read sub-paths or MATCH/SEARCH indexes, so they now use the existing parse_to_variant convention when enable_variant_v2 is on: predefine_comment_escape, sub_path_order, search_subcolumn_snii, sparse_ancestor_promotion, sparse_ancestor_promotion_extended, snii_compaction, var_index_snii.

dotted_key_legacy_rewrite/_semantics characterize legacy path encoding with V2 execution and explicit variant_escape_dots_in_keys=false; their old V1 form could not insert the V2 value they need. sparse_ancestor_promotion_extended.out (CAST of string/bool elements, empty arrays preserved as []) and dotted_key_legacy_rewrite.out (extended DESC after the heavy rewrite) are regenerated from the current engine.

Validation

Variant P0 full directory (210 suites), ASAN production BE (ASAN_UT, MAKE_TEST=OFF), live FE/BE enable_debug_points=true, global enable_variant_v2=true and storage activation enabled:

  • Default unescaped storage (dotted off): 208/210 pass, 0 fatal, 0 skipped. No BE crash. doc_mode/agg, doc_mode/test_compaction and test_topn_lazy_materialize_sparse_variant pass. The two failures are outfile_csv_variant_doc_value (OSS InvalidAccessKeyId) and duplicate_json_path (V2 rejects duplicate object keys; the case still expects the V1 last-wins input).
  • Temporary test-only default-escaped storage (dotted on, not committed): 187/210 pass. The 23 failures are 15 suites that force enable_variant_v2=false while the temporary default escapes their tables (gate error), 4 fixtures that assume an omitted property creates an unescaped table, 2 single-bracket dotted-literal index cases, plus the duplicate-key and OSS failures. The seven suites above pass and no BE crash occurs.

Earlier P2 evidence on this branch (Release, FE/BE debug points off): 44,273,863 public GitHub Events rows loaded plus 20,000,000 performance rows; 107/110 framework passes. The three failures are two empty-array CAST(... AS STRING) counts (151,471 rows where V2 keeps [] non-NULL) and one JSON object key-order text diff whose ten rows are structurally equal. No .out was rewritten to hide them.

Real Cloud and mixed-old-node runtime execution remain unverified.

…s distinct (apache#69995)

4.2 adaptation: variant_escape_dots_in_keys requires enable_variant_v2=true.
Escaped schemas retain V2 for storage maintenance and reject V1 writer inputs.

(cherry picked from commit b37b47a)
@eldenmoon
eldenmoon requested a review from yiguolei as a code owner November 16, 2026 13:50
@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloudut

@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@eldenmoon

Copy link
Copy Markdown
Member Author

run beut

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloudut

@eldenmoon

Copy link
Copy Markdown
Member Author

run feut

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloudut

2 similar comments
@eldenmoon

Copy link
Copy Markdown
Member Author

run cloudut

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloudut

@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@eldenmoon

Copy link
Copy Markdown
Member Author

run cloudut

@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

…ield

ColumnVariant::get() returns an empty Field for an empty object (and a NULL
value). Field::destroy() frees the owned VariantField but leaves its pointer in
the Field union storage, so the returned Field has no live VariantField.

Aggregation readers (ReaderReplaceFunction::insert_result_into),
insert_from()/insert_indices_from() (RowId fetch) and compaction call
ColumnVariant::try_insert() with that Field and dereference get<TYPE_VARIANT>(),
reading the freed path map. ASAN reproduced the same heap-use-after-free in
vertical compaction, query aggregation and RowId fetch on unescaped V1 tables.

Handle a NULL field as a default Variant value in try_insert(), matching
ColumnVariantV2::insert(), and add a unit test that inserts the Field returned
by get() for a default row.
V2 execution keeps a JSON string as a string Variant value; the V1 sink parsed
it. Seven suites inserted JSON strings into VARIANT columns and then read
sub-paths or MATCH/SEARCH indexes, so they must use the existing
parse_to_variant convention when enable_variant_v2 is on:
predefine_comment_escape, sub_path_order, search_subcolumn_snii,
sparse_ancestor_promotion, sparse_ancestor_promotion_extended, snii_compaction
and var_index_snii.

dotted_key_legacy_rewrite/_semantics characterize legacy path encoding with V2
execution, so they set enable_variant_v2=true and explicit
variant_escape_dots_in_keys=false. Their old V1 form could not insert the
parse_to_variant value it needed.

Regenerate the two expected-output files from the current engine:
sparse_ancestor_promotion_extended.out (CAST of string/bool elements, empty
arrays preserved as []) and dotted_key_legacy_rewrite.out (extended DESC after
the heavy schema rewrite).
@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant