Repository navigation
Conversation
…s distinct (apache#69995) 4.2 adaptation: variant_escape_dots_in_keys requires enable_variant_v2=true. Escaped schemas retain V2 for storage maintenance and reject V1 writer inputs. (cherry picked from commit b37b47a)
Member
Author
|
run buildall |
Member
Author
|
run cloudut |
Member
Author
|
run buildall |
Member
Author
|
run buildall |
Member
Author
|
run beut |
Member
Author
|
run cloudut |
Member
Author
|
run feut |
Member
Author
|
run cloudut |
2 similar comments
Member
Author
|
run cloudut |
Member
Author
|
run cloudut |
Member
Author
|
run buildall |
Member
Author
|
run cloudut |
Member
Author
|
run buildall |
This reverts commit 6b0b793.
Member
Author
|
run buildall |
…ield ColumnVariant::get() returns an empty Field for an empty object (and a NULL value). Field::destroy() frees the owned VariantField but leaves its pointer in the Field union storage, so the returned Field has no live VariantField. Aggregation readers (ReaderReplaceFunction::insert_result_into), insert_from()/insert_indices_from() (RowId fetch) and compaction call ColumnVariant::try_insert() with that Field and dereference get<TYPE_VARIANT>(), reading the freed path map. ASAN reproduced the same heap-use-after-free in vertical compaction, query aggregation and RowId fetch on unescaped V1 tables. Handle a NULL field as a default Variant value in try_insert(), matching ColumnVariantV2::insert(), and add a unit test that inserts the Field returned by get() for a default row.
V2 execution keeps a JSON string as a string Variant value; the V1 sink parsed it. Seven suites inserted JSON strings into VARIANT columns and then read sub-paths or MATCH/SEARCH indexes, so they must use the existing parse_to_variant convention when enable_variant_v2 is on: predefine_comment_escape, sub_path_order, search_subcolumn_snii, sparse_ancestor_promotion, sparse_ancestor_promotion_extended, snii_compaction and var_index_snii. dotted_key_legacy_rewrite/_semantics characterize legacy path encoding with V2 execution, so they set enable_variant_v2=true and explicit variant_escape_dots_in_keys=false. Their old V1 form could not insert the parse_to_variant value it needed. Regenerate the two expected-output files from the current engine: sparse_ancestor_promotion_extended.out (CAST of string/bool elements, empty arrays preserved as []) and dotted_key_legacy_rewrite.out (extended DESC after the heavy schema rewrite).
Member
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry-pick #69995
Preserve literal dotted and percent-containing Variant keys on
branch-4.2. The format stays behind the default-offenable_variant_escape_dots_in_keysactivation gate, and escaped tables execute with the V2 kernel.Compatibility and activation
docs/variant-escaped-key-compatibility.md.V1 Variant use-after-free (fixed in this head)
ColumnVariant::get()returns an emptyFieldfor an empty object (and a NULL value).Field::destroy()frees the ownedVariantFieldbut leaves its pointer in the Field union storage, so the returned Field has no liveVariantField.Aggregation readers (
ReaderReplaceFunction::insert_result_into),insert_from()/insert_indices_from()(RowId fetch) and compaction callColumnVariant::try_insert()with that Field and dereferenceget<TYPE_VARIANT>(), reading the freed path map. ASAN reproduced the same heap-use-after-free in vertical compaction, query aggregation and RowId fetch on unescaped V1 tables.try_insert()now handles a NULL field as a default Variant value, matchingColumnVariantV2::insert(). A unit test inserts the Field returned byget()for a default row. The ASAN UT case and the four neighbouringinsert_fromcases pass; the three previously crashing SQL suites pass in the full P0 round below.Test fixtures
V2 execution keeps a JSON string as a string Variant value; the V1 sink parsed it. Seven suites inserted JSON strings into VARIANT columns and then read sub-paths or MATCH/SEARCH indexes, so they now use the existing
parse_to_variantconvention whenenable_variant_v2is on:predefine_comment_escape,sub_path_order,search_subcolumn_snii,sparse_ancestor_promotion,sparse_ancestor_promotion_extended,snii_compaction,var_index_snii.dotted_key_legacy_rewrite/_semanticscharacterize legacy path encoding with V2 execution and explicitvariant_escape_dots_in_keys=false; their old V1 form could not insert the V2 value they need.sparse_ancestor_promotion_extended.out(CAST of string/bool elements, empty arrays preserved as[]) anddotted_key_legacy_rewrite.out(extended DESC after the heavy rewrite) are regenerated from the current engine.Validation
Variant P0 full directory (210 suites), ASAN production BE (
ASAN_UT,MAKE_TEST=OFF), live FE/BEenable_debug_points=true, globalenable_variant_v2=trueand storage activation enabled:doc_mode/agg,doc_mode/test_compactionandtest_topn_lazy_materialize_sparse_variantpass. The two failures areoutfile_csv_variant_doc_value(OSSInvalidAccessKeyId) andduplicate_json_path(V2 rejects duplicate object keys; the case still expects the V1 last-wins input).enable_variant_v2=falsewhile the temporary default escapes their tables (gate error), 4 fixtures that assume an omitted property creates an unescaped table, 2 single-bracket dotted-literal index cases, plus the duplicate-key and OSS failures. The seven suites above pass and no BE crash occurs.Earlier P2 evidence on this branch (Release, FE/BE debug points off): 44,273,863 public GitHub Events rows loaded plus 20,000,000 performance rows; 107/110 framework passes. The three failures are two empty-array
CAST(... AS STRING)counts (151,471 rows where V2 keeps[]non-NULL) and one JSON object key-order text diff whose ten rows are structurally equal. No.outwas rewritten to hide them.Real Cloud and mixed-old-node runtime execution remain unverified.