Skip to content

[fix](fe) Make SHOW CREATE ROUTINE LOAD output recreate the same job - #70118

Merged
morningman merged 1 commit into
apache:masterfrom
morningman:b2m-t44a-routine-load-show-create
Nov 5, 2026
Merged

morningman merged 1 commit into
apache:masterfrom
morningman:b2m-t44a-routine-load-show-create

Conversation

@morningman

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: #69749

Problem Summary:

In short. SHOW CREATE ROUTINE LOAD prints the statement a job is recreated from, but a job recreated from it was not the same job: eight properties that CREATE accepts were never printed, a value with a quote or a backslash made the statement unparseable, TEMPORARY partitions turned into formal ones, and the comment was dropped. This PR makes the statement carry all of them.

Background. Users recreate a routine load job from SHOW CREATE ROUTINE LOAD to rename it, to move it to another database, or to start it over after STOP; the statement prints the next Kafka offsets to consume for exactly that purpose. #69749 made the statement carry the compute_group a job declares, and found the rest of the list incomplete: RoutineLoadJob.getShowCreateInfo() writes the properties by hand and covered 17 of the 25 keys in CreateRoutineLoadInfo.PROPERTIES_SET.

The problem, and what it cost. Measured on master 1aa505cb0be with a unit test that creates jobs from SQL, recreates each one from its SHOW CREATE output and compares every property:

Job Recreated from its SHOW CREATE output
CSV job with enclose ", escape \, empty_field_as_null, send_batch_parallelism = 3, load_to_single_tablet, workload_group all six back to their defaults; quoted fields get split at the separators inside the quotes
unique_key_update_mode = UPDATE_FLEXIBLE_COLUMNS with partial_update_new_key_behavior = ERROR an UPSERT job with APPEND
partial columns with partial_update_new_key_behavior = ERROR APPEND: rows with new keys get inserted instead of failing the load
JSON job with jsonpaths such as ["$.k1","$.v1"] the statement does not parse (mismatched input '$')
Kafka property whose value holds a ", such as a JAAS config the statement does not parse
TEMPORARY PARTITION(tp1) printed as PARTITION(tp1): Partition tp1 does not exist
COMMENT '...' gone

The jobs themselves keep running as declared; only the statement is wrong. The cost is a recreated job that silently loads differently (CSV parsing, partial update semantics, single-tablet loading, workload group, parallelism), or a statement that cannot be executed at all.

How this PR fixes it.

  • getShowCreateInfo() prints the eight missing properties wherever leaving one out would give the recreated job a different value: unique_key_update_mode when it is not UPSERT (partial_columns alone cannot express UPDATE_FLEXIBLE_COLUMNS), partial_update_new_key_behavior when it is ERROR and a partial update mode is on (CREATE rejects the policy otherwise), enclose and escape when set, empty_field_as_null and load_to_single_tablet when true, and workload_group when set. send_batch_parallelism is always printed, as the timezone is, because CREATE otherwise takes it from the session variable of whoever recreates the job. The values are the ones the job runs with.
  • Keys and values are quoted with SqlLiteralUtils.quoteStringLiteral, which follows the session's NO_BACKSLASH_ESCAPES mode. A value without a quote or a backslash prints exactly as before; the column separator is left as it is, since it is kept as typed and already reads back.
  • TEMPORARY PARTITION(...) keeps TEMPORARY, and the job's comment is printed as COMMENT "...".
  • CREATE ROUTINE LOAD reads its COMMENT with SqlLiteralUtils.parseStringLiteral, as table, view and column comments are read, so that a printed comment with a quote reads back unchanged. It used to strip the outer quotes only, so COMMENT 'it''s' was stored as it''s.

For a job that uses none of these, the output gains a single line, "send_batch_parallelism" = "<the job's value>".

Results.

Job Before After
CSV job above 6 properties and the comment lost recreated with the same definition
flexible partial update with ERROR UPSERT / APPEND same definition
JSON job with jsonpaths statement does not parse same definition
Kafka property with " and \ statement does not parse same definition
temporary partition Partition tp1 does not exist same definition
NO_BACKSLASH_ESCAPES session, enclose ", escape \ both lost same definition

The same test also found that a job's max_filter_ratio, exec_mem_limit, send_batch_parallelism, load_to_single_tablet, enclose and escape fall back to their defaults when a follower replays the job or the FE restarts, and that ALTER ROUTINE LOAD on enclose / escape does not reach the running job. That is a separate defect and will be fixed in its own PR; since this PR prints the values the job runs with, the statement follows that fix. branch-4.1 and branch-4.2 have the same gaps as master.

Release note

SHOW CREATE ROUTINE LOAD now prints send_batch_parallelism, load_to_single_tablet, workload_group, unique_key_update_mode, partial_update_new_key_behavior, enclose, escape, empty_field_as_null, TEMPORARY partitions and the job comment, and quotes property values, so that the statement it prints can be executed again. CREATE ROUTINE LOAD now reads a doubled quote in its COMMENT as a single quote.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

    New RoutineLoadShowCreateRoundTripTest (8 cases on a mocked FE: the CSV job above, flexible and fixed partial updates, defaults that must not be printed, WITH MERGE, a temporary partition, a quoted Kafka property, a NO_BACKSLASH_ESCAPES session) recreates each job from its statement, compares every property and requires the recreated job to print the same statement. compute_group can only be declared in cloud mode and stays covered by RoutineLoadJobTest#testGetShowCreateInfoCarriesDeclaredComputeGroup; a new RoutineLoadJobTest case pins the printed format. 11 routine load test classes, 108 cases, pass. With the two production files reverted to master, 6 of the 8 round trip cases and the 3 exact-output cases fail; the other 2 round trip cases guard that defaults stay out of the statement. Each part of the fix was also removed on its own, and each removal fails its own case.

  • Behavior changed:

    • No.
    • Yes.
      The output of SHOW CREATE ROUTINE LOAD, as in the release note; jobs that use none of the new properties only gain the send_batch_parallelism line. CREATE ROUTINE LOAD reads a doubled quote in COMMENT as one quote.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69749

Problem Summary:

SHOW CREATE ROUTINE LOAD prints the statement a job is recreated from (to
rename it, move it, or start it over after STOP), but a job recreated from
it was not the same job:

- Eight properties that CREATE ROUTINE LOAD accepts were never printed:
  send_batch_parallelism, load_to_single_tablet, workload_group,
  unique_key_update_mode, partial_update_new_key_behavior, enclose, escape
  and empty_field_as_null. A job using UPDATE_FLEXIBLE_COLUMNS came back as
  an UPSERT job, one with partial_update_new_key_behavior ERROR came back
  with APPEND, and a CSV job lost its enclose and escape, so quoted fields
  were split at the separators inside the quotes.
- Values were printed between double quotes without escaping, so the
  statement did not parse when a value held a quote or a backslash: every
  JSON job with jsonpaths such as ["$.k1"], a Kafka property such as a
  JAAS config, and the usual enclose " and escape \.
- TEMPORARY was dropped from TEMPORARY PARTITION(...), so the statement
  named formal partitions that do not exist.
- The job's COMMENT was not printed.

The statement now prints those properties wherever leaving them out would
change the recreated job; send_batch_parallelism is always printed, like
the timezone, because CREATE otherwise takes it from the recreating
session. Keys and values are quoted with SqlLiteralUtils.quoteStringLiteral
under the session's NO_BACKSLASH_ESCAPES mode, TEMPORARY is kept and the
COMMENT is printed. CREATE ROUTINE LOAD now reads its COMMENT as a string
literal, as column comments are, so that a printed comment with quotes
reads back unchanged ('it''s' used to be stored as it''s).

Found while porting the compute_group declaration in apache#69749, which made
the statement carry compute_group; branch-4.1 and branch-4.2 have the
same gaps.

### Release note

SHOW CREATE ROUTINE LOAD now prints send_batch_parallelism,
load_to_single_tablet, workload_group, unique_key_update_mode,
partial_update_new_key_behavior, enclose, escape, empty_field_as_null,
TEMPORARY partitions and the job comment, and quotes property values, so
that the statement it prints can be run again. CREATE ROUTINE LOAD reads
a doubled quote in its COMMENT as a single quote.

### Check List (For Author)

- Test: Unit Test
    - New RoutineLoadShowCreateRoundTripTest recreates 8 kinds of jobs from the statement and compares every property; RoutineLoadJobTest pins the printed format
- Behavior changed: Yes (the output of SHOW CREATE ROUTINE LOAD, and how CREATE ROUTINE LOAD reads a doubled quote in COMMENT)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

Copy link
Copy Markdown
Contributor Author

run buildall

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41631125631

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28116 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 791accac8458bbefd979a270730269cc518d8c29, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17570	5144	5015	5015
q2	2137	292	245	245
q3	10293	1348	813	813
q4	4675	460	337	337
q5	7436	805	529	529
q6	182	182	153	153
q7	753	795	586	586
q8	9258	1315	1030	1030
q9	5459	4176	4152	4152
q10	6809	1278	1010	1010
q11	424	250	224	224
q12	638	394	287	287
q13	18096	3515	2981	2981
q14	274	272	244	244
q15	q16	716	704	663	663
q17	1180	726	569	569
q18	6873	6170	6102	6102
q19	1106	951	593	593
q20	365	328	226	226
q21	5001	2107	2239	2107
q22	340	299	250	250
Total cold run time: 99585 ms
Total hot run time: 28116 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5570	5590	5578	5578
q2	668	554	528	528
q3	5173	5367	4878	4878
q4	1981	2138	1540	1540
q5	5243	5113	5295	5113
q6	268	204	156	156
q7	2260	2071	1858	1858
q8	3062	2721	2626	2626
q9	8092	7960	7962	7960
q10	3739	3617	3466	3466
q11	603	436	428	428
q12	704	690	511	511
q13	3564	3841	3375	3375
q14	325	325	282	282
q15	q16	733	737	655	655
q17	8329	6966	6829	6829
q18	13011	12165	12880	12165
q19	879	794	778	778
q20	2152	2127	1912	1912
q21	5525	4618	4690	4618
q22	481	455	407	407
Total cold run time: 72362 ms
Total hot run time: 65663 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 150947 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 791accac8458bbefd979a270730269cc518d8c29, data reload: false

query5	4313	562	422	422
query6	422	206	192	192
query7	4793	442	230	230
query8	316	169	149	149
query9	8785	3114	3118	3114
query10	482	310	250	250
query11	6039	3818	3568	3568
query12	154	89	85	85
query13	1248	453	333	333
query14	6126	3891	3613	3613
query14_1	3409	3391	3398	3391
query15	205	198	181	181
query16	944	434	406	406
query17	832	650	524	524
query18	2413	418	312	312
query19	206	190	139	139
query20	81	83	83	83
query21	205	126	103	103
query22	12887	12983	12884	12884
query23	13355	12805	12414	12414
query23_1	12456	12552	12439	12439
query24	7360	820	460	460
query24_1	467	487	470	470
query25	510	389	338	338
query26	1266	244	133	133
query27	2787	447	268	268
query28	4625	2015	2017	2015
query29	1539	574	417	417
query30	300	219	180	180
query31	998	926	809	809
query32	133	90	89	89
query33	521	290	230	230
query34	919	842	493	493
query35	732	742	631	631
query36	811	845	719	719
query37	121	92	81	81
query38	1794	1746	1656	1656
query39	795	784	744	744
query39_1	735	740	761	740
query40	218	112	95	95
query41	68	63	62	62
query42	89	84	82	82
query43	348	371	316	316
query44	1344	715	726	715
query45	178	183	164	164
query46	867	978	578	578
query47	2879	2861	2828	2828
query48	297	315	225	225
query49	571	393	288	288
query50	675	277	201	201
query51	10343	10277	10282	10277
query52	77	82	67	67
query53	182	237	147	147
query54	236	184	174	174
query55	69	65	63	63
query56	231	218	215	215
query57	1649	1636	1570	1570
query58	285	257	254	254
query59	2152	2226	2004	2004
query60	297	234	219	219
query61	148	142	146	142
query62	396	339	283	283
query63	181	154	161	154
query64	2749	907	769	769
query65	3345	3256	3275	3256
query66	1772	417	296	296
query67	20327	20123	20187	20123
query68	3046	1001	614	614
query69	373	277	247	247
query70	822	773	770	770
query71	291	229	213	213
query72	2797	2458	2391	2391
query73	521	523	294	294
query74	4621	4526	4360	4360
query75	2339	2281	1927	1927
query76	2329	998	581	581
query77	351	381	330	330
query78	9702	9722	9222	9222
query79	984	892	524	524
query80	540	421	332	332
query81	540	316	266	266
query82	585	131	109	109
query83	303	192	178	178
query84	311	120	103	103
query85	840	415	363	363
query86	328	237	223	223
query87	2031	1980	1820	1820
query88	3754	2758	2746	2746
query89	326	282	239	239
query90	1888	166	164	164
query91	165	151	124	124
query92	97	82	87	82
query93	999	996	572	572
query94	437	303	265	265
query95	559	383	302	302
query96	673	484	234	234
query97	2451	2429	2337	2337
query98	161	159	147	147
query99	761	776	644	644
Total cold run time: 233674 ms
Total hot run time: 150947 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 791accac8458bbefd979a270730269cc518d8c29, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.05	0.06
query3	0.26	0.09	0.08
query4	1.61	0.11	0.10
query5	0.24	0.24	0.26
query6	1.17	0.67	0.67
query7	0.04	0.01	0.01
query8	0.05	0.03	0.04
query9	0.40	0.34	0.35
query10	0.57	0.58	0.57
query11	0.21	0.15	0.15
query12	0.17	0.14	0.14
query13	0.50	0.49	0.49
query14	1.03	1.01	1.02
query15	0.62	0.59	0.60
query16	0.34	0.35	0.34
query17	1.09	1.11	1.16
query18	0.20	0.20	0.20
query19	2.10	2.03	1.98
query20	0.02	0.02	0.01
query21	15.48	0.20	0.13
query22	4.85	0.05	0.06
query23	16.28	0.33	0.12
query24	2.64	0.65	0.42
query25	0.12	0.06	0.04
query26	1.18	0.20	0.15
query27	0.05	0.05	0.05
query28	3.87	0.82	0.37
query29	12.54	4.17	3.23
query30	0.28	0.13	0.10
query31	2.77	0.55	0.33
query32	3.22	0.59	0.50
query33	3.15	3.19	3.29
query34	15.31	3.95	3.34
query35	3.37	3.33	3.33
query36	0.55	0.46	0.43
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.04
query40	0.17	0.16	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.87 s
Total hot run time: 24.23 s

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41654024681

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@morningman

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of head 791accac8458bbefd979a270730269cc518d8c29: one verified P2 finding. The new enclose/escape rendering changes accepted high-bit delimiter bytes on a UTF-8 FE, so the exported statement can still recreate a job with different CSV parsing semantics. The inline comment covers both adjacent conversions. No P0/P1 findings and no existing blocking comments to retain.

Review converged after 2 rounds: both complete-review agents and the separate literal/byte risk reviewer reported NO_NEW_VALUABLE_FINDINGS in round 2. Every candidate and risk has an evidence-backed disposition. All four changed files and their relevant CREATE/SHOW/ALTER/replay and Kafka/Kinesis consumers were inspected. No additional user focus was supplied.

Critical checkpoint conclusions:

  1. Goal and proof: the property, comment and temporary-partition changes address the normal recreation cases. The accepted non-ASCII delimiter case remains incorrect; the new round-trip tests cover ASCII quote/backslash only. Tests were inspected, not executed.
  2. Scope and reuse: two focused production files plus two test files; existing SqlLiteralUtils is reused for matching rendering/decoding.
  3. Concurrency and locks: no new threads, locks, nested lock ordering or external IO. SHOW retains existing field/property access patterns; existing snapshot limitations predate this change.
  4. Lifecycle: no new owned resources, cleanup, callbacks or static initialization dependencies. Comment is installed during creation and stored with the job.
  5. Configuration: no new configuration. Always emitting active send_batch_parallelism correctly removes dependence on the recreating session's default. Literal rendering uses the current SQL mode.
  6. Compatibility: no new persisted schema or function-symbol changes. The stored comment remains in the existing serialized field; origStmt reparse only rebuilds RoutineLoadDesc. The new high-bit byte representation is the reported compatibility defect.
  7. Thrift/protobuf: no changed definitions, ids, enums or wire fields.
  8. Parallel paths: Kafka and Kinesis share the base serializer/setOptional and both pass the same active fields to NereidsRoutineLoadTaskInfo. Declared workload/compute bindings retain their existing intended semantics; cloud compute declaration behavior is unchanged.
  9. Conditional checks: non-UPSERT/new-key-policy gates match CREATE validation; false flags retain false defaults; zero delimiter remains disabled; TEMPORARY matches the parser grammar. No additional issue found.
  10. Test coverage: inspected all eight new JUnit5 round-trip cases and the three exact-output expectations, plus relevant existing group tests. The new tests cover CSV/JSON, fixed/flexible updates, policies/default omissions, merge, temporary partitions, Kafka literals, comments and NO_BACKSLASH_ESCAPES. They do not cover the reported non-ASCII case, restart/ALTER, Kinesis execution or every mode transition; those control paths were inspected statically.
  11. Expected results: updated exact SQL strings agree with rendering order and ordinary escaping. No regression .out artifacts changed.
  12. Observability: existing SHOW and parser/error surfaces are adequate for this synchronous formatting change; no additional logging/metrics needed.
  13. Persistence/failover: JSON read/write, gsonPostProcess and ALTER paths were inspected. Existing optional-field restore and ALTER delimiter propagation defects are outside this diff and were not reposted as new findings.
  14. Data writes/atomicity: no transaction, version-visibility or delete-bitmap logic changes. The reported delimiter mismatch affects the definition of a subsequently recreated writer.
  15. FE/BE passing: no new transmitted variables; existing task-info constructors/getters retain the active byte/mode/policy fields.
  16. Performance and memory: bounded string rendering is linear in output size; no new hot-path loop, significant allocation, RPC or retry behavior. BE memory-tracking paths are untouched.
  17. Other issues/error handling: matching literal decoding, persisted comments, special/default cases and alternative source paths were rechecked. No further substantiated regression or unresolved review candidate remains.

Validation limits: static inspection only. No builds, tests, benchmarks or runtime reproductions were run. Convergence records completion of the bounded static review, not runtime verification or absence of the reported P2 defect.

Review converged after 2 round(s).

appendProperties(sb, JsonFileFormatProperties.PROP_FUZZY_PARSE, isFuzzyParse(), false);
appendProperties(sb, JsonFileFormatProperties.PROP_JSON_ROOT, getJsonRoot(), false);
if (getEnclose() != 0) {
appendProperties(sb, CsvFileFormatProperties.PROP_ENCLOSE, new String(new byte[]{getEnclose()}), false);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve high-bit CSV delimiter bytes when rendering them

On a UTF-8 FE (the default in conf/fe.conf), decoding an isolated high-bit byte with new String(byte[]) produces U+FFFD. CREATE accepts enclose = 'é' and stores byte 0xE9 because CsvFileFormatProperties casts the single character to a byte. This new output prints the replacement character, and recreating it stores 0xFD. The adjacent escape conversion has the same problem: escape = 'é' changes from 0xC3 to 0xEF. The recreated job therefore uses a different CSV parsing byte. Use a reversible representation matching each property's CREATE decoder and add non-ASCII round-trip cases; explicitly selecting UTF-8 alone would preserve the loss.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed here. The byte this line prints is not the byte the job's tasks use either: when a task is planned, NereidsDataDescription (lines 422-423, since #50882) turns the job's enclose and escape back into text with the same new String(new byte[]{...}), and CsvFileFormatProperties parses that text again. Every byte from 0x80 up becomes U+FFFD there: a throwaway unit test got 0xFD as enclose and 0xEF as escape for every such byte through NereidsDataDescription, and for 0xE9 / 0xC3 through the planner a routine load task uses. So on master a job created with "enclose" = "é" (0xE9 on the job) and "escape" = "é" (0xC3) already runs with 0xFD and 0xEF, and the job recreated from the printed U+FFFD (0xFD and 0xEF on the job) runs with the same two bytes. SHOW ROUTINE LOAD shows U+FFFD for both, since CREATE records the byte with the same conversion. The two jobs differ in a field, not in how they parse CSV.

What is wrong is that no byte from 0x80 up means anything as enclose or escape here. Routine load data is validated as UTF-8 by default, and there such a byte is either absent or part of a character: 0xC3 as escape would cut the first byte out of every é. So #70121, which restores these values on replay and applies them on ALTER, now makes CREATE and ALTER ROUTINE LOAD take only an ASCII character for the two (3f9e13b4f74), and ignores the non-ASCII values older versions recorded. Every byte a job can then hold prints and reads back unchanged, whichever PR merges first.

@liaoxin01 liaoxin01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@924060929 924060929 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes SHOW CREATE ROUTINE LOAD preserve job settings when its SQL is used to recreate a job. It adds omitted properties, temporary partitions and comments, and reuses the existing SQL literal encoder/decoder.

Completed an independent static FE review of 791acca against merge-base 856194e, including CREATE/SHOW, Kafka and Kinesis task planning, ALTER/replay boundaries and the added tests. No substantiated PR-introduced P0/P1/P2 findings remain. APPROVE.

The earlier high-bit delimiter finding was rechecked against its reply and the actual planning path. NereidsDataDescription already performs the same byte-to-text conversion in the base version; original and recreated jobs reach the same effective CSV parsing bytes. That existing defect does not block this PR: #70118 (comment).

No builds or tests were run during this review. The supplied current-head snapshot reports successful compilation, FE UT and P0 checks, but FE coverage failed; required-check rules were not supplied. This approval covers the FE static review, not merge readiness.

Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.

@morningman
morningman merged commit 3cf8cc6 into apache:master Nov 5, 2026
39 of 40 checks passed
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

morningman added a commit that referenced this pull request Nov 5, 2026
…tered enclose and escape (#70121)

### What problem does this PR solve?

Issue Number: None

Related PR: #70118

Problem Summary:

**In short.** When an FE rebuilds a routine load job from metadata (a
follower replaying the edit log, a restart, a new master), the job loses
`max_filter_ratio`, `exec_mem_limit`, `send_batch_parallelism`,
`load_to_single_tablet`, `enclose`, `escape` and the
`enable_memtable_on_sink_node` it was created with, and runs with
defaults from then on. Separately, `ALTER ROUTINE LOAD` on `enclose` /
`escape` never reaches the running job, and a multi-table CSV job never
applies `enclose` / `escape` at all. This PR restores the values when
the job is rebuilt, applies the altered ones, and hands both to the
tasks of multi-table jobs.

**Background.** A routine load job is persisted with GSON, which writes
only the fields annotated with `@SerializedName`; these seven are plain
fields. `gsonPostProcess` re-parses the stored CREATE statement, but
only to rebuild the `RoutineLoadDesc` (columns, filters, separators,
partitions, merge type), and restores the partial update settings from
the persisted `jobProperties` map. CREATE records the other values in
`jobProperties` as well (all but `exec_mem_limit`), and SHOW ROUTINE
LOAD prints them from there, so the loss does not show. Found while
making `SHOW CREATE ROUTINE LOAD` print these properties (#70118).

**The problem, and what it cost.** Measured on master `856194ece0e` with
a unit test that creates jobs from SQL and rebuilds them with a GSON
round trip on a thread without a session, as a follower or a restarting
FE does:

| | Job as created | Job rebuilt from metadata |
|---|---|---|
| `max_filter_ratio` = 0.5 | 0.5 | 1.0: up to every row may be filtered
instead of the job pausing |
| `exec_mem_limit` = 4294967296 | 4 GB | 2 GB |
| `send_batch_parallelism` = 3, or 4 from the creating session | 3 / 4 |
1 |
| `load_to_single_tablet` = true | true | false |
| `enclose` `"`, `escape` `\` | `"` / `\` | none: quoted CSV fields are
split at the separators inside the quotes |
| `enable_memtable_on_sink_node` of the creating session (on by default)
| true | false |
| `ALTER ... ("enclose" = "'", "escape" = "\|")` | tasks keep `"` / `\`
| SHOW ROUTINE LOAD shows the new values, tasks never use them |
| `ALTER ... ("enclose" = "ab")` | accepted, without effect | CREATE
rejects it |

The CSV and filter ratio cases change what gets loaded, without any
error, after an event (a failover, a restart) that has nothing to do
with the job.

A multi-table CSV job (one created without a target table) does not use
its `enclose` and `escape` even before it is rebuilt. Each of its tasks
is planned from the backend's per-table request, which does not carry
them, with the job's settings laid over it by
`NereidsStreamLoadTask#setMultiTableBaseTaskInfo`, which copies the
separators and the JSON properties but not these two. So its quoted
fields are split at the separators inside the quotes, whatever CREATE or
ALTER set.

**How this PR fixes it.**

- `gsonPostProcess` restores `max_filter_ratio`,
`send_batch_parallelism`, `load_to_single_tablet`, `enclose` and
`escape` from `jobProperties`, in the loop that already restores the
partial update settings from there. CREATE has recorded these values
since each property was introduced, and ALTER records the new ones; a
job created before one of them existed has no key and keeps the default,
as today. `exec_mem_limit`, which no ALTER changes and `jobProperties`
does not hold, is taken from the CREATE statement that `gsonPostProcess`
already re-parses.
- CREATE records the creating session's `enable_memtable_on_sink_node`
in the job's persisted session variables, next to `sql_mode`, and
`gsonPostProcess` restores it. A job created before this has no record
and keeps `false`, which is what it has run with after every restart so
far.
- `ALTER ROUTINE LOAD` checks `enclose` and `escape` the way CREATE
does, and `modifyCommonJobProperties`, which Kafka and Kinesis jobs and
the replay of an ALTER share, installs them on the job. The two rules
move into `CsvFileFormatProperties.parseEnclose` / `parseEscape`, which
`analyzeFileFormatProperties` now calls without any change in behavior.
- `NereidsStreamLoadTask#setMultiTableBaseTaskInfo` copies `enclose` and
`escape` too, so the tasks of a multi-table job use the values CREATE or
ALTER set, as those of a single-table job do.
- CREATE and ALTER ROUTINE LOAD take only an ASCII character as
`enclose` or `escape` (`CreateRoutineLoadInfo#checkCsvControlValue`),
which is what the documentation of both properties asks for: a
single-byte character. No other value works for a routine load. Each
task is planned from the job's byte turned into text with `new
String(new byte[]{b})` and parsed again (`NereidsDataDescription`, since
#50882), which keeps every ASCII byte but turns every byte from 0x80 up
into 0xFD as enclose and 0xEF as escape. A job created with `"escape" =
"é"` (0xC3) has therefore run with escape 0xEF from its first task on,
which drops the first byte of full-width characters such as `,` in every
string cell. And routine load validates its data as UTF-8 by default,
where a single byte from 0x80 up is absent or part of a character
anyway: 0xC3 as escape would cut the first byte out of every `é`. CREATE
records the byte with `new String(new byte[]{b})` as it always has,
which keeps every value it now takes.
- Values that an older version accepted do not break the rebuild. The
ASCII check runs only when a statement is admitted, not when
`gsonPostProcess` re-parses a stored CREATE statement, as the compute
group and workload group checks beside it already do; failing there
would turn the job CANCELLED for good. A value from an older version
that CREATE and ALTER now reject, one that is not ASCII (an older CREATE
recorded every byte from 0x80 up as U+FFFD) or one that an older ALTER
stored without any check, is ignored with a warning instead of failing
the replay, in the same way when the job is rebuilt and when the ALTER
is replayed. The replay of such an ALTER leaves the job's value as it
was, and a rebuilt job keeps no enclose or escape, which is what it has
run with after every restart so far; ALTER ROUTINE LOAD can set an ASCII
value again.

No new persisted field: the only addition is one key in the job's
existing session variable map, which older FEs ignore.

**Results.**

| | Before | After |
|---|---|---|
| declared properties above, job rebuilt from metadata | defaults | as
declared, also in what tasks are built from |
| `send_batch_parallelism` and `enable_memtable_on_sink_node` from the
creating session | 1 / false | the session's values |
| ALTER of `enclose` / `escape` | shown, never used | used by the next
task, kept when the job is rebuilt |
| `enclose` / `escape` of a multi-table CSV job, from CREATE or ALTER |
never used | used by its tasks |
| ALTER with `"enclose" = "ab"` | accepted | `enclose should not be
longer than one byte.` |
| CREATE or ALTER with an `enclose` / `escape` that is not ASCII (`é`,
U+FFFD, `,`) | CREATE takes a byte of it, which tasks get as 0xFD / 0xEF
from 0x80 up; ALTER accepts it without effect | `enclose must be an
ASCII character` |
| such a value, or an invalid one, stored by an older version | no
effect | no effect, with a warning; the replay of the job and of the
ALTER goes on |

branch-4.1 and branch-4.2 have the same gaps.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. backport/4.2-master reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants