Repository navigation
[fix](fe) Make SHOW CREATE ROUTINE LOAD output recreate the same job - #70118
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#69749 Problem Summary: SHOW CREATE ROUTINE LOAD prints the statement a job is recreated from (to rename it, move it, or start it over after STOP), but a job recreated from it was not the same job: - Eight properties that CREATE ROUTINE LOAD accepts were never printed: send_batch_parallelism, load_to_single_tablet, workload_group, unique_key_update_mode, partial_update_new_key_behavior, enclose, escape and empty_field_as_null. A job using UPDATE_FLEXIBLE_COLUMNS came back as an UPSERT job, one with partial_update_new_key_behavior ERROR came back with APPEND, and a CSV job lost its enclose and escape, so quoted fields were split at the separators inside the quotes. - Values were printed between double quotes without escaping, so the statement did not parse when a value held a quote or a backslash: every JSON job with jsonpaths such as ["$.k1"], a Kafka property such as a JAAS config, and the usual enclose " and escape \. - TEMPORARY was dropped from TEMPORARY PARTITION(...), so the statement named formal partitions that do not exist. - The job's COMMENT was not printed. The statement now prints those properties wherever leaving them out would change the recreated job; send_batch_parallelism is always printed, like the timezone, because CREATE otherwise takes it from the recreating session. Keys and values are quoted with SqlLiteralUtils.quoteStringLiteral under the session's NO_BACKSLASH_ESCAPES mode, TEMPORARY is kept and the COMMENT is printed. CREATE ROUTINE LOAD now reads its COMMENT as a string literal, as column comments are, so that a printed comment with quotes reads back unchanged ('it''s' used to be stored as it''s). Found while porting the compute_group declaration in apache#69749, which made the statement carry compute_group; branch-4.1 and branch-4.2 have the same gaps. ### Release note SHOW CREATE ROUTINE LOAD now prints send_batch_parallelism, load_to_single_tablet, workload_group, unique_key_update_mode, partial_update_new_key_behavior, enclose, escape, empty_field_as_null, TEMPORARY partitions and the job comment, and quotes property values, so that the statement it prints can be run again. CREATE ROUTINE LOAD reads a doubled quote in its COMMENT as a single quote. ### Check List (For Author) - Test: Unit Test - New RoutineLoadShowCreateRoundTripTest recreates 8 kinds of jobs from the statement and compares every property; RoutineLoadJobTest pins the printed format - Behavior changed: Yes (the output of SHOW CREATE ROUTINE LOAD, and how CREATE ROUTINE LOAD reads a doubled quote in COMMENT) - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
TPC-H: Total hot run time: 28116 ms |
TPC-DS: Total hot run time: 150947 ms |
ClickBench: Total hot run time: 24.23 s |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
There was a problem hiding this comment.
Static review of head 791accac8458bbefd979a270730269cc518d8c29: one verified P2 finding. The new enclose/escape rendering changes accepted high-bit delimiter bytes on a UTF-8 FE, so the exported statement can still recreate a job with different CSV parsing semantics. The inline comment covers both adjacent conversions. No P0/P1 findings and no existing blocking comments to retain.
Review converged after 2 rounds: both complete-review agents and the separate literal/byte risk reviewer reported NO_NEW_VALUABLE_FINDINGS in round 2. Every candidate and risk has an evidence-backed disposition. All four changed files and their relevant CREATE/SHOW/ALTER/replay and Kafka/Kinesis consumers were inspected. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: the property, comment and temporary-partition changes address the normal recreation cases. The accepted non-ASCII delimiter case remains incorrect; the new round-trip tests cover ASCII quote/backslash only. Tests were inspected, not executed.
- Scope and reuse: two focused production files plus two test files; existing SqlLiteralUtils is reused for matching rendering/decoding.
- Concurrency and locks: no new threads, locks, nested lock ordering or external IO. SHOW retains existing field/property access patterns; existing snapshot limitations predate this change.
- Lifecycle: no new owned resources, cleanup, callbacks or static initialization dependencies. Comment is installed during creation and stored with the job.
- Configuration: no new configuration. Always emitting active send_batch_parallelism correctly removes dependence on the recreating session's default. Literal rendering uses the current SQL mode.
- Compatibility: no new persisted schema or function-symbol changes. The stored comment remains in the existing serialized field; origStmt reparse only rebuilds RoutineLoadDesc. The new high-bit byte representation is the reported compatibility defect.
- Thrift/protobuf: no changed definitions, ids, enums or wire fields.
- Parallel paths: Kafka and Kinesis share the base serializer/setOptional and both pass the same active fields to NereidsRoutineLoadTaskInfo. Declared workload/compute bindings retain their existing intended semantics; cloud compute declaration behavior is unchanged.
- Conditional checks: non-UPSERT/new-key-policy gates match CREATE validation; false flags retain false defaults; zero delimiter remains disabled; TEMPORARY matches the parser grammar. No additional issue found.
- Test coverage: inspected all eight new JUnit5 round-trip cases and the three exact-output expectations, plus relevant existing group tests. The new tests cover CSV/JSON, fixed/flexible updates, policies/default omissions, merge, temporary partitions, Kafka literals, comments and NO_BACKSLASH_ESCAPES. They do not cover the reported non-ASCII case, restart/ALTER, Kinesis execution or every mode transition; those control paths were inspected statically.
- Expected results: updated exact SQL strings agree with rendering order and ordinary escaping. No regression .out artifacts changed.
- Observability: existing SHOW and parser/error surfaces are adequate for this synchronous formatting change; no additional logging/metrics needed.
- Persistence/failover: JSON read/write, gsonPostProcess and ALTER paths were inspected. Existing optional-field restore and ALTER delimiter propagation defects are outside this diff and were not reposted as new findings.
- Data writes/atomicity: no transaction, version-visibility or delete-bitmap logic changes. The reported delimiter mismatch affects the definition of a subsequently recreated writer.
- FE/BE passing: no new transmitted variables; existing task-info constructors/getters retain the active byte/mode/policy fields.
- Performance and memory: bounded string rendering is linear in output size; no new hot-path loop, significant allocation, RPC or retry behavior. BE memory-tracking paths are untouched.
- Other issues/error handling: matching literal decoding, persisted comments, special/default cases and alternative source paths were rechecked. No further substantiated regression or unresolved review candidate remains.
Validation limits: static inspection only. No builds, tests, benchmarks or runtime reproductions were run. Convergence records completion of the bounded static review, not runtime verification or absence of the reported P2 defect.
Review converged after 2 round(s).
| appendProperties(sb, JsonFileFormatProperties.PROP_FUZZY_PARSE, isFuzzyParse(), false); | ||
| appendProperties(sb, JsonFileFormatProperties.PROP_JSON_ROOT, getJsonRoot(), false); | ||
| if (getEnclose() != 0) { | ||
| appendProperties(sb, CsvFileFormatProperties.PROP_ENCLOSE, new String(new byte[]{getEnclose()}), false); |
There was a problem hiding this comment.
[P2] Preserve high-bit CSV delimiter bytes when rendering them
On a UTF-8 FE (the default in conf/fe.conf), decoding an isolated high-bit byte with new String(byte[]) produces U+FFFD. CREATE accepts enclose = 'é' and stores byte 0xE9 because CsvFileFormatProperties casts the single character to a byte. This new output prints the replacement character, and recreating it stores 0xFD. The adjacent escape conversion has the same problem: escape = 'é' changes from 0xC3 to 0xEF. The recreated job therefore uses a different CSV parsing byte. Use a reversible representation matching each property's CREATE decoder and add non-ASCII round-trip cases; explicitly selecting UTF-8 alone would preserve the loss.
There was a problem hiding this comment.
Not changed here. The byte this line prints is not the byte the job's tasks use either: when a task is planned, NereidsDataDescription (lines 422-423, since #50882) turns the job's enclose and escape back into text with the same new String(new byte[]{...}), and CsvFileFormatProperties parses that text again. Every byte from 0x80 up becomes U+FFFD there: a throwaway unit test got 0xFD as enclose and 0xEF as escape for every such byte through NereidsDataDescription, and for 0xE9 / 0xC3 through the planner a routine load task uses. So on master a job created with "enclose" = "é" (0xE9 on the job) and "escape" = "é" (0xC3) already runs with 0xFD and 0xEF, and the job recreated from the printed U+FFFD (0xFD and 0xEF on the job) runs with the same two bytes. SHOW ROUTINE LOAD shows U+FFFD for both, since CREATE records the byte with the same conversion. The two jobs differ in a field, not in how they parse CSV.
What is wrong is that no byte from 0x80 up means anything as enclose or escape here. Routine load data is validated as UTF-8 by default, and there such a byte is either absent or part of a character: 0xC3 as escape would cut the first byte out of every é. So #70121, which restores these values on replay and applies them on ALTER, now makes CREATE and ALTER ROUTINE LOAD take only an ASCII character for the two (3f9e13b4f74), and ignores the non-ASCII values older versions recorded. Every byte a job can then hold prints and reads back unchanged, whichever PR merges first.
924060929
left a comment
There was a problem hiding this comment.
This change makes SHOW CREATE ROUTINE LOAD preserve job settings when its SQL is used to recreate a job. It adds omitted properties, temporary partitions and comments, and reuses the existing SQL literal encoder/decoder.
Completed an independent static FE review of 791acca against merge-base 856194e, including CREATE/SHOW, Kafka and Kinesis task planning, ALTER/replay boundaries and the added tests. No substantiated PR-introduced P0/P1/P2 findings remain. APPROVE.
The earlier high-bit delimiter finding was rechecked against its reply and the actual planning path. NereidsDataDescription already performs the same byte-to-text conversion in the base version; original and recreated jobs reach the same effective CSV parsing bytes. That existing defect does not block this PR: #70118 (comment).
No builds or tests were run during this review. The supplied current-head snapshot reports successful compilation, FE UT and P0 checks, but FE coverage failed; required-check rules were not supplied. This approval covers the FE static review, not merge readiness.
Automated FE-focused static review (gpt-6.1-sol, high). No build or tests were run.
|
PR approved by anyone and no changes requested. |
|
PR approved by at least one committer and no changes requested. |
…tered enclose and escape (#70121) ### What problem does this PR solve? Issue Number: None Related PR: #70118 Problem Summary: **In short.** When an FE rebuilds a routine load job from metadata (a follower replaying the edit log, a restart, a new master), the job loses `max_filter_ratio`, `exec_mem_limit`, `send_batch_parallelism`, `load_to_single_tablet`, `enclose`, `escape` and the `enable_memtable_on_sink_node` it was created with, and runs with defaults from then on. Separately, `ALTER ROUTINE LOAD` on `enclose` / `escape` never reaches the running job, and a multi-table CSV job never applies `enclose` / `escape` at all. This PR restores the values when the job is rebuilt, applies the altered ones, and hands both to the tasks of multi-table jobs. **Background.** A routine load job is persisted with GSON, which writes only the fields annotated with `@SerializedName`; these seven are plain fields. `gsonPostProcess` re-parses the stored CREATE statement, but only to rebuild the `RoutineLoadDesc` (columns, filters, separators, partitions, merge type), and restores the partial update settings from the persisted `jobProperties` map. CREATE records the other values in `jobProperties` as well (all but `exec_mem_limit`), and SHOW ROUTINE LOAD prints them from there, so the loss does not show. Found while making `SHOW CREATE ROUTINE LOAD` print these properties (#70118). **The problem, and what it cost.** Measured on master `856194ece0e` with a unit test that creates jobs from SQL and rebuilds them with a GSON round trip on a thread without a session, as a follower or a restarting FE does: | | Job as created | Job rebuilt from metadata | |---|---|---| | `max_filter_ratio` = 0.5 | 0.5 | 1.0: up to every row may be filtered instead of the job pausing | | `exec_mem_limit` = 4294967296 | 4 GB | 2 GB | | `send_batch_parallelism` = 3, or 4 from the creating session | 3 / 4 | 1 | | `load_to_single_tablet` = true | true | false | | `enclose` `"`, `escape` `\` | `"` / `\` | none: quoted CSV fields are split at the separators inside the quotes | | `enable_memtable_on_sink_node` of the creating session (on by default) | true | false | | `ALTER ... ("enclose" = "'", "escape" = "\|")` | tasks keep `"` / `\` | SHOW ROUTINE LOAD shows the new values, tasks never use them | | `ALTER ... ("enclose" = "ab")` | accepted, without effect | CREATE rejects it | The CSV and filter ratio cases change what gets loaded, without any error, after an event (a failover, a restart) that has nothing to do with the job. A multi-table CSV job (one created without a target table) does not use its `enclose` and `escape` even before it is rebuilt. Each of its tasks is planned from the backend's per-table request, which does not carry them, with the job's settings laid over it by `NereidsStreamLoadTask#setMultiTableBaseTaskInfo`, which copies the separators and the JSON properties but not these two. So its quoted fields are split at the separators inside the quotes, whatever CREATE or ALTER set. **How this PR fixes it.** - `gsonPostProcess` restores `max_filter_ratio`, `send_batch_parallelism`, `load_to_single_tablet`, `enclose` and `escape` from `jobProperties`, in the loop that already restores the partial update settings from there. CREATE has recorded these values since each property was introduced, and ALTER records the new ones; a job created before one of them existed has no key and keeps the default, as today. `exec_mem_limit`, which no ALTER changes and `jobProperties` does not hold, is taken from the CREATE statement that `gsonPostProcess` already re-parses. - CREATE records the creating session's `enable_memtable_on_sink_node` in the job's persisted session variables, next to `sql_mode`, and `gsonPostProcess` restores it. A job created before this has no record and keeps `false`, which is what it has run with after every restart so far. - `ALTER ROUTINE LOAD` checks `enclose` and `escape` the way CREATE does, and `modifyCommonJobProperties`, which Kafka and Kinesis jobs and the replay of an ALTER share, installs them on the job. The two rules move into `CsvFileFormatProperties.parseEnclose` / `parseEscape`, which `analyzeFileFormatProperties` now calls without any change in behavior. - `NereidsStreamLoadTask#setMultiTableBaseTaskInfo` copies `enclose` and `escape` too, so the tasks of a multi-table job use the values CREATE or ALTER set, as those of a single-table job do. - CREATE and ALTER ROUTINE LOAD take only an ASCII character as `enclose` or `escape` (`CreateRoutineLoadInfo#checkCsvControlValue`), which is what the documentation of both properties asks for: a single-byte character. No other value works for a routine load. Each task is planned from the job's byte turned into text with `new String(new byte[]{b})` and parsed again (`NereidsDataDescription`, since #50882), which keeps every ASCII byte but turns every byte from 0x80 up into 0xFD as enclose and 0xEF as escape. A job created with `"escape" = "é"` (0xC3) has therefore run with escape 0xEF from its first task on, which drops the first byte of full-width characters such as `,` in every string cell. And routine load validates its data as UTF-8 by default, where a single byte from 0x80 up is absent or part of a character anyway: 0xC3 as escape would cut the first byte out of every `é`. CREATE records the byte with `new String(new byte[]{b})` as it always has, which keeps every value it now takes. - Values that an older version accepted do not break the rebuild. The ASCII check runs only when a statement is admitted, not when `gsonPostProcess` re-parses a stored CREATE statement, as the compute group and workload group checks beside it already do; failing there would turn the job CANCELLED for good. A value from an older version that CREATE and ALTER now reject, one that is not ASCII (an older CREATE recorded every byte from 0x80 up as U+FFFD) or one that an older ALTER stored without any check, is ignored with a warning instead of failing the replay, in the same way when the job is rebuilt and when the ALTER is replayed. The replay of such an ALTER leaves the job's value as it was, and a rebuilt job keeps no enclose or escape, which is what it has run with after every restart so far; ALTER ROUTINE LOAD can set an ASCII value again. No new persisted field: the only addition is one key in the job's existing session variable map, which older FEs ignore. **Results.** | | Before | After | |---|---|---| | declared properties above, job rebuilt from metadata | defaults | as declared, also in what tasks are built from | | `send_batch_parallelism` and `enable_memtable_on_sink_node` from the creating session | 1 / false | the session's values | | ALTER of `enclose` / `escape` | shown, never used | used by the next task, kept when the job is rebuilt | | `enclose` / `escape` of a multi-table CSV job, from CREATE or ALTER | never used | used by its tasks | | ALTER with `"enclose" = "ab"` | accepted | `enclose should not be longer than one byte.` | | CREATE or ALTER with an `enclose` / `escape` that is not ASCII (`é`, U+FFFD, `,`) | CREATE takes a byte of it, which tasks get as 0xFD / 0xEF from 0x80 up; ALTER accepts it without effect | `enclose must be an ASCII character` | | such a value, or an invalid one, stored by an older version | no effect | no effect, with a warning; the replay of the job and of the ALTER goes on | branch-4.1 and branch-4.2 have the same gaps.
What problem does this PR solve?
Issue Number: None
Related PR: #69749
Problem Summary:
In short.
SHOW CREATE ROUTINE LOADprints the statement a job is recreated from, but a job recreated from it was not the same job: eight properties that CREATE accepts were never printed, a value with a quote or a backslash made the statement unparseable,TEMPORARYpartitions turned into formal ones, and the comment was dropped. This PR makes the statement carry all of them.Background. Users recreate a routine load job from
SHOW CREATE ROUTINE LOADto rename it, to move it to another database, or to start it over afterSTOP; the statement prints the next Kafka offsets to consume for exactly that purpose. #69749 made the statement carry thecompute_groupa job declares, and found the rest of the list incomplete:RoutineLoadJob.getShowCreateInfo()writes the properties by hand and covered 17 of the 25 keys inCreateRoutineLoadInfo.PROPERTIES_SET.The problem, and what it cost. Measured on master
1aa505cb0bewith a unit test that creates jobs from SQL, recreates each one from itsSHOW CREATEoutput and compares every property:enclose",escape\,empty_field_as_null,send_batch_parallelism= 3,load_to_single_tablet,workload_groupunique_key_update_mode=UPDATE_FLEXIBLE_COLUMNSwithpartial_update_new_key_behavior=ERRORUPSERTjob withAPPENDpartial_update_new_key_behavior=ERRORAPPEND: rows with new keys get inserted instead of failing the loadjsonpathssuch as["$.k1","$.v1"]mismatched input '$')", such as a JAAS configTEMPORARY PARTITION(tp1)PARTITION(tp1):Partition tp1 does not existCOMMENT '...'The jobs themselves keep running as declared; only the statement is wrong. The cost is a recreated job that silently loads differently (CSV parsing, partial update semantics, single-tablet loading, workload group, parallelism), or a statement that cannot be executed at all.
How this PR fixes it.
getShowCreateInfo()prints the eight missing properties wherever leaving one out would give the recreated job a different value:unique_key_update_modewhen it is notUPSERT(partial_columnsalone cannot expressUPDATE_FLEXIBLE_COLUMNS),partial_update_new_key_behaviorwhen it isERRORand a partial update mode is on (CREATE rejects the policy otherwise),encloseandescapewhen set,empty_field_as_nullandload_to_single_tabletwhen true, andworkload_groupwhen set.send_batch_parallelismis always printed, as the timezone is, because CREATE otherwise takes it from the session variable of whoever recreates the job. The values are the ones the job runs with.SqlLiteralUtils.quoteStringLiteral, which follows the session'sNO_BACKSLASH_ESCAPESmode. A value without a quote or a backslash prints exactly as before; the column separator is left as it is, since it is kept as typed and already reads back.TEMPORARY PARTITION(...)keepsTEMPORARY, and the job's comment is printed asCOMMENT "...".CREATE ROUTINE LOADreads itsCOMMENTwithSqlLiteralUtils.parseStringLiteral, as table, view and column comments are read, so that a printed comment with a quote reads back unchanged. It used to strip the outer quotes only, soCOMMENT 'it''s'was stored asit''s.For a job that uses none of these, the output gains a single line,
"send_batch_parallelism" = "<the job's value>".Results.
ERRORUPSERT/APPENDjsonpaths"and\Partition tp1 does not existNO_BACKSLASH_ESCAPESsession,enclose",escape\The same test also found that a job's
max_filter_ratio,exec_mem_limit,send_batch_parallelism,load_to_single_tablet,encloseandescapefall back to their defaults when a follower replays the job or the FE restarts, and thatALTER ROUTINE LOADonenclose/escapedoes not reach the running job. That is a separate defect and will be fixed in its own PR; since this PR prints the values the job runs with, the statement follows that fix. branch-4.1 and branch-4.2 have the same gaps as master.Release note
SHOW CREATE ROUTINE LOAD now prints send_batch_parallelism, load_to_single_tablet, workload_group, unique_key_update_mode, partial_update_new_key_behavior, enclose, escape, empty_field_as_null, TEMPORARY partitions and the job comment, and quotes property values, so that the statement it prints can be executed again. CREATE ROUTINE LOAD now reads a doubled quote in its COMMENT as a single quote.
Check List (For Author)
Test
New
RoutineLoadShowCreateRoundTripTest(8 cases on a mocked FE: the CSV job above, flexible and fixed partial updates, defaults that must not be printed,WITH MERGE, a temporary partition, a quoted Kafka property, aNO_BACKSLASH_ESCAPESsession) recreates each job from its statement, compares every property and requires the recreated job to print the same statement.compute_groupcan only be declared in cloud mode and stays covered byRoutineLoadJobTest#testGetShowCreateInfoCarriesDeclaredComputeGroup; a newRoutineLoadJobTestcase pins the printed format. 11 routine load test classes, 108 cases, pass. With the two production files reverted to master, 6 of the 8 round trip cases and the 3 exact-output cases fail; the other 2 round trip cases guard that defaults stay out of the statement. Each part of the fix was also removed on its own, and each removal fails its own case.Behavior changed:
The output of SHOW CREATE ROUTINE LOAD, as in the release note; jobs that use none of the new properties only gain the
send_batch_parallelismline. CREATE ROUTINE LOAD reads a doubled quote in COMMENT as one quote.Does this need documentation?
Check List (For Reviewer who merge this PR)