Repository navigation
[feat](refactor-param) refactor brokerLoad's code about fileformat - #50882
Merged
Merged
Conversation
BePPPower
requested review from
dataroaring,
morningman,
xiaokang and
yiguolei
as code owners
May 14, 2025 03:46
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34291 ms |
TPC-DS: Total hot run time: 193146 ms |
ClickBench: Total hot run time: 29.99 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33898 ms |
TPC-DS: Total hot run time: 185342 ms |
ClickBench: Total hot run time: 29.22 s |
morningman
reviewed
May 15, 2025
| getOrigStmt() != null ? getOrigStmt().originStmt : "", copyFromParam.getFileColumns(), | ||
| copyFromParam.getColumnMappingList(), copyFromParam.getFileFilterExpr()); | ||
| } | ||
| dataDescProperties.put(FileFormatProperties.PROP_COMPRESS_TYPE, copyIntoProperties.getCompression()); |
Contributor
There was a problem hiding this comment.
There are some format properties in copyIntoProperties. I think we should remove them too. But may be in next PR.
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34298 ms |
TPC-DS: Total hot run time: 193435 ms |
ClickBench: Total hot run time: 28.92 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33601 ms |
TPC-DS: Total hot run time: 192319 ms |
ClickBench: Total hot run time: 28.72 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34431 ms |
TPC-DS: Total hot run time: 192903 ms |
ClickBench: Total hot run time: 29.17 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34195 ms |
TPC-DS: Total hot run time: 186551 ms |
TPC-H: Total hot run time: 34076 ms |
TPC-DS: Total hot run time: 195464 ms |
ClickBench: Total hot run time: 29.41 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34440 ms |
TPC-DS: Total hot run time: 185738 ms |
ClickBench: Total hot run time: 29.15 s |
Contributor
Author
|
run external |
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
…pache#50882) Issue Number:apache#50238 Problem Summary: Previously, we refactored the code of the fileFormat attribute (apache#50225). However, we only added the relevant code without modifying the business code. This pull request modifies the code of the BrokerLoad feature that is related to the fileformat.
CalvinKirs
pushed a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Jun 27, 2025
…leformat (apache#50882) apache#50882 (cherry picked from commit 0b95a4d)
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Jul 12, 2025
…pache#50882) Issue Number:apache#50238 Problem Summary: Previously, we refactored the code of the fileFormat attribute (apache#50225). However, we only added the relevant code without modifying the business code. This pull request modifies the code of the BrokerLoad feature that is related to the fileformat.
This was referenced Jul 12, 2025
Merged
morrySnow
pushed a commit
that referenced
this pull request
Jul 14, 2025
morningman
added a commit
that referenced
this pull request
Jul 15, 2025
### What problem does this PR solve? Related PR: #50882
morningman
added a commit
to morningman/doris
that referenced
this pull request
Nov 4, 2026
…scape ### What problem does this PR solve? Issue Number: None Related PR: apache#70121 Problem Summary: The previous commit rejects U+FFFD as enclose or escape in `CreateRoutineLoadInfo#checkJobProperties`, which also runs when a job is rebuilt from metadata: `gsonPostProcess` re-parses the stored CREATE statement with `setReplay(true)`, validates it, and turns any failure into the final CANCELLED state. Older versions accept U+FFFD there, and a statement holds it whenever a client that does not send UTF-8 writes a single non-ASCII character, since the FE decodes every statement as UTF-8. So the first restart after an upgrade cancelled such a job for good, and during a rolling upgrade a follower replaying the CREATE of an older master cancelled it on its own. The check now runs only when a statement is admitted, as the compute group and workload group checks beside it do. No enclose or escape from 0x80 up works for a routine load in the first place. Each task is planned from the job's byte turned into text with `new String(new byte[]{b})` and parsed again (`NereidsDataDescription`, since apache#50882). Measured with a throwaway unit test, a task with enclose 0xE9 and escape 0xC3 (what CREATE takes for `é`) gets 0xFD and 0xEF in its Thrift text parameters, and so does every byte from 0x80 up, while every ASCII byte is kept. An escape of 0xEF drops the first byte of full-width characters such as `,`, and routine load validates its data as UTF-8 by default, where a single byte from 0x80 up is absent or part of a character anyway. CREATE and ALTER ROUTINE LOAD therefore take only an ASCII character as enclose or escape (`<key> must be an ASCII character`), which replaces the U+FFFD check and is what the documentation asks for: a single-byte character. A value that is not ASCII and that an older version stored is ignored with a warning, in the same way when the job is rebuilt from the image and when an ALTER is replayed from the edit log; until now only the rebuild ignored U+FFFD, and the ALTER replay parsed it. Such a job keeps no enclose and no escape, which is what it has run with after every restart. CREATE again records the byte as `new String(new byte[]{b})`, which keeps every ASCII byte: the reason for recording the given value instead, that a restored escape of 0xEF would corrupt full-width characters, did not hold, since planning produced 0xEF before any restart as well. Multi-table CSV jobs never applied enclose and escape: `NereidsStreamLoadTask#setMultiTableBaseTaskInfo` lays the job's settings over the backend's per-table request, which does not carry the two, and did not copy them, so the tasks ran without them whatever CREATE or ALTER set. It copies both now. ### Release note CREATE and ALTER ROUTINE LOAD accept only an ASCII character as enclose or escape. Multi-table CSV routine load jobs now apply their enclose and escape, which they ignored before. ### Check List (For Author) - Test: Unit Test / Regression test - `RoutineLoadJobReplayTest#testOlderCreateStatementWithNonAsciiValueReplays` (a stored CREATE statement with U+FFFD or `é` and U+FFFD in the metadata rebuild to NEED_SCHEDULE without enclose and escape), `#testCreateRejectsNonAsciiEncloseAndEscape` and `AlterRoutineLoadCommandTest#testValidateRejectsNonAsciiEncloseAndEscape` (`é`, U+FFFD and `,` under both keys), `RoutineLoadJobReplayTest#testNonAsciiValueAnOlderAlterStoredIsIgnored` (neither the ALTER replay nor the rebuild applies `é` or U+FFFD) and `NereidsStreamLoadTaskTest#testMultiTableBaseTaskCopiesCsvProperties`. `#testCreateRecordsTheGivenEncloseAndEscape` keeps only its half for a job without the two, as `#testCreateWithoutEncloseAndEscapeRecordsTheZeroByte`. - 17 test classes (routine load, ALTER, CSV / JSON file formats, the multi-table overlay, the stream load handler), 167 cases, pass; fe-core Checkstyle reports nothing. Each of these fails only its target cases: checking on the rebuild too, rejecting only U+FFFD, parsing a replayed value without the check, skipping only U+FFFD on the rebuild while the ALTER replay parses every value, and leaving out either copy in the multi-table overlay. - `test_routine_load_property` expects the new error for U+FFFD and for `é` under both keys; parsed offline, not run locally. - Behavior changed: Yes. CREATE and ALTER ROUTINE LOAD reject an enclose or escape that is not ASCII, and multi-table CSV jobs apply theirs. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
morningman
added a commit
that referenced
this pull request
Nov 5, 2026
…tered enclose and escape (#70121) ### What problem does this PR solve? Issue Number: None Related PR: #70118 Problem Summary: **In short.** When an FE rebuilds a routine load job from metadata (a follower replaying the edit log, a restart, a new master), the job loses `max_filter_ratio`, `exec_mem_limit`, `send_batch_parallelism`, `load_to_single_tablet`, `enclose`, `escape` and the `enable_memtable_on_sink_node` it was created with, and runs with defaults from then on. Separately, `ALTER ROUTINE LOAD` on `enclose` / `escape` never reaches the running job, and a multi-table CSV job never applies `enclose` / `escape` at all. This PR restores the values when the job is rebuilt, applies the altered ones, and hands both to the tasks of multi-table jobs. **Background.** A routine load job is persisted with GSON, which writes only the fields annotated with `@SerializedName`; these seven are plain fields. `gsonPostProcess` re-parses the stored CREATE statement, but only to rebuild the `RoutineLoadDesc` (columns, filters, separators, partitions, merge type), and restores the partial update settings from the persisted `jobProperties` map. CREATE records the other values in `jobProperties` as well (all but `exec_mem_limit`), and SHOW ROUTINE LOAD prints them from there, so the loss does not show. Found while making `SHOW CREATE ROUTINE LOAD` print these properties (#70118). **The problem, and what it cost.** Measured on master `856194ece0e` with a unit test that creates jobs from SQL and rebuilds them with a GSON round trip on a thread without a session, as a follower or a restarting FE does: | | Job as created | Job rebuilt from metadata | |---|---|---| | `max_filter_ratio` = 0.5 | 0.5 | 1.0: up to every row may be filtered instead of the job pausing | | `exec_mem_limit` = 4294967296 | 4 GB | 2 GB | | `send_batch_parallelism` = 3, or 4 from the creating session | 3 / 4 | 1 | | `load_to_single_tablet` = true | true | false | | `enclose` `"`, `escape` `\` | `"` / `\` | none: quoted CSV fields are split at the separators inside the quotes | | `enable_memtable_on_sink_node` of the creating session (on by default) | true | false | | `ALTER ... ("enclose" = "'", "escape" = "\|")` | tasks keep `"` / `\` | SHOW ROUTINE LOAD shows the new values, tasks never use them | | `ALTER ... ("enclose" = "ab")` | accepted, without effect | CREATE rejects it | The CSV and filter ratio cases change what gets loaded, without any error, after an event (a failover, a restart) that has nothing to do with the job. A multi-table CSV job (one created without a target table) does not use its `enclose` and `escape` even before it is rebuilt. Each of its tasks is planned from the backend's per-table request, which does not carry them, with the job's settings laid over it by `NereidsStreamLoadTask#setMultiTableBaseTaskInfo`, which copies the separators and the JSON properties but not these two. So its quoted fields are split at the separators inside the quotes, whatever CREATE or ALTER set. **How this PR fixes it.** - `gsonPostProcess` restores `max_filter_ratio`, `send_batch_parallelism`, `load_to_single_tablet`, `enclose` and `escape` from `jobProperties`, in the loop that already restores the partial update settings from there. CREATE has recorded these values since each property was introduced, and ALTER records the new ones; a job created before one of them existed has no key and keeps the default, as today. `exec_mem_limit`, which no ALTER changes and `jobProperties` does not hold, is taken from the CREATE statement that `gsonPostProcess` already re-parses. - CREATE records the creating session's `enable_memtable_on_sink_node` in the job's persisted session variables, next to `sql_mode`, and `gsonPostProcess` restores it. A job created before this has no record and keeps `false`, which is what it has run with after every restart so far. - `ALTER ROUTINE LOAD` checks `enclose` and `escape` the way CREATE does, and `modifyCommonJobProperties`, which Kafka and Kinesis jobs and the replay of an ALTER share, installs them on the job. The two rules move into `CsvFileFormatProperties.parseEnclose` / `parseEscape`, which `analyzeFileFormatProperties` now calls without any change in behavior. - `NereidsStreamLoadTask#setMultiTableBaseTaskInfo` copies `enclose` and `escape` too, so the tasks of a multi-table job use the values CREATE or ALTER set, as those of a single-table job do. - CREATE and ALTER ROUTINE LOAD take only an ASCII character as `enclose` or `escape` (`CreateRoutineLoadInfo#checkCsvControlValue`), which is what the documentation of both properties asks for: a single-byte character. No other value works for a routine load. Each task is planned from the job's byte turned into text with `new String(new byte[]{b})` and parsed again (`NereidsDataDescription`, since #50882), which keeps every ASCII byte but turns every byte from 0x80 up into 0xFD as enclose and 0xEF as escape. A job created with `"escape" = "é"` (0xC3) has therefore run with escape 0xEF from its first task on, which drops the first byte of full-width characters such as `,` in every string cell. And routine load validates its data as UTF-8 by default, where a single byte from 0x80 up is absent or part of a character anyway: 0xC3 as escape would cut the first byte out of every `é`. CREATE records the byte with `new String(new byte[]{b})` as it always has, which keeps every value it now takes. - Values that an older version accepted do not break the rebuild. The ASCII check runs only when a statement is admitted, not when `gsonPostProcess` re-parses a stored CREATE statement, as the compute group and workload group checks beside it already do; failing there would turn the job CANCELLED for good. A value from an older version that CREATE and ALTER now reject, one that is not ASCII (an older CREATE recorded every byte from 0x80 up as U+FFFD) or one that an older ALTER stored without any check, is ignored with a warning instead of failing the replay, in the same way when the job is rebuilt and when the ALTER is replayed. The replay of such an ALTER leaves the job's value as it was, and a rebuilt job keeps no enclose or escape, which is what it has run with after every restart so far; ALTER ROUTINE LOAD can set an ASCII value again. No new persisted field: the only addition is one key in the job's existing session variable map, which older FEs ignore. **Results.** | | Before | After | |---|---|---| | declared properties above, job rebuilt from metadata | defaults | as declared, also in what tasks are built from | | `send_batch_parallelism` and `enable_memtable_on_sink_node` from the creating session | 1 / false | the session's values | | ALTER of `enclose` / `escape` | shown, never used | used by the next task, kept when the job is rebuilt | | `enclose` / `escape` of a multi-table CSV job, from CREATE or ALTER | never used | used by its tasks | | ALTER with `"enclose" = "ab"` | accepted | `enclose should not be longer than one byte.` | | CREATE or ALTER with an `enclose` / `escape` that is not ASCII (`é`, U+FFFD, `,`) | CREATE takes a byte of it, which tasks get as 0xFD / 0xEF from 0x80 up; ALTER accepts it without effect | `enclose must be an ASCII character` | | such a value, or an invalid one, stored by an older version | no effect | no effect, with a warning; the replay of the job and of the ALTER goes on | branch-4.1 and branch-4.2 have the same gaps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Number:#50238
Problem Summary:
Previously, we refactored the code of the fileFormat attribute (#50225). However, we only added the relevant code without modifying the business code. This pull request modifies the code of the BrokerLoad feature that is related to the fileformat.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)