Repository navigation
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 28749 ms |
TPC-DS: Total hot run time: 150970 ms |
ClickBench: Total hot run time: 24.05 s |
FE Regression Coverage ReportIncrement line coverage |
|
/review |
|
Thank you, @Aranya7, for investigating this issue and contributing the fix and recovery tests. I compared this PR with #70121, which was merged into master on November 5. That change addresses the recovery of all four properties covered here: The merged implementation restores the filter ratio, parallelism, and single-tablet setting from the persisted job properties, preserving an altered filter ratio. It restores I'm closing this PR as superseded by #70121. We appreciate the work you put into identifying the problem and covering the recovery scenarios. If you find a case that remains affected after #70121, please share it so we can follow up. |
|
Codex automated review failed and did not complete. Error: Review recovery stopped: No final review submission was declared Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
What problem does this PR solve?
Issue Number: close #69928
Related PR: N/A
Problem Summary:
Routine Load jobs can silently revert to default execution properties after a Frontend (FE) restart or metadata recovery, even when nondefault values were explicitly configured during job creation.
The affected properties include:
max_filter_ratioexec_mem_limitsend_batch_parallelismload_to_single_tabletThese properties are correctly initialized when a job is created. However, their corresponding runtime fields are not independently serialized, and the existing metadata recovery path does not restore them from the persisted configuration.
As a result, a successfully recovered Routine Load job may execute with settings that differ from its user-defined configuration, without an explicit
ALTER ROUTINE LOADoperation.Impact
This issue can affect ingestion correctness and operational reliability:
max_filter_ratiocan revert from a strict value such as0.0to the default1.0, allowing a substantially higher proportion of filtered rows to be tolerated than intended.exec_mem_limitcan revert to its default, potentially changing task memory consumption or causing unexpected memory-limit failures.send_batch_parallelismcan revert to its default, changing batch submission parallelism and throughput.load_to_single_tabletcan revert tofalse, changing the configured loading strategy.The primary concern is that an FE restart can silently alter ingestion semantics. This PR addresses that risk; it does not claim that a specific production data-loss incident has occurred.
Root Cause
RoutineLoadJobmaintains the affected configuration values in runtime fields that are not independently persisted in FE metadata.Although some of these values are available in the serialized
jobPropertiesmap,gsonPostProcess()reconstructs the Routine Load description without restoring the corresponding runtime fields.Additionally,
exec_mem_limitwas not previously recorded injobPropertiesduring job creation.Consequently, metadata deserialization can reconstruct a valid job object while leaving its execution properties initialized to defaults.
Changes
Persist the effective execution memory limit
exec_mem_limitinjobPropertiesduring Routine Load job creation.Restore runtime configuration during metadata recovery
max_filter_ratio,exec_mem_limit,send_batch_parallelism, andload_to_single_tabletduringgsonPostProcess().jobProperties, including values modified through ALTER.Preserve ALTER semantics
max_filter_ratiowas initially0.5and subsequently altered to0.0, the recovered value remains0.0.Validate recovered properties
Add regression coverage
Backward Compatibility
The fix uses the existing serialized
jobPropertiesmap and the original CREATE statement. It does not introduce new metadata fields or require a serialization-format migration.For existing jobs whose metadata does not contain
exec_mem_limit, the value can be recovered from the original CREATE statement when available.This is particularly important during upgrades: an existing job may have been executing with an unintended default after an earlier FE restart. Once the fix is applied, its effective configuration can return to the value originally specified by the user.
The change is limited to configuration persistence and recovery. It does not intentionally modify job identity, Kafka partition offsets, ingestion progress, or task scheduling logic.
Release note
Fix Routine Load jobs unexpectedly reverting to default execution properties after FE restart or metadata recovery.
Check List (For Author)
Unit Test
Added
RoutineLoadConfigRecoveryTestto cover:exec_mem_limitduring job creation.Existing Routine Load unit tests should also be executed to check for regressions.
Manual Test
Start an unpatched FE and create a Routine Load job with:
PROPERTIES ( "max_filter_ratio" = "0.5", "exec_mem_limit" = "3333333333", "send_batch_parallelism" = "2", "load_to_single_tablet" = "true" )Pause the job and modify its filter ratio:
ALTER ROUTINE LOAD FOR rl_recovery_legacy PROPERTIES ( "max_filter_ratio" = "0.0" );Capture its configuration, job ID, state, and Kafka progress using
SHOW ROUTINE LOAD.Gracefully stop the unpatched FE, preserve its metadata, and start the patched FE using the same metadata.
Verify the recovered job retains:
max_filter_ratio0.0exec_mem_limit3333333333send_batch_parallelism2load_to_single_tablettrueVerify that the job ID, paused state, and stored Kafka offsets remain unchanged.
Create a new Routine Load job using the patched FE with the same nondefault properties. Restart the patched FE and verify that its configuration is also preserved.
Test results: Update the checkboxes and include actual execution results before submitting or marking validation complete.
Behavior changed:
Routine Load jobs now recover their explicitly configured execution properties instead of silently reverting to defaults.
For existing jobs previously affected by this issue, the effective execution settings may change on the first FE restart with the fix, restoring the user-configured values. This is the intended correctness improvement.
Does this need documentation?
No new SQL syntax, configuration options, or user-facing APIs are introduced. This change restores the expected behavior of existing Routine Load properties.
Check List (For Reviewer who merge this PR)