Repository navigation
Conversation
…action stream load Expose the multi-table transaction stream load capability of StarRocks Flink connector 1.2.16 behind an opt-in option: - Add 'sink.transaction.multi-table.enabled' (default false) and 'sink.transaction.multi-table.buffer-size' options, passed through to the StarRocks connector - Fail fast at pipeline submission when the option is enabled but the StarRocks server version is below 4.0 or unreachable - Fail fast when schema change events target more than one database, since a shared StarRocks transaction is single-database scoped - Mark every serialized row with a fixed source partition and a transaction-end flag when enabled, so tables written by the same sink subtask within one flush cycle share one transaction label and become visible atomically, while keeping the at-least-once flush cadence - Add StarRocksMultiTableTransactionITCase against a StarRocks 4.0.16 container, a version-check fail-fast case on the 3.5.10 container, and unit tests for the new logic; update EN/ZH docs Depends on apache#4578. Generated-by: CodeBuddy (GLM-5.3)
sd4324530
force-pushed
the
starrocks-multi-table-transaction
branch
from
October 27, 2026 14:16
cda5575 to
e829c2c
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of this pull request?
Exposes the multi-table transaction stream load capability introduced by
the StarRocks Flink connector 1.2.16 (upstream PR #487, hardened in
1.2.16) behind an opt-in option for the StarRocks pipeline connector.
When enabled, all tables written by the same sink subtask within the
same flush cycle share one StarRocks transaction label, so they become
visible atomically, and the number of StarRocks transactions/labels per
flush cycle drops from one-per-table to one-per-subtask, which reduces
FE pressure for wide-table-count sync jobs.
Depends on #4578 (StarRocks connector upgrade to 1.2.16), please merge
that first.
Brief change log
sink.transaction.multi-table.enabled(defaultfalse) andsink.transaction.multi-table.buffer-sizeoptions, passed through tothe StarRocks connector. With the option disabled, behavior is
unchanged.
StarRocks server version is below 4.0 or unreachable (JDBC
SELECT CURRENT_VERSION()check)since a shared StarRocks transaction is single-database scoped
and a transaction-end flag, so tables written by the same sink
subtask within one flush cycle share one transaction label while
keeping the current at-least-once flush cadence
StarRocksMultiTableTransactionITCaseagainst StarRocks 4.0.16 and aversion-check fail-fast case on the existing 3.5.10 container
Verifying this change
This change added tests and can be verified as follows:
StarRocksDataSinkFactoryTest(version parsing,fail-fast on unreachable server),
EventRecordSerializationSchemaTest(multi-table row marks, default-marks regression) and
StarRocksMetadataApplierTest(cross-database rejection)StarRocksMultiTableTransactionITCase— end-to-end multi-tableatomic write against a StarRocks 4.0.16 container
StarRocksPipelineITCase#testMultiTableTransactionFailsFastOnUnsupportedServerVersion— version check fails fast on StarRocks 3.5.10
(
StarRocksPipelineITCase,StarRocksMetadataApplierITCase) stillpass on StarRocks 3.5.10 with the option disabled
Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: CodeBuddy (GLM-5.3)