Skip to content

[FLINK-41008][pipeline-connector/starrocks] Support multi-table transaction stream load - #4579

Open
sd4324530 wants to merge 1 commit into
apache:masterfrom
sd4324530:starrocks-multi-table-transaction
Open

sd4324530 wants to merge 1 commit into
apache:masterfrom
sd4324530:starrocks-multi-table-transaction

Conversation

@sd4324530

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

Exposes the multi-table transaction stream load capability introduced by
the StarRocks Flink connector 1.2.16 (upstream PR #487, hardened in
1.2.16) behind an opt-in option for the StarRocks pipeline connector.

When enabled, all tables written by the same sink subtask within the
same flush cycle share one StarRocks transaction label, so they become
visible atomically, and the number of StarRocks transactions/labels per
flush cycle drops from one-per-table to one-per-subtask, which reduces
FE pressure for wide-table-count sync jobs.

Depends on #4578 (StarRocks connector upgrade to 1.2.16), please merge
that first.

Brief change log

  • Add sink.transaction.multi-table.enabled (default false) and
    sink.transaction.multi-table.buffer-size options, passed through to
    the StarRocks connector. With the option disabled, behavior is
    unchanged.
  • Fail fast at pipeline submission when the option is enabled but the
    StarRocks server version is below 4.0 or unreachable (JDBC
    SELECT CURRENT_VERSION() check)
  • Fail fast when schema change events target more than one database,
    since a shared StarRocks transaction is single-database scoped
  • When enabled, mark every serialized row with a fixed source partition
    and a transaction-end flag, so tables written by the same sink
    subtask within one flush cycle share one transaction label while
    keeping the current at-least-once flush cadence
  • Parameterize the test StarRocks container version; add
    StarRocksMultiTableTransactionITCase against StarRocks 4.0.16 and a
    version-check fail-fast case on the existing 3.5.10 container
  • Update EN/ZH connector docs with the new options and limitations

Verifying this change

This change added tests and can be verified as follows:

  • Added unit tests in StarRocksDataSinkFactoryTest (version parsing,
    fail-fast on unreachable server), EventRecordSerializationSchemaTest
    (multi-table row marks, default-marks regression) and
    StarRocksMetadataApplierTest (cross-database rejection)
  • Added integration tests:
    • StarRocksMultiTableTransactionITCase — end-to-end multi-table
      atomic write against a StarRocks 4.0.16 container
    • StarRocksPipelineITCase#testMultiTableTransactionFailsFastOnUnsupportedServerVersion
      — version check fails fast on StarRocks 3.5.10
  • Verified all existing unit tests (64) and IT cases
    (StarRocksPipelineITCase, StarRocksMetadataApplierITCase) still
    pass on StarRocks 3.5.10 with the option disabled

Documentation

  • Does this pull request introduce a new feature? (yes / no): yes
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented): docs

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: CodeBuddy (GLM-5.3)

@github-actions github-actions Bot added docs Improvements or additions to documentation e2e-tests starrocks-pipeline-connector labels Oct 15, 2026
…action stream load

Expose the multi-table transaction stream load capability of StarRocks
Flink connector 1.2.16 behind an opt-in option:

- Add 'sink.transaction.multi-table.enabled' (default false) and
  'sink.transaction.multi-table.buffer-size' options, passed through to
  the StarRocks connector
- Fail fast at pipeline submission when the option is enabled but the
  StarRocks server version is below 4.0 or unreachable
- Fail fast when schema change events target more than one database,
  since a shared StarRocks transaction is single-database scoped
- Mark every serialized row with a fixed source partition and a
  transaction-end flag when enabled, so tables written by the same sink
  subtask within one flush cycle share one transaction label and become
  visible atomically, while keeping the at-least-once flush cadence
- Add StarRocksMultiTableTransactionITCase against a StarRocks 4.0.16
  container, a version-check fail-fast case on the 3.5.10 container, and
  unit tests for the new logic; update EN/ZH docs

Depends on apache#4578.

Generated-by: CodeBuddy (GLM-5.3)
@sd4324530
sd4324530 force-pushed the starrocks-multi-table-transaction branch from cda5575 to e829c2c Compare October 27, 2026 14:16

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation e2e-tests starrocks-pipeline-connector

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant