Skip to content

[FLINK-40966][pipeline-connector/fluss] Reuse metadata clients across schema operations - #4570

Open
fxbing wants to merge 3 commits into
apache:masterfrom
fxbing:fix/20261009-cdc-metadata-connection
Open

fxbing wants to merge 3 commits into
apache:masterfrom
fxbing:fix/20261009-cdc-metadata-connection

Conversation

@fxbing

@fxbing fxbing commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

Fluss schema operations create and close a connection for each lookup or DDL. Waiting for the Netty event loop to shut down adds repeated delays to multi-table initialization. This PR reuses the metadata clients for the applier lifecycle.

JIRA: https://issues.apache.org/jira/browse/FLINK-40966

Brief change log

  • Lazily create transient Connection/Admin instances and reuse them for schema lookups and DDL; release both on close.
  • Stop coordinator events and schema workers before closing the metadata applier.
  • Add real-client tests for reuse, serialization, instance isolation and cleanup after schema errors; cover coordinator shutdown ordering and deferred cleanup after a worker timeout.

Verifying this change

  • Verified the new client lifecycle and coordinator shutdown regressions fail against the original implementation and pass with the fix.
  • Java 11 / Flink 1.20.3: 100 focused runtime schema tests and 21 Fluss tests passed. A clean reactor run also passed all 19 coordinator tests and the 21 Fluss tests.
  • Java 17 / Flink 2.2.0: 84 focused runtime tests and the same 21 Fluss tests passed (-Pflink2).
  • The new lifecycle ITCase uses a real Fluss cluster and client threads; no dependencies were added.

Focused command (Java 11):

mvn -o -pl flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss -am \
  -Dtest=FlussMetadataApplierTest,FlussMetadataApplierLifecycleITCase,FlussEventSerializationSchemaTest,FlussSinkITCase#testWrongTableOptions,SchemaCoordinatorTest \
  -Dsurefire.failIfNoSpecifiedTests=false clean test

For Java 17 / Flink 2.2, use -Pflink2, add ExistingTableSchemaExpanderTest to the test selection, and run package -DskipITs after cleaning. Packaging is required by the profile's reactor JAR-copy step. The selected lifecycle ITCase is explicitly executed by Surefire.

Documentation

  • Does this pull request introduce a new feature? No.
  • If yes, how is the feature documented? Not applicable.
Was generative AI tooling used to co-author this PR?
  • Yes — OpenAI Codex

Generated-by: OpenAI Codex

… schema operations

Reuse lazy transient clients per metadata applier and release them on close.
Stop schema workers before releasing metadata resources, deferring cleanup on timeout.
Cover client ownership and coordinator shutdown with lifecycle regressions.

Generated-by: OpenAI Codex
@loserwang1024

Copy link
Copy Markdown
Contributor

still need apache/fluss#4200

@loserwang1024
loserwang1024 self-requested a review October 10, 2026 03:30
Merge upstream/master and retain its explicit bucket-key validation alongside
metadata client reuse and lifecycle cleanup.

Generated-by: OpenAI Codex
…data conflicts

Merge upstream/master while retaining metadata client reuse and lifecycle cleanup.
Preserve upstream ADD COLUMN replay handling and schema table options.

Generated-by: OpenAI Codex
@loserwang1024

Copy link
Copy Markdown
Contributor

@fxbing ,apache/fluss#4200 is merged, pushing forward it now.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants