Skip to content

[FLINK-41207][pipeline-connector/fluss] Preserve consumed offsets across empty log batches - #4602

Open
fxbing wants to merge 1 commit into
apache:masterfrom
fxbing:reproduce-fluss-reader-empty-log-checkpoint-offset
Open

fxbing wants to merge 1 commit into
apache:masterfrom
fxbing:reproduce-fluss-reader-empty-log-checkpoint-offset

Conversation

@fxbing

@fxbing fxbing commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of this pull request?

Fixes FLINK-41207.

The Fluss scanner can advance across empty log intervals without returning records. The pipeline source currently discards this progress, leaving the checkpointed split offset behind and rescanning the same interval after recovery.

Brief change log

  • Queue progress-only elements after each bucket's records when consumed offsets are not represented by those records.
  • Apply progress on the reader thread for log and hybrid splits without emitting data, consuming pending schema events, or inflating input metrics.
  • Preserve the existing split serialization format.

Verifying this change

  • Added a real Fluss cluster regression using ProduceLog RPC. With the old reader, the checkpoint remains at offset 1 instead of 2; with the fix, checkpoint serialization and restoration preserve offset 2.
  • Added coverage for empty and mixed polls, trailing gaps, snapshot ordering, pending schema events, inactive tables, and input metrics.
  • All 33 selected tests passed on Java 11 / Flink 1.20.3 and Java 17 / Flink 2.2.0. Spotless and Checkstyle passed.

Full job failover E2E and performance tests were not run.

Documentation

  • Does this pull request introduce a new feature? No.
  • If yes, how is the feature documented? Not applicable.

Was generative AI tooling used to co-author this PR?
  • Yes — OpenAI Codex.

Generated-by: Codex CLI 0.164.0

…oss empty log batches

Queue consumed-offset markers after records and apply them on the reader thread without emitting data or changing schema state.

Keep input metrics accurate and cover checkpoint serialization and recovery through a real ProduceLog RPC on both Flink versions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant