Skip to content

Core: Bound Idempotency-Key retries to the advertised key lifetime - #18070

Merged
huaxingao merged 1 commit into
apache:mainfrom
HotSushi:idempotency-key-expiry
Sep 29, 2026
Merged

huaxingao merged 1 commit into
apache:mainfrom
HotSushi:idempotency-key-expiry

Conversation

@HotSushi

Copy link
Copy Markdown
Contributor

What / Why

Follow-up to #17947 (client-side Idempotency-Key retries). #17947 auto-retries keyed POSTs on retriable errors but never honors the server-advertised key lifetime, so a slow retry can fire after the key has expired — when the server no longer dedupes — risking a duplicate mutation.

The REST spec obligation on the client:

Clients SHOULD NOT reuse an Idempotency-Key after this window elapses.

Change

  • ExponentialHttpRequestRetryStrategy now accepts the advertised key lifetime and declines a retry for a keyed request once the elapsed time since the first attempt plus the next retry interval would reach the lifetime window. Enforced on both the response and network-exception retry paths; non-keyed / idempotent requests are unaffected.
  • The lifetime advertised in GET /v1/config (ISO-8601, e.g. PT30M) is plumbed to the HTTP client through the client properties and parsed once in HTTPClient; a malformed value is ignored (disables enforcement) rather than failing client construction.
  • Extracted a shared isRetrySafe(...) predicate to remove duplication between the two retry paths (addresses a review nit; also clarifies that "retry-safe" is not the same as strictly idempotent).

Tests

Added unit tests in TestExponentialHttpRequestRetryStrategy: keyed retry abandoned once the window is exceeded (both response and network-exception paths), keyed retry honored within the window and when no lifetime is advertised, and an idempotent GET unaffected by the key lifetime. :iceberg-core rest.* tests pass locally.

Notes

@github-actions github-actions Bot added the core label Sep 11, 2026
@HotSushi
HotSushi force-pushed the idempotency-key-expiry branch 2 times, most recently from e29c318 to ed0d254 Compare September 17, 2026 17:16
@HotSushi
HotSushi marked this pull request as ready for review September 17, 2026 17:29
@HotSushi
HotSushi force-pushed the idempotency-key-expiry branch 5 times, most recently from 284acf2 to 602595b Compare September 18, 2026 21:25
@HotSushi
HotSushi force-pushed the idempotency-key-expiry branch from 602595b to 66d4eab Compare September 21, 2026 23:29
@HotSushi
HotSushi force-pushed the idempotency-key-expiry branch from 66d4eab to 3461331 Compare September 22, 2026 17:15

@huaxingao huaxingao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@huaxingao

Copy link
Copy Markdown
Contributor

cc @flyrain

Auto-retrying a keyed mutation could previously fire after the server's advertised Idempotency-Key lifetime elapsed, at which point the server no longer dedupes the key and could apply the mutation twice. This records when a request is first sent (in HTTPClient.execute) and, for requests carrying an Idempotency-Key, declines a retry once the elapsed time since the first attempt plus the next retry interval would reach the advertised lifetime. The lifetime from GET /v1/config is plumbed through the client properties into ExponentialHttpRequestRetryStrategy; a non-positive window declines all keyed retries, while an absent or unparseable one leaves the bound disabled. The check applies to the response and network-exception retry paths; non-keyed and idempotent requests are unaffected.

Per Yufei's review, the check compares against the upper bound of the next retry interval (Retry-After verbatim, or exponential backoff plus its maximum jitter) rather than a freshly sampled value, so a retry the check permits cannot then sleep longer than checked and overshoot the window.
@HotSushi
HotSushi force-pushed the idempotency-key-expiry branch from 3461331 to 0509fd2 Compare September 25, 2026 21:12
@huaxingao
huaxingao merged commit 7d6d369 into apache:main Sep 29, 2026
42 checks passed
@huaxingao

Copy link
Copy Markdown
Contributor

Thanks @HotSushi for the PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants