Repository navigation
Conversation
rahulsmahadev
left a comment
There was a problem hiding this comment.
Thanks @linzch3! The retry check matches Java's wouldExceedKeyLifetime from apache/iceberg#18070, and the new tests of the bound fail on main and pass here. I left one suggestion inline about urllib3 1.x.
A smaller suggestion: no test checks that time already spent counts toward the lifetime. A version that compares only the next sleep with the lifetime still passes all of test_rest.py, so a case where earlier retries use up most of the lifetime would pin it.
| from typing_extensions import override | ||
| from urllib3.connectionpool import ConnectionPool | ||
| from urllib3.exceptions import MaxRetryError, ResponseError | ||
| from urllib3.response import BaseHTTPResponse |
There was a problem hiding this comment.
With urllib3 1.26, which requests>=2.20.0 still allows, import pyiceberg.catalog.rest now fails with ImportError: cannot import name 'BaseHTTPResponse', and 1.26's Retry has no backoff_jitter or backoff_max either. One option: import this under TYPE_CHECKING, and since pyiceberg never sets a jitter, have _max_sleep return self.get_backoff_time(); with both changes the new tests pass on urllib3 1.26.20 and 2.8.0. Requiring urllib3>=2 in pyproject.toml would also work.
There was a problem hiding this comment.
thx for checking 1.26! moved the BaseHTTPResponse import under TYPE_CHECKING and _max_sleep now just uses get_backoff_time() since we never set jitter. also added a test where earlier retries already eat most of the lifetime.
… lifetime Import BaseHTTPResponse only for type checking and take the next backoff from get_backoff_time(), since no jitter is configured. Co-authored-by: Rahul Shivu Mahadev <51690557+rahulsmahadev@users.noreply.github.com>
Closes #4428
Rationale for this change
#4310 sends
Idempotency-Keyon mutations and letsrest.client.*retry keyedPOSTs, but it never looks at theidempotency-key-lifetimethe server returns from/v1/configbeyond checking it's there. urllib3 sleeps for the fullRetry-After(and backoff can add up too), so a retry can go out after the key has expired, and the server may then apply the same mutation twice. The spec says clients SHOULD NOT reuse a key after that window.This follows what the Java client does (
HTTPClient.parseKeyLifetime+ExponentialHttpRequestRetryStrategy, apache/iceberg#18070):idempotency-key-lifetimeis parsed as an ISO-8601 duration (TypeAdapter(timedelta))._RetryTimeoutHTTPAdapter.sendrecords a deadline (now + lifetime, monotonic clock) in aContextVarwhen the request carries anIdempotency-Key.Retrysubclass checks it inincrement: ifnow + next_sleep >= deadlinethe retry is declined.next_sleepis whatRetry.sleepwould wait:Retry-Afterwhen urllib3 would honor it, otherwise the backoff plus the max jitter.MaxRetryError, i.e. it behaves exactly like running out of retries: withraise_on_status=Falseurllib3 hands back the last response, so_handle_non_200_responsestill raises the typed error (e.g.ServiceUnavailableError); a connection error is re-raised as before.Requests without a key (all
GET/HEAD, and everything when the server doesn't advertise a lifetime) are untouched.Are these changes tested?
Yes, in
tests/catalog/test_rest.py, using the existing loopback server (now with an optionalRetry-Afterheader):Retry-After: 60withPT30Slifetime: no retry,ServiceUnavailableError, 1 callPT5S, factor 10): stops before the second retryRetry-After: 0withinPT30M: still retried and succeedsPT0Sand-PT5M: keyed POST never retried, key still sentPT0Sdoesn't affect unkeyed GETs, they still retry_parse_idempotency_key_lifetimeunit casesmake lintpasses.pytest tests/catalog -m unitpasses except 4 kerberos tests intest_hive.pythat fail the same way onmainhere (kerberos extra not installed locally).Are there any user-facing changes?
Only for catalogs whose server advertises
idempotency-key-lifetimeand that setrest.client.max-retries: a keyed mutation is no longer retried once the next attempt would land outside the key's lifetime; the error that would have been raised after exhausting retries is raised earlier instead.Limitations
table/updatecreates a new key per attempt, so that's fine).POSTon status codes, so it isn't changed here.#4310 cc @YuangGao. Happy to adjust if u'd prefer a different shape (e.g. doing this in the adapter instead of a
Retrysubclass). thx!