Repository navigation
fix(models): send gs:// file parts to Claude as inline base64 - #7710
danieldutu wants to merge 1 commit into
Conversation
Somay-kousis
left a comment
There was a problem hiding this comment.
What I ran (head 869d88d, merge base 25d1917):
pytest tests/unittests/models/test_anthropic_llm.pyon head: 190 passed.- Same file with
src/reverted to the merge base,-k gcs: 3 failed, 1 passed. Small note: the one that passes on main istest_part_to_message_block_rejects_unsupported_gcs_file_type, not the missing-dependency test the description names. - A probe that runs
AnthropicLlm.generate_content_asyncend to end with a mockedgoogle.cloud.storage.Client(download takes 0.5 s viatime.sleep) and a mocked Anthropic client. No network.
Findings
-
The GCS download blocks the event loop (blocking).
_download_gcs_uri(anthropic_llm.py:97 and :107) makes synchronousstorage.Client()/blob.download_as_bytes()calls. It runs from_part_to_message_block(:725), whichgenerate_content_asynccalls directly at :1320. Nothing moves it off the loop. In the probe, a 10 ms heartbeat task stalled for 0.538 s during one call. Every other coroutine on the loop (other sessions on an API server, streaming, live) waits out the whole download, up to 32 MB. The same file already puts blocking client construction inasyncio.to_thread(:1657). Suggest resolving file_data parts before the sync conversion, for example by collecting the gs:// URIs and downloading them withawait asyncio.to_thread(...)insidegenerate_content_async. -
The object is downloaded again on every model call (worth considering). The file_data part stays in
llm_request.contents, so each later turn converts it again. Probe: 3 turns of one conversation, 3 downloads of the same gs:// object. Combined with 1, a long chat over a large PDF pays the full blocking download on every turn. A per-request or per-URI cache (keyed on URI plus generation) would avoid it. -
Unsupported image subtypes are downloaded before they are rejected (worth considering).
is_image(:722) accepts anyimage/*, but the media type is only checked by_normalize_image_media_typeat :739, after the download at :725. Probe:gs://b/scan.tiffwithimage/tiffdownloads once, then raisesValueError: Unsupported Anthropic image MIME type: image/tiff. Calling_normalize_image_media_typebefore_download_gcs_urifails fast with the same error. -
gs://bucket-onlyraises an unclear error (nit). :96 unpackssplit("/", 1)into two names, so a URI with no object path raisesValueError: not enough values to unpack (expected 2, got 1)(probe output). A check that names the URI would be clearer. -
The 32 MB cap is checked on raw bytes (suspicion, not run against the API). :82 and :107 compare the raw size with 32 MB, but the request carries the base64 text, which is about 4/3 larger. If Claude's 32 MB limit applies to the request body, as I understand Anthropic's docs to say, a 25 to 32 MB file passes this check and is still rejected by the API. A raw-byte cap near 24 MB would match. I did not confirm this with a live request.
What I did not check: a live Vertex or Anthropic call, real GCS credentials and project resolution for storage.Client() with no explicit project, and the LiteLLM path the description mentions.
Probe and raw output: probes/probe_7710.py, review_7710_evidence.txt.
A file_data part with a gs:// URI raised NotImplementedError in _part_to_message_block (anthropic_llm.py:751), so Claude on Vertex AI could not ingest a GCS file while Gemini handled the same part. Claude rejects URL sources upstream, so the object is now downloaded from GCS and sent inline as a base64 document or image block. generate_content_async downloads the distinct gs:// objects of a request with asyncio.to_thread before the synchronous conversion (anthropic_llm.py:141, :1354). A 0.3 s download no longer stalls the event loop. An image subtype Claude does not accept (image/tiff) is rejected before any download, and a gs:// URI without an object path raises a ValueError that names the URI. google.cloud.storage is imported lazily. The size cap is 24 MB of raw bytes, so the base64 body stays under Anthropic's 32 MB request limit; it is checked against the blob size before downloading. Fixes google#2466
869d88d to
a25cc3f
Compare
|
Thanks for the probes and the raw output.
The test description note is corrected: the one test that passes on Not covered: a live Vertex or Anthropic call, real GCS credentials and project resolution for |
(Update the PR description with these changed parts; the rest stays.)
Solution:
As surajksharma07 noted on the issue, Claude's API does not support URL sources upstream, so the fix downloads the GCS object and sends it inline.
_part_to_message_blockgained afile_databranch that, for ags://bucket/objectURI withapplication/pdfor a supported image MIME type, returns the same base64DocumentBlockParamorImageBlockParamblock the inline branches produce.generate_content_asyncdownloads the distinct gs:// objects of a request withasyncio.to_threadbefore the synchronous conversion, so the download does not run on the event loop.google.cloud.storageis imported lazily inside the function (thegcs_artifact_servicepattern) and a missing install raises anImportErrorthat namespip install google-adk[gcp]. The cap is 24 MB of raw bytes: base64 grows the body by 4/3, so 24 MB raw stays under Anthropic's 32 MB request limit. The cap is checked against the blob size before downloading, with a post-download check for clients that do not expose the size. Non-gs://URIs and unsupported MIME types keep the existingNotImplementedError. An unsupported image subtype (image/tiff) is rejected before any download.Testing Plan:
Tests in
tests/unittests/models/test_anthropic_llm.py:test_part_to_message_block_with_gcs_pdf_document: gs:// PDF downloads and produces the base64 document block.test_part_to_message_block_rejects_oversized_gcs_file: a 25 MB blob (under 32 MB raw, over 32 MB as base64) raisesValueErrornaming the URI, size and cap.test_part_to_message_block_rejects_unsupported_gcs_file_type:video/mp4raisesNotImplementedError. This is the only one of the first four that passes onmain.test_part_to_message_block_missing_gcs_dependency: withoutgoogle-cloud-storage, the error namesgoogle-adk[gcp].test_generate_content_async_gcs_download_does_not_block_loop: a 0.3 s mocked download runs while a 10 ms heartbeat task runs; the largest heartbeat gap stays under 0.15 s and the request carries the base64 payload.test_part_to_message_block_rejects_unsupported_gcs_image_before_download:image/tiffraisesValueErrorand the storage client is never used.test_part_to_message_block_rejects_gcs_uri_without_object:gs://bucket-onlyraises aValueErrorthat names the URI.Against
mainand the pre-review branch,-k gcsgives4 failed, 3 passed; with the fix,7 passed.uv run pytest tests/unittests/models/test_anthropic_llm.py->193 passed in 1.72s.uv run pytest tests/unittests/models->1774 passed, 31 warnings in 16.18s.uv run pytest tests/unittests/test_import_loading.py->29 passed.uv run pre-commit run --files src/google/adk/models/anthropic_llm.py tests/unittests/models/test_anthropic_llm.py-> all hooks passed.Manual End-to-End (E2E) Tests: unchanged: no live Vertex AI call was made; needs a project with Claude on Vertex AI enabled.
Additional context:
storage.Client()is built per download with application default credentials. A gs:// object that appears in several turns is downloaded once per model call (see the reply, finding 2). Not covered: non-gs://file_dataURIs and http(s) URL sources, which keep the currentNotImplementedError.