Skip to content

fix(models): send gs:// file parts to Claude as inline base64 - #7710

Open
danieldutu wants to merge 1 commit into
google:mainfrom
danieldutu:fix/models-claude-file-parts
Open

danieldutu wants to merge 1 commit into
google:mainfrom
danieldutu:fix/models-claude-file-parts

Conversation

@danieldutu

@danieldutu danieldutu commented Oct 24, 2026 •

Copy link
Copy Markdown

(Update the PR description with these changed parts; the rest stays.)

Solution:

As surajksharma07 noted on the issue, Claude's API does not support URL sources upstream, so the fix downloads the GCS object and sends it inline. _part_to_message_block gained a file_data branch that, for a gs://bucket/object URI with application/pdf or a supported image MIME type, returns the same base64 DocumentBlockParam or ImageBlockParam block the inline branches produce. generate_content_async downloads the distinct gs:// objects of a request with asyncio.to_thread before the synchronous conversion, so the download does not run on the event loop. google.cloud.storage is imported lazily inside the function (the gcs_artifact_service pattern) and a missing install raises an ImportError that names pip install google-adk[gcp]. The cap is 24 MB of raw bytes: base64 grows the body by 4/3, so 24 MB raw stays under Anthropic's 32 MB request limit. The cap is checked against the blob size before downloading, with a post-download check for clients that do not expose the size. Non-gs:// URIs and unsupported MIME types keep the existing NotImplementedError. An unsupported image subtype (image/tiff) is rejected before any download.

Testing Plan:

Tests in tests/unittests/models/test_anthropic_llm.py:

  • test_part_to_message_block_with_gcs_pdf_document: gs:// PDF downloads and produces the base64 document block.
  • test_part_to_message_block_rejects_oversized_gcs_file: a 25 MB blob (under 32 MB raw, over 32 MB as base64) raises ValueError naming the URI, size and cap.
  • test_part_to_message_block_rejects_unsupported_gcs_file_type: video/mp4 raises NotImplementedError. This is the only one of the first four that passes on main.
  • test_part_to_message_block_missing_gcs_dependency: without google-cloud-storage, the error names google-adk[gcp].
  • test_generate_content_async_gcs_download_does_not_block_loop: a 0.3 s mocked download runs while a 10 ms heartbeat task runs; the largest heartbeat gap stays under 0.15 s and the request carries the base64 payload.
  • test_part_to_message_block_rejects_unsupported_gcs_image_before_download: image/tiff raises ValueError and the storage client is never used.
  • test_part_to_message_block_rejects_gcs_uri_without_object: gs://bucket-only raises a ValueError that names the URI.

Against main and the pre-review branch, -k gcs gives 4 failed, 3 passed; with the fix, 7 passed.
uv run pytest tests/unittests/models/test_anthropic_llm.py -> 193 passed in 1.72s.
uv run pytest tests/unittests/models -> 1774 passed, 31 warnings in 16.18s.
uv run pytest tests/unittests/test_import_loading.py -> 29 passed.
uv run pre-commit run --files src/google/adk/models/anthropic_llm.py tests/unittests/models/test_anthropic_llm.py -> all hooks passed.

Manual End-to-End (E2E) Tests: unchanged: no live Vertex AI call was made; needs a project with Claude on Vertex AI enabled.

Additional context: storage.Client() is built per download with application default credentials. A gs:// object that appears in several turns is downloaded once per model call (see the reply, finding 2). Not covered: non-gs:// file_data URIs and http(s) URL sources, which keep the current NotImplementedError.

@Somay-kousis Somay-kousis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I ran (head 869d88d, merge base 25d1917):

  • pytest tests/unittests/models/test_anthropic_llm.py on head: 190 passed.
  • Same file with src/ reverted to the merge base, -k gcs: 3 failed, 1 passed. Small note: the one that passes on main is test_part_to_message_block_rejects_unsupported_gcs_file_type, not the missing-dependency test the description names.
  • A probe that runs AnthropicLlm.generate_content_async end to end with a mocked google.cloud.storage.Client (download takes 0.5 s via time.sleep) and a mocked Anthropic client. No network.

Findings

  1. The GCS download blocks the event loop (blocking). _download_gcs_uri (anthropic_llm.py:97 and :107) makes synchronous storage.Client() / blob.download_as_bytes() calls. It runs from _part_to_message_block (:725), which generate_content_async calls directly at :1320. Nothing moves it off the loop. In the probe, a 10 ms heartbeat task stalled for 0.538 s during one call. Every other coroutine on the loop (other sessions on an API server, streaming, live) waits out the whole download, up to 32 MB. The same file already puts blocking client construction in asyncio.to_thread (:1657). Suggest resolving file_data parts before the sync conversion, for example by collecting the gs:// URIs and downloading them with await asyncio.to_thread(...) inside generate_content_async.

  2. The object is downloaded again on every model call (worth considering). The file_data part stays in llm_request.contents, so each later turn converts it again. Probe: 3 turns of one conversation, 3 downloads of the same gs:// object. Combined with 1, a long chat over a large PDF pays the full blocking download on every turn. A per-request or per-URI cache (keyed on URI plus generation) would avoid it.

  3. Unsupported image subtypes are downloaded before they are rejected (worth considering). is_image (:722) accepts any image/*, but the media type is only checked by _normalize_image_media_type at :739, after the download at :725. Probe: gs://b/scan.tiff with image/tiff downloads once, then raises ValueError: Unsupported Anthropic image MIME type: image/tiff. Calling _normalize_image_media_type before _download_gcs_uri fails fast with the same error.

  4. gs://bucket-only raises an unclear error (nit). :96 unpacks split("/", 1) into two names, so a URI with no object path raises ValueError: not enough values to unpack (expected 2, got 1) (probe output). A check that names the URI would be clearer.

  5. The 32 MB cap is checked on raw bytes (suspicion, not run against the API). :82 and :107 compare the raw size with 32 MB, but the request carries the base64 text, which is about 4/3 larger. If Claude's 32 MB limit applies to the request body, as I understand Anthropic's docs to say, a 25 to 32 MB file passes this check and is still rejected by the API. A raw-byte cap near 24 MB would match. I did not confirm this with a live request.

What I did not check: a live Vertex or Anthropic call, real GCS credentials and project resolution for storage.Client() with no explicit project, and the LiteLLM path the description mentions.

Probe and raw output: probes/probe_7710.py, review_7710_evidence.txt.

A file_data part with a gs:// URI raised NotImplementedError in
_part_to_message_block (anthropic_llm.py:751), so Claude on Vertex AI
could not ingest a GCS file while Gemini handled the same part. Claude
rejects URL sources upstream, so the object is now downloaded from GCS
and sent inline as a base64 document or image block.

generate_content_async downloads the distinct gs:// objects of a request
with asyncio.to_thread before the synchronous conversion
(anthropic_llm.py:141, :1354). A 0.3 s download no longer stalls the
event loop. An image subtype Claude does not accept (image/tiff) is
rejected before any download, and a gs:// URI without an object path
raises a ValueError that names the URI. google.cloud.storage is imported
lazily. The size cap is 24 MB of raw bytes, so the base64 body stays
under Anthropic's 32 MB request limit; it is checked against the blob
size before downloading.

Fixes google#2466
@danieldutu
danieldutu force-pushed the fix/models-claude-file-parts branch from 869d88d to a25cc3f Compare October 24, 2026 16:00
@danieldutu

Copy link
Copy Markdown
Author

Thanks for the probes and the raw output.

  1. Download off the loop. _download_gcs_parts (anthropic_llm.py:141) runs _download_gcs_uri through asyncio.to_thread, one task per distinct URI, and generate_content_async (:1354) awaits it before the synchronous conversion. Pinned by test_generate_content_async_gcs_download_does_not_block_loop: a 0.3 s mocked download, 10 ms heartbeat, largest gap under 0.15 s. Before the change the heartbeat never ran during the call.
  2. Repeated download across turns. Not changed. Within one request each distinct URI downloads once. A cache across calls needs a key of URI plus generation and an owner object with a lifetime (the conversion functions are stateless), which is more than 20 lines; I propose it as a follow-up.
  3. image/tiff check order. _gcs_file_kind (:124) calls _normalize_image_media_type before the download. Pinned by test_part_to_message_block_rejects_unsupported_gcs_image_before_download, which asserts the storage client is never used.
  4. gs://bucket-only. _download_gcs_uri (:97-100) raises ValueError: gs:// URI must name a bucket and an object: gs://bucket-only. Pinned by test_part_to_message_block_rejects_gcs_uri_without_object.
  5. Cap on raw bytes. _MAX_FILE_DATA_SIZE (:83) is 24 * 1024 * 1024 and the docstring states the 4/3 reason. test_part_to_message_block_rejects_oversized_gcs_file now uses a 25 MB blob. The cap assumes the 32 MB limit applies to the request body; I did not confirm that against the API.

The test description note is corrected: the one test that passes on main is test_part_to_message_block_rejects_unsupported_gcs_file_type.

Not covered: a live Vertex or Anthropic call, real GCS credentials and project resolution for storage.Client(), and the LiteLLM path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants