semantic build fails against VLM-backed embedders: data:image/...;base64 in chunk text is parsed as an image input #817

Summary

With semantic.backend = "openai_compatible" pointed at an NVIDIA NIM embeddings deployment, any
repository whose source text contains the literal prefix data:image/<mime>;base64 fails its entire
semantic build:

failed to build semantic index: openai compatible request failed (HTTP 400 Bad Request):
{"object": "error", "message": "image inputs require VLM serving to be enabled on this server",
 "type": "invalid_request_error"}

One offending row rejects the whole batch, so the build dies at the same chunk every session and
restarts from chunk 0 the next time. In the affected installation the warning had been firing every
session for 48 hours: ~8,500 chunks (134 of 326 batches, ~220 s) re-embedded per attempt, then the
same 400.

Environment

• AFT 0.62.0 (@cortexkit/aft-opencode@0.62.0), OpenCode on Linux x86_64.
• Config: semantic = { backend: "openai_compatible", model: "nvidia/nemotron-3-embed-1b", base_url: "https://integrate.api.nvidia.com/v1", api_key_env: "NVIDIA_API_KEY" }.
• Failing roots: a ComfyUI checkout and a TypeScript/Node web app.

Reproduction without AFT

curl -s https://integrate.api.nvidia.com/v1/embeddings \
  -H "Authorization: Bearer $NVIDIA_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"nvidia/nemotron-3-embed-1b","input":["data:image/png;base64,"]}'
# HTTP 400 image inputs require VLM serving to be enabled on this server

The same string without the prefix, or with a non-image mime type, returns HTTP 200.

Reproduction with AFT

A one-file repository containing ordinary image-encoding code:

def to_data_uri(payload: bytes) -> str:
    return f"data:image/png;base64,{payload.decode('ascii')}"

plus a user config with a standing root for it and the NVIDIA semantic backend, then:

XDG_CONFIG_HOME=<tmp>/config AFT_STORAGE_DIR=<tmp>/storage aft index

snapshot entry="/tmp/.../repo" kind=search  status=success result_count=1
snapshot entry="/tmp/.../repo" kind=semantic status=gap usable=false
  reason="openai compatible request failed (HTTP 400 Bad Request): {... \"message\": \"image inputs
  require VLM serving to be enabled on this server\" ...}"
exit=2

Root cause

SemanticEmbeddingModel::embed_texts sends chunk text verbatim as the embeddings input:

crates/aft/src/semantic_index.rs (pre-patch, line 3558):

let body = serde_json::json!({
    "input": texts,
    "model": model,
});

NVIDIA's current embedding deployments are VLM-backed. The deployment scans each input string for
the data-URI image grammar and, on a hit, treats that row as an image part; with VLM serving disabled
the request is rejected and every row in the batch is lost. The other live NVIDIA embedding models
behave the same (nemotron-3.5-embed-1b/8b additionally require input_type), and the older
text-only ones are EOL:

So there is no text-only NVIDIA embedding model to point at instead.

Measured trigger boundaries on nemotron-3-embed-1b (each row sent as input: [row]):

The check is a case-sensitive data:image/ prefix match with no payload validation, so a string
literal in code is enough. Real triggers that were hit in practice:

• ComfyUI comfy_api_nodes/nodes_openai.py:945, nodes_grok.py:444, nodes_qwen.py,
  nodes_recraft.py, nodes_wan.py, comfy_api_nodes/apis/bytedance_llm.py,
  comfy_api_nodes/util/conversions.py:224 (docstring).
• career-ops web/src/lib/apply/session.ts:198, web/src/lib/apply/drive.ts:147,
  hired-wall-build.mjs:132, tests/cv-visual/fixtures.mjs:3, tests/*.test.mjs.

This is not exotic content: any codebase that builds screenshot/avatar/thumbnail data URIs contains
it, and it is exactly the code a semantic index wants to cover.

Proposed fix

Neutralize the image data-URI prefix in the OpenAI-compatible request body only, at the wire
boundary:

const IMAGE_DATA_URI_PREFIX: &str = "data:image/";
const NEUTRALIZED_IMAGE_DATA_URI_PREFIX: &str = "data-uri:image/";

fn neutralize_image_data_uris(text: &str) -> Cow<'_, str> {
    if !text.contains(IMAGE_DATA_URI_PREFIX) {
        return Cow::Borrowed(text);
    }
    Cow::Owned(text.replace(IMAGE_DATA_URI_PREFIX, NEUTRALIZED_IMAGE_DATA_URI_PREFIX))
}

let sanitized_texts: Vec<Cow<'_, str>> = texts
    .iter()
    .map(|text| neutralize_image_data_uris(text))
    .collect();
let body = serde_json::json!({ "input": sanitized_texts, "model": model });

Notes on the design:

• Literal prefix rewrite, not a regex. The backend's trigger is the literal, case-sensitive
  data:image/ prefix and it does not require a valid payload, so nothing needs parsing. One
  contains scan per row keeps the common case allocation-free (Cow::Borrowed); a pattern
  rewrite would allocate for every row and add a regex to the hot path for no gain.
• Wire boundary, not chunk creation. Chunk text is user-visible in search snippets and
  provenance, and local fastembed/ollama backends do not interpret data URIs. Rewriting only the
  remote OpenAI-compatible payload leaves the persisted index byte-identical and keeps the behavior
  scoped to the backend that needs it. embed_texts is the single request builder, so build batches
  and interactive query embeds are both covered.
• Only data:image/. The table above shows the other mime types and the uppercase form already
  pass, so rewriting them would only perturb embeddings without fixing anything.

Test plan

• Unit tests for the helper: borrowed pass-through, template literal, multiple occurrences, docstring
  mention without payload, non-image mime types untouched, uppercase untouched, and a serialized
  body assertion that no data:image/ survives.
• End-to-end with the one-file fixture above and the NVIDIA backend: the stock 0.62.0 binary
  reproduces the failure (verified: kind=semantic status=gap usable=false, exit 2); the patched
  binary reports kind=semantic status=success, exit 0.

Workarounds available today

None that keep the backend: per-file .aftignore also removes the path from trigram search and the
call graph, the local fastembed backend rebuilds every index on CPU, and a request-rewriting proxy
adds a supervised process to the embedding path.