[as992949791]

Summary

Count decoded float32 elements when an OpenAI embedding response uses explicit encoding_format="base64". Preserve the previous string-length fallback for malformed base64 or decoded bytes that do not form complete float32 elements. SDK responses pass through unchanged, and trace output still contains only dimensions and count.

- valid 3D base64 embedding: {"dimensions": 16, "count": 1}
+ valid 3D base64 embedding: {"dimensions": 3, "count": 1}

Type of change

• Bug fix

What changed

• Decode base64 with validation and count dimensions only when decoded byte length is divisible by four.
• Retain the prior fallback for decoding errors, non-ASCII input, ignored invalid characters, and incomplete float32 bytes.
• Exercise the real OpenAI response parser with httpx.MockTransport and the shared in-memory exporter, asserting both trace output and unchanged SDK-returned strings.

Verification

uv run --frozen pytest tests/unit/test_openai.py::test_base64_embedding_exports_vector_dimensions -q
uv run --frozen pytest tests/unit/test_openai.py tests/unit/test_openai_prompt_extraction.py -q
uv run --frozen ruff check .
uv run --frozen ruff format --check .
uv run --frozen mypy langfuse --no-error-summary

All 7 base64 cases and all 68 affected OpenAI unit tests passed. Ruff, whole-tree format check, whole-package mypy and git diff --check passed. No live provider or Langfuse server was used; the remote Python-version matrix was not run locally.

Evidence

The original valid-base64 regressions report string lengths on the base implementation and float32 dimensions with the fix. The additional !!!! and AAAA cases fail on the earlier PR head with 0 dimensions instead of the fallback 4; both now pass alongside valid vectors, malformed input, non-ASCII passthrough, and existing float-array coverage.

Merge Danger

Door: two-way. Blast Radius: embedding trace output for explicit base64 responses. No public signatures or provider responses change.

Checklist

• I self-reviewed the diff using code_review.md (Codex review).
• I added or updated tests for behavior changes.
• I updated docs, examples, or .env.template if needed. No documentation becomes stale from correcting this count.
• I did not hand-edit generated files.
• I did not commit secrets or credentials.

AI disclosure: The implementation, regression tests, and self-review were produced with OpenAI Codex. No human review is claimed.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Malformed strings bypass the fallback

base64.b64decode silently discards invalid characters by default. A provider response of "!!!!" therefore exports 0 dimensions instead of the previous 4. Likewise, "AAAA" decodes to three bytes, which // 4 truncates to 0 despite not containing a complete float32.

Use strict decoding and check that the decoded length is divisible by four before counting dimensions. Add both cases to the fallback tests.

Prompt To Fix With AI

This is a comment left during a code review.
Path: langfuse/openai.py
Line: 971

Comment:
**Malformed strings bypass the fallback**

`base64.b64decode` silently discards invalid characters by default. A provider response of `"!!!!"` therefore exports 0 dimensions instead of the previous 4. Likewise, `"AAAA"` decodes to three bytes, which `// 4` truncates to 0 despite not containing a complete float32.

Use strict decoding and check that the decoded length is divisible by four before counting dimensions. Add both cases to the fallback tests.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[as992949791]

Addressed in 5f2ecb6. Base64 decoding now uses strict validation, and decoded bytes are counted as float32 dimensions only when their length is divisible by four. Both !!!! and AAAA retain the previous string-length fallback of 4; the SDK response remains unchanged.

The two added cases fail on the previous PR head and pass with this fix. All 7 base64 cases and 68 affected OpenAI unit tests passed, along with Ruff, whole-tree formatting, and whole-package mypy. This follow-up was prepared and self-reviewed with Codex.

[chrikrah]

@as992949791 From my perspective this can be merged: at 5f2ecb6 a base64 embedding of 1536 floats exports dimensions: 1536, where base exported the string length, 8192.

# git archive 5f2ecb6 (base 043321f), Python 3.12.14, openai 3.42.1, langfuse.__file__ under the tree
$ python -m pytest tests/unit/test_openai.py -k embedding -q
9 passed, 36 deselected in 1.05s

# langfuse/openai.py from 043321f, tests kept
3 failed, 6 passed, 36 deselected in 0.89s

The CLA gate is the one red check on this head. Could you sign it via the CLAassistant link above so it can go to a maintainer?