[viliamgonda]

Fixes langfuse/langfuse#19610.

Summary

This fixes two interrelated accounting and pagination boundary issues in BatchEvaluationRunner.run_async:

1. EvaluatorStats Score Count Accounting:
   When _add_observation_scores_to_trace=True, _create_score creates two distinct score records in Langfuse (one on the observation and one duplicated onto the parent trace), and result.total_scores_created increments by 2 per evaluation. However, stats.total_scores_created was incremented by len(eval_results) (1 per evaluation), causing per-evaluator score counts to report half of the actual scores created.

   • Fix: Multiplies len(eval_results) by 2 if _add_observation_scores_to_trace else 1 when updating stats.total_scores_created, ensuring per-evaluator stats align with result.total_scores_created.

2. max_items Premature Halting on Deduplicated Pages:
   In run_async, page clamping and loop termination were checked against total_items_fetched >= max_items rather than processed items. When ClickHouse page boundary duplicates or resumed items were skipped via _deduplicate_page, total_items_fetched reached max_items early, terminating the loop before total_items_processed reached max_items.

   • Fix: Evaluates loop completion against total_items_processed >= max_items, calculates page limit as min(limit, max_items - total_items_processed), and clamps items_to_process to remaining needed items.

Verification

• uv run pytest tests/unit/test_batch_evaluation.py: 18/18 passed.
• Added regression tests covering evaluator score duplication tracking and boundary deduplication with max_items.
• uv run ruff check langfuse/batch_evaluation.py tests/unit/test_batch_evaluation.py: All checks passed.
• uv run ruff format --check langfuse/batch_evaluation.py tests/unit/test_batch_evaluation.py: Clean.
• uv run mypy langfuse/batch_evaluation.py: Success, no issues found.

Confidence Score: 4/5

Fix the failed-item limit before merging so bounded runs cannot attempt the entire dataset.

Summary

This PR counts trace-level score copies in evaluator statistics and lets pagination continue past skipped duplicates.

• Batch runs keep processing until they reach the requested item count.
• Evaluator stats count both scores when observation scores are copied to traces.

_{Reviews (1) · Last reviewed commit: "fix(evaluations): account for score dupl..." · Reviewed by Greptile}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[greptile-apps]

Failed items bypass the limit

max_items now counts only successful items. If the mapper raises or score creation fails, total_items_failed increases but the runner keeps fetching replacements. With max_items=1 and a mapper that fails for every item, the runner attempts the entire matching dataset instead of stopping after one item. This can greatly extend runtime and work done by the mapper or evaluators.

Count successful and failed unique items toward the limit, while excluding skipped duplicates.

Prompt To Fix With AI

This is a comment left during a code review.
Path: langfuse/batch_evaluation.py
Line: 945

Comment:
**Failed items bypass the limit**

`max_items` now counts only successful items. If the mapper raises or score creation fails, `total_items_failed` increases but the runner keeps fetching replacements. With `max_items=1` and a mapper that fails for every item, the runner attempts the entire matching dataset instead of stopping after one item. This can greatly extend runtime and work done by the mapper or evaluators.

Count successful and failed unique items toward the limit, while excluding skipped duplicates.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Test never reaches the duplicate

The fake API sorts these rows as obs-3, obs-2, obs-1, duplicate, obs-0. With max_items=3, the runner fetches two rows and then one row, so it stops before fetching the duplicate. Both the old and new code pass, leaving the pagination fix unguarded.

Use max_items=4 and assert all four unique items are processed, or move the duplicate into a page fetched before the limit.

Prompt To Fix With AI

This is a comment left during a code review.
Path: tests/unit/test_batch_evaluation.py
Line: 435-438

Comment:
**Test never reaches the duplicate**

The fake API sorts these rows as `obs-3`, `obs-2`, `obs-1`, duplicate, `obs-0`. With `max_items=3`, the runner fetches two rows and then one row, so it stops before fetching the duplicate. Both the old and new code pass, leaving the pagination fix unguarded.

Use `max_items=4` and assert all four unique items are processed, or move the duplicate into a page fetched before the limit.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!