[rudycelekli]

Defect

def generate_inner(self, inputs, system_prompt=None, **kwargs) -> str:
    payload = {
        'model': self.model,
        'max_tokens': self.max_tokens,
        'messages': self.prepare_inputs(inputs),
        **kwargs
    }

temperature is accepted and stored by __init__ (claude.py:40, :51) and then never used. grep -c 'self.temperature' vlmeval/api/claude.py returns 1 — the assignment. Because temperature is a named __init__ parameter it never falls into **kwargs, so it cannot reach BaseAPI.default_kwargs and cannot arrive through generate_inner(**kwargs) either. There is no route by which the configured value reaches the request.

Why this is a defect and not a convention

1. vlmeval/config.py sets temperature=0 for every Claude entry — Claude3V_Opus (config.py:548), Claude3V_Sonnet, Claude3-5V_Sonnet, Claude3-5V_Sonnet_20241022, Claude4_Sonnet, … The repo asks for greedy decoding and does not get it.
2. Four sibling wrappers in the same package already forward it, and gpt.py is the house idiom:

   # vlmeval/api/gpt.py:268
   temperature = kwargs.pop('temperature', self.temperature)
   ...
   payload = dict(model=self.model, messages=input_msgs, n=1, temperature=temperature, **kwargs)

   Same in gcp_vertex.py:188,:201 (which talks to the same Anthropic Messages API via Vertex), bedrock.py:138,:146 and litellm_api.py:120.
3. max_tokens, declared on the adjacent line of the same __init__, is read back into the payload. Only temperature is dropped.

Reachability

ANTHROPIC_BACKEND=official ANTHROPIC_API_KEY=... python run.py --data MMBench_DEV_EN --model Claude3-5V_Sonnet
  -> supported_VLM['Claude3-5V_Sonnet']()      (config.py, temperature=0)
  -> Claude3V.generate -> BaseAPI.generate
  -> Claude_Wrapper.generate_inner             (claude.py:120)
  -> requests.request('POST', self.url, data=json.dumps(payload))

The Anthropic Messages API default temperature is 1.0, so every Claude evaluation in this repo has been running sampled rather than greedy: re-running the same benchmark on the same model produces different scores, and the temperature=0 in the repo's own config is silently a no-op. Both backends (alles and official) are affected, since both go through this one payload.

Fix

Adopt gpt.py's idiom. A per-call temperature still wins over the configured one, which matters because BaseAPI.generate forwards default_kwargs and a few dataset evaluators vary temperature across retries.

max_tokens gets the same kwargs.pop treatment for symmetry. That is not a behaviour change: a max_tokens entry in **kwargs already shadowed the dict literal (a later **kwargs key overrides an earlier literal key), so the effective value is the same either way — the test suite pins both orderings.

Tests

tests/test_claude_temperature.py — 10 cases, offline. requests.request is monkeypatched on the module so the real generate_inner runs and the actual serialised payload is captured; the wrapper is built with object.__new__ so no key or env var is needed.

At 4d0dc80, 5 cases fail by name:

$ python3 -m pytest tests/test_claude_temperature.py -q
FAILED ...::test_configured_temperature_is_sent                 # KeyError: 'temperature'
FAILED ...::test_any_configured_temperature_is_sent[0]
FAILED ...::test_any_configured_temperature_is_sent[0.2]
FAILED ...::test_any_configured_temperature_is_sent[0.7]
FAILED ...::test_any_configured_temperature_is_sent[1.0]
5 failed, 5 passed     exit 1

With this change: 10 passed, exit 0.

The 5 that pass at base are the guards, and one of them is worth calling out: test_per_call_temperature_overrides_the_configured_one already passed, because an explicit temperature= kwarg did reach the payload via **kwargs. Only the configured default was being dropped, and the test pins that the kwargs route keeps working. The others pin model, max_tokens (configured and per-call), messages, system, an arbitrary passthrough kwarg (top_p), and the response parsing.

Whole-suite check in the same selection shape, before and after: identical, 81 failed, 82 passed, 2 skipped, 21 errors either way (local-environment failures — 13 test files need Python 3.10+ for str | None in vlmeval/smp/dataset_alias.py, the rest need vlmeval installed). flake8 --max-line-length=120 --ignore=W503 on both touched files: exit 0. git diff --check: exit 0. I have not run the GPU or mock-API CI jobs and have not made a live Anthropic API call.