Skip to content

Custom endpoints behind a gateway never get a context length, so Claude assumes 200k #754

Closed

Description

@opticon454

On Codeman 1.47.0, a custom model endpoint that is a gateway in front of llama-swap (for example Bifrost) has no context length saved, so CLAUDE_CODE_MAX_CONTEXT_TOKENS is never set. Claude Code then prints Model "..." isn't recognized, context assumed to be 200k tokens and does not compact before a smaller server overflows. Codex, OpenCode and DeepSeek get no limit either.

Cause: modelContextLengths is only filled from llama-swap's /running launch command (and /props), which exists only for a loaded model and only when Codeman talks to llama-swap directly. A gateway has no /running. Discovery never reads the size from GET /v1/models, although current llama-swap sends context_length and context_window there (see llama-swap api.go) and Bifrost passes context_window through (Bifrost openai/models.go). OpenRouter-style servers send context_length.

Expected:

  • Discovery reads a positive integer context_length, else context_window, from each /v1/models entry. The llama-swap /running value keeps priority for a direct llama-swap endpoint.
  • An optional per-endpoint defaultContextLength is used for models whose size cannot be discovered, so a CLI never falls back to its own large default. Discovery must never overwrite it.
  • Regression tests for both deployments (direct llama-swap and a gateway).

Activity

  1. added 2 commits that reference this issue on Nov 1, 2026
    36f620c
    a97fa0d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions