Skip to content

[schema-descriptions] Put probability criteria on properties - #32

Merged
egafni merged 8 commits into
fix/choice-score-descriptions-survive-reffrom
erik/property-schema-descriptions
Sep 6, 2026
Merged

egafni merged 8 commits into
fix/choice-score-descriptions-survive-reffrom
erik/property-schema-descriptions

Conversation

@egafni

@egafni egafni commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Put each Choice/Score criterion on its concrete probability property.
  • Keep the question on the unique probability-map definition and the $ref site bare.
  • Document the silent provider-schema transformation failure and the exact affected-mode matrix.
  • Refresh only the four affected probability fixtures across OpenAI/Anthropic and native/prompted transport.
  • Store cassette request and response JSON bodies as nested objects instead of escaped strings.
  • Add behavioral probes whose correct answers require reading the question instructions and criteria across every provider, transport, and answer-mode combination.

What happened

Probability-mode Choice and Score answers use nested probability-map models. Before PR #31, Pydantic emitted their question and criteria in a description beside the field's $ref. Anthropic's Python SDK structured-output transformer silently emitted only $ref, without an error or warning, so Anthropic received option names and numeric bounds but not the question or criteria. OpenAI preserved the description, and prompted mode serialized the unmodified raw schema into the prompt.

The same early-return behavior is discussed in anthropic-sdk-python #1642, although that issue focuses on a dropped $defs sibling.

Questions and answer mode Output transport OpenAI Anthropic
Choice/Score probabilities Native (structured_outputs=True) Not affected Affected before PR #31
Choice/Score probabilities Prompted (structured_outputs=False) Not affected Not affected
Choice/Score discrete Native or prompted Not affected Not affected
Noul probabilities or discrete Native or prompted Not affected Not affected

Schema placement

  • The probability-map definition describes the question and probability semantics once.
  • Each concrete property describes only its corresponding option or rubric criterion.
  • The containing answer field is a bare $ref, so behavior-critical context does not depend on sibling-keyword preservation or Pydantic deduplication.
  • Recorded requests confirm each question and sampled criterion appears exactly once in all four provider/transport combinations.
  • The existing cassette-backed end-to-end test follows the Choice $ref in both native provider requests and verifies that its question and criteria remain inside the provider schema.
  • A separate cassette-backed behavioral test uses opaque Choice labels so the models cannot answer one probe without its instruction or the other without its criteria. It requires the expected choice to receive probability greater than 0.9; all four probability-mode recordings returned 1.0 for both probes.

Dependency

Stacked on #31, which establishes the referenced-definition workaround. This layer refines where criteria live and adds the compatibility documentation.

Verification

  • uv run pytest --record-mode=none — 70 passed.
  • uv run ruff check --fix src/system_one_client_adapter/utils/pydantic_utils.py tests/test_client_with_live_apis.py
  • uv run git diff --check
  • Re-recorded and reviewed the four probability fixtures only.
  • Verified the readable-body migration preserved the normalized contents of all nine pre-existing cassettes exactly.
  • Recorded and reviewed all eight behavioral-probe combinations: OpenAI/Anthropic × native/prompted × probabilities/discrete.

@egafni egafni changed the title erik/property schema descriptions [schema-descriptions] Put probability criteria on properties Sep 6, 2026
@egafni
egafni merged commit 27f2315 into main Sep 6, 2026
@danielgafni
danielgafni deleted the erik/property-schema-descriptions branch September 15, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant