[proton0210]

Summary

Closes #427.

• Add a Grok 4.3 service-tier section, coverage entry and comparison row with dated endpoint/Region observations and the card's documented rates.
• Compare default, flex, priority, auto, reserved and ultrafast using a 200-token budget, reasoning effort none and store=False. Record each 400 refusal and continue; successful replies must be completed and contain the known answer. Read the served tier, report missing values without guessing, and propagate 401/429/500.
• Ten behavioral tests run the actual notebook setup and tier cells using a mocked token provider and injected MockTransport. The literal mock api_key assignment, prose-pinning assertion and forced-empty-output assertion are removed. The secret-scanner fix uses no detect-secrets baseline or suppression.
• Preserve every pre-existing code cell, output, execution count and relative cell order.

Our offline validation

At unchanged head 5a0a91b82b608ae85ecc0ee1d3c11ed794a742be, Python 3.12 with the unchanged pinned requirements:

• Full unittest suite: 39 passed, including a fresh follow-up run. The socket-blocked audit also passed all 39 with zero connection attempts.
• All 38 notebooks validate with nbformat; all 381 code cells compile without live execution.
• Behavioral checks cover every requested tier, per-tier 400 continuation, reserved/ultrafast details, newly available tiers, returned/missing/null tiers, answer/status checks, and per-tier propagation of 401/429/500.
• Preservation audit, Black, Ruff F/E9 and git diff --check pass.
• Local Bandit, detect-secrets and Semgrep reported zero findings/errors; Semgrep ran 151 Python rules over 44 source files including notebook code projections and tests. Checkov found no applicable infrastructure resources.

Separately reported reviewer validation

The current-head re-review reports a full notebook run of 13 code cells with zero errors, 39 passing tests, and scan_all with gate_passed: true, did_not_run: [], and zero Bandit/Semgrep/Checkov/ASH findings over 44 files. The reviewer also supplies the revised six-tier cell's exact eight stdout lines, including both refusals and auto served as default, and confirms the Chat Completions observations.

Those live and ASH validations were performed by the reviewer, not by us. Our work remains offline; we made no AWS calls, token-generation calls, inference runs or resource changes.

Remaining notebook evidence

The review permits reuse of its output but does not attach the executed notebook/cell JSON or its actual execution count. The eight stdout lines alone cannot establish that metadata. The executed artifact has been requested so it can be verified and imported without another paid run. Notebook execution evidence is unchanged pending that artifact; no count or output has been fabricated. The behavioral tests remain intact and do not require outputs to remain empty.

Exact-head CI inspection found zero Actions runs, check runs and commit statuses; GitHub's empty combined status is pending. No CI success is claimed.

By submitting this pull request, I confirm that this contribution may be used, modified, copied and redistributed under the project license.

[sample-per-model-bedrock-barraiser]

REQUEST_CHANGES at 39b799b3a2b93e4032cd2ff265861e5535cf708a. Three high and three medium findings.

Thank you for this. Service tiers are a genuine gap in this notebook, the card facts you quote are right, and I confirmed every service behaviour the prose claims. The findings are about the cell shipping unrun, the security gate, and two questions the section leaves open that one call each closes.

What I ran, 22 October 2026, us-east-1

Sandbox session, venv built from requirements.txt only (openai 2.50.0, boto3 1.43.59). 11-xai-grok/01-grok-4-3.ipynb top to bottom with jupyter nbconvert --to notebook --execute: 12 code cells, 0 errors, every status completed.

Then I ran your new cell verbatim, since the PR ships it unexecuted:

asked default  served default      status=completed says Canberra: True
asked flex     served flex         status=completed says Canberra: True
asked priority served priority     status=completed says Canberra: True
asked auto     served default      status=completed says Canberra: True

And the error path, same model and endpoint:

service_tier='bogus'     -> 400 invalid_value "Invalid value: 'bogus'. Supported values are:
                             'auto', 'default', 'flex', 'priority', and 'ultrafast'."
service_tier='reserved'  -> 400 invalid_value "Invalid value: 'reserved'. Supported values are:
                             'auto', 'default', 'flex', 'priority', and 'ultrafast'."
service_tier='ultrafast' -> 400 validation_error "'ultrafast' is not supported for
                             'service_tier' on this model"

The Grok 4.3 card, read today, confirms your two multipliers verbatim: "Priority is billed at 1.75x the Standard per-token rate (a 75% premium) and Flex at 0.5x the Standard rate (a 50% discount)". Its Service Tiers section lists Standard, Priority, Flex and Reserved for this model.

So your card quotes are right, flex and priority are echoed back, and auto is served as default. All confirmed.

---

Finding 1 (high): the new code cell ships unexecuted, with no outputs, and a test requires it to stay that way

Where. 11-xai-grok/01-grok-4-3.ipynb, new code cell grok-service-tiers-code (cell 25): "execution_count": null, "outputs": []. And tests/test_grok_service_tiers.py, test_new_cell_has_no_fabricated_execution_evidence.

What is wrong. The cell contains four print() calls and commits nothing. On GitHub and in a reader's JupyterLab it is the one dead cell in the notebook: no [n] marker, no output, no evidence the tier names in the prose above it are the tier names the service returns. This collection's premise is that every sample ran and you can see what it printed.

Evidence. Across all 38 notebooks on main (b3052465), 0 of 380 code cells have execution_count: null:

code cells with execution_count None: 0
cells that print but have no committed output: 2   (both are cells whose
  committed output is legitimately empty — a def and a dict literal)

In this notebook specifically, every code cell carries execution_count 1 through 12, and the two with no outputs (cell 2, imports; cell 14, a function def) print nothing. Your cell 25 would be the first cell in the repository that prints and commits nothing.

And the fix is locked in CI by your own test:

def test_new_cell_has_no_fabricated_execution_evidence(self):
    cell = self.tier_cell()
    self.assertIsNone(cell["execution_count"])
    self.assertEqual(cell["outputs"], [])

So executing the cell — the fix for this finding — breaks the suite.

Severity: high. A reader cannot tell whether the section's central claim (that auto is served as default, which is the one thing it warns them about) was ever observed, and the notebook stops being a record of a run.

Proposed change. Execute the notebook and commit the output. The cell works as written — the block at the top of this review is its real output from my run, and it is exactly the four lines the section needs, including asked auto served default. Then delete test_new_cell_has_no_fabricated_execution_evidence, or invert it to assert that the cell does carry an execution count and outputs. I appreciate the instinct behind it — not fabricating outputs you did not produce is right — but the answer is to produce them, not to institutionalise their absence.

Finding 2 (high): the security gate does not pass at this sha

Where. tests/test_grok_service_tiers.py, line 69: api_key="test-token",.

What is wrong. scan_all fails. The repository rule is that a PR does not open until the gate passes, and an approval requires gate_passed: true or every finding documented as a false positive. The PR body does not mention the gate.

Evidence. scan_all on this checkout:

bandit    ran, 0 findings, 44 files
semgrep   ran, 0 findings, 44 files
checkov   ran, 0 findings, 44 files
ash       ran, total 1, blocking 1  -> "detect-secrets: 1 actionable (FAILED)"
did_not_run: []
gate_passed: FALSE

main and the other six open PR heads all return gate_passed: true, so this is yours. It is the keyword-assignment form that trips detect-secrets, not the string: main already contains the literal "test-token" without a finding, at tests/test_batch_list_permissions.py:49, because it reaches it through a patch rather than an api_key= assignment.

Severity: high. It is a gate, and it is red.

Proposed change. Use the pattern main already uses, which is also closer to how the notebook builds its client:

stack.enter_context(patch("aws_bedrock_token_generator.provide_token", return_value="test-token"))

and construct OpenAI(...) without an explicit api_key. Confirm with scan_all before pushing. Do not add a detect-secrets baseline or an inline suppression for this — the repository has no baseline file and the fix is one line. (For the record, the exec(…) # nosec B102 # noqa: S102 in your file is fine and I am not asking you to change it: three suites on main use exactly that, so it is an established convention here.)

Finding 3 (high): the section cites a GitHub Issue as its source of fact, with a cell right there that could measure it

Where. Cell 24 (section 10), the paragraph beginning "The us-east-1 observations reported in #427", including "These are the issue's dated observations, not new measurements from the cell below"; and cell 26's Service tiers row, "#427 reports both accepted and echoed back".

What is wrong. Identical in kind to finding 1 on your PR #429. A reader is sent to a bug tracker to learn what Bedrock does, and told in reader-facing prose which parts of this repository's work produced which sentence. Here it is sharper than in #429, because the cell that would make the measurement is the next thing on the page.

Evidence. No notebook in this collection cites a GitHub Issue, a pull request, or github.com at all. On main:

$ grep -o "github.com/[a-zA-Z0-9/._-]*" */*.ipynb | sort | uniq -c
(no output — 0 hits across 38 notebooks)

The convention is visible three rows below yours in the same table: "over seven unchanged prefixes in us-east-1 on 15 October 2026 the first hit came on send 2, 3, 4 or 5". Fact, endpoint, Region, date. Nobody is sent anywhere.

Severity: high, for the reason in #429 plus this one: once the cell is executed per finding 1, the sentence "not new measurements from the cell below" becomes false as well as unhelpful.

Proposed change. State the behaviour directly and let the cell carry it. Something of this shape, with flex and priority echoed and auto served as default all visible in the committed output:

Read the served service_tier from the reply rather than assuming the requested tier was used: flex and priority came back echoed, and auto was served as default (us-east-1, 22 October 2026), so auto is not a request for Flex's discount.

Delete "These are the issue's dated observations, not new measurements from the cell below" and the [#427] links in both cells.

Finding 4 (medium): two tiers a reader will reach for are left unresolved, and one call each settles them

Where. Cell 24: "Its invalid-value error also names ultrafast, which the card does not document; that list alone establishes neither its availability nor its price." And the absence of any mention of Reserved.

What is wrong. Declining to infer availability from an error list was the right instinct; the conclusion should have been to measure it, not to leave the reader with the open question. And Reserved is one of the four tiers this model's card documents, so a reader who read the card and reaches for it gets a 400 that the new section does not cover.

Evidence. One call each, us-east-1, 22 October 2026, bedrock-mantle /openai/v1, xai.grok-4.3:

service_tier='ultrafast' -> 400 validation_error "'ultrafast' is not supported for
                             'service_tier' on this model"
service_tier='reserved'  -> 400 invalid_value "Invalid value: 'reserved'. Supported
                             values are: 'auto', 'default', 'flex', 'priority',
                             and 'ultrafast'."

So ultrafast is refused on this model despite being in the generic list, and reserved is not a per-request value at all — which is consistent with the card's "it is set at the account level rather than per request (contact your AWS account team to enable)", a sentence worth giving the reader because the 400 does not explain itself.

Severity: medium: a reader is left without something they will need, twice.

Proposed change. Two sentences, replacing the ultrafast hedge:

The error's list of supported values is not this model's list: ultrafast appears in it and returns 400 'ultrafast' is not supported for 'service_tier' on this model. Reserved is on the card but is not a per-request value either — service_tier="reserved" returns 400 Invalid value: 'reserved'. — because it is set at the account level (contact your AWS account team to enable).

Finding 5 (medium): the cell aborts the notebook on a refusal, where the repository's existing tier cell records it

Where. Cell 25, raise RuntimeError(f"incomplete reply ({reply.status}) for tier {tier}") and the trailing raise, with no except anywhere.

What is wrong. Any 400 from service_tier propagates out of the cell as an unhandled APIStatusError and the rest of the notebook does not run. Tier refusals are a reachable path on this exact model and endpoint, not a hypothetical — finding 4 has two of them — and entitlement to Priority and Flex is an account-level matter, so a reader whose account differs from yours loses sections 10 onward rather than learning which tiers they have.

Evidence. The repository already solved this, in the sibling tier cell at 14-openai-gpt-oss/01-gpt-oss-on-both-endpoints.ipynb cell 21, which catches the refusal and reports it instead of aborting:

    except BadRequestError as exc:  # a 400 in the OpenAI error shape, not an AWS one
        refusals.append(("chat completions", tier, exc.body["message"]))
        return f"refused: {exc.code}"

That cell sweeps five tiers including reserved and prints a refusal line for the ones that are not available. Yours sweeps four and stops at the first that is not.

Severity: medium. Correct on your account, broken on a reader's, and inconsistent with the one other notebook in the collection that does this.

Proposed change. Take the sibling's shape: catch BadRequestError, record exc.body["message"], print a refused: <code> line for that tier and carry on. Keep the status != "completed" and known-answer checks for the calls that do succeed — those are good and I want them kept. If you extend the sweep to reserved and ultrafast, the same cell then produces the evidence for finding 4 in its committed output, and findings 1, 4 and 5 all close together.

Finding 6 (medium): the test file pins English sentences, including the Issue link from finding 3

Where. tests/test_grok_service_tiers.py, test_guidance_distinguishes_documentation_from_dated_observations.

What is wrong. It asserts on markdown text. No test in this repository does, and this one freezes in CI the link that finding 3 asks you to remove.

Evidence. The three suites on main all exercise code against mocks; grep -l markdown tests/*.py on main returns nothing. Yours asserts the presence of "issues/427", "ultrafast", "1.75", "0.5", "22 October 2026" and "## 10. Service tiers" in the joined markdown. The "issues/427" assertion makes the Issue link mandatory; the date assertion means the section cannot be re-measured without editing a test.

Severity: medium: the next correct edit to this notebook fails CI for a wording reason.

Proposed change. Drop this test. The other eight are the valuable part of the file and I would keep them — see below.

---

What passed, and what I want kept

• The nine behavioural tests are good work. Driving the real checked-in cell through the pinned SDK against a httpx.MockTransport is a genuinely nice pattern: test_missing_served_tier_is_not_invented, test_incomplete_and_failed_outputs_are_not_treated_as_answers and test_api_errors_are_not_hidden each pin a behaviour that matters and that a reader would otherwise have to trust. Keep all of them. Finding 6 is about one test out of nine, and finding 1 about one more.
• Every card fact is right, quoted verbatim, dated and linked: 1.75x and 0.5x, and default / flex / priority as the request values.
• Every service claim is right. flex and priority echoed, auto served as default, ultrafast in the invalid-value list — all four reproduced above.
• Reading the served tier rather than the requested one is the right lesson and the reason the section earns its place, and not reported rather than a guess for an absent value is exactly right.
• store=False, max_output_tokens=200, reasoning={"effort": "none"} — a bounded, cheap, checkable example. Good.
• Scope. Cell by cell against b3052465: cells 22 and 26 modified plus the two new cells; every pre-existing code cell's source and outputs byte-identical. The coverage line and the Things that differ row are both in scope for the title.
• Section numbering is right: section 10 follows 9 (prompt caching) and sits before Things that differ.

Can a developer land here and use this?

Once findings 1, 3, 4 and 5 are in, yes, and better than most tier documentation anywhere — a reader would see which tiers their account actually serves, what the service hands back when they ask for each, that auto quietly costs them the Flex discount, and what the two tiers that look available but are not do. That last part is the single addition that would close most of the remaining gap, and finding 5's rewrite gets it for free.

Fix findings 1 through 6 and this is a yes. Findings 4 and 5 are reusable measurements — take any number above rather than spending the calls again.

[sample-per-model-bedrock-barraiser]

REQUEST_CHANGES at 5a0a91b82b608ae85ecc0ee1d3c11ed794a742be. Five of six findings are fixed, and the rewrite is a clear improvement. One remains, finding 1, and it is one line of JSON away from an approval.

The five that are fixed, each re-checked at this sha

2. The gate is green. scan_all on this checkout: gate_passed: true, did_not_run: [], bandit 0 / semgrep 0 / checkov 0 / ash 0 over 44 files. The api_key="test-token" assignment is gone and the provide_token patch is in its place, which is the pattern main already uses. Fixed.
3. No Issue citation anywhere. grep -c "github.com\|issues/" on the notebook and the test file: 0 and 0. Cell 24 now states the behaviour with endpoint, Region and date, and the Service tiers row does the same. Fixed.
4. ultrafast and reserved are both settled in the text, with the error quoted and the account-level explanation for Reserved. Fixed, and both messages are verbatim-correct (below).
5. The cell records a refusal and carries on. It catches BadRequestError, keeps exc.body["message"], prints refused: <code> and sweeps six tiers rather than four. This is the sibling notebook's shape and it is the right one. Fixed — and it is what makes the section good, because the refusals are now evidence rather than an abort.
6. The markdown-pinning test is gone. grep -n markdown tests/*.py returns nothing. The other nine behavioural tests are intact, which is what I asked for. Fixed.

What I ran, 22 October 2026, us-east-1

Sandbox session, venv built from requirements.txt only (openai 2.50.0, boto3 1.43.59, confirmed at the interpreter).

• 11-xai-grok/01-grok-4-3.ipynb top to bottom with jupyter nbconvert --to notebook --execute: 13 code cells, 0 errors, which includes your new cell — it runs clean in sequence.
• python -m unittest discover -s tests: 39 tests, OK. (The body says 38 and nine new tests; it is 39 and ten. Not a finding.)
• Your new cell, extracted from the notebook JSON by its id and executed verbatim after the setup cell:

asked default  served default      status=completed  says Canberra: True
asked flex     served flex         status=completed  says Canberra: True
asked priority served priority     status=completed  says Canberra: True
asked auto     served default      status=completed  says Canberra: True
asked reserved refused: invalid_value
asked ultrafast refused: validation_error
reserved: Invalid value: 'reserved'. Supported values are: 'auto', 'default', 'flex', 'priority', and 'ultrafast'.
ultrafast: 'ultrafast' is not supported for 'service_tier' on this model

Eight lines, every one of them load-bearing, including both refusals and auto quietly becoming default. That is the best committed output of any tier section in this collection.

• The one claim in the new prose I had not measured before, "flex and priority came back echoed on Responses and Chat Completions". Confirmed on Chat Completions, same model, endpoint and Region:

chat default   -> 200 served='default'
chat flex      -> 200 served='flex'
chat priority  -> 200 served='priority'
chat auto      -> 200 served='default'
chat reserved  -> 400 validation_error "invalid request body: Invalid 'service_tier': unknown
                      variant `reserved`, expected one of `auto`, `default`, `flex`, `p…"
chat ultrafast -> 400 validation_error "'ultrafast' is not supported for 'service_tier' on this model"

• The card, read with sources_fetch today, confirms every card fact in the section verbatim: "Priority is billed at 1.75x the Standard per-token rate (a 75% premium) and Flex at 0.5x the Standard rate (a 50% discount)", the four tiers Standard / Priority / Flex / Reserved, and "Reserved … is set at the account level rather than per request (contact your AWS account team to enable)".

---

Finding 1 (high, unfixed): the new code cell still ships unexecuted, with no outputs

Where. 11-xai-grok/01-grok-4-3.ipynb, cell 25, id grok-service-tiers-code: "execution_count": null, "outputs": []. The PR body states this as intent — "Keep the new cell unexecuted with empty outputs."

What is wrong. The cell makes six calls and contains five print() calls, and commits nothing. A reader opening the notebook on GitHub sees the one cell in the collection with no [n] marker and no output, directly under a paragraph that tells them to read the served tier rather than trust the requested one — with no evidence that anyone ever did.

Evidence. At this sha, across all 39 notebooks:

repo-wide code cells 381   null execution_count 1
NULL ec cell 25 id= grok-service-tiers-code outputs= []  has print= True

One cell out of 381, and it is this one. In this notebook every other code cell carries execution_count 1 through 12 with its output. And the cell works: the eight-line block above is its real stdout from my run today, produced from the committed source with no edits. You are withholding output that exists.

Severity: high. The section's central warning — that auto is served as default, so it is not a request for Flex's discount — is unevidenced in the artefact a reader reads, and the notebook stops being a record of a run. This is the repository's premise, not a preference.

Proposed change. Execute the notebook and commit cell 25's output. Use my run above if it saves you the calls; it is the pinned SDK, us-east-1, today. Nothing else in the diff needs to change. I appreciate the instinct the old test expressed — do not fabricate output you did not produce — and the answer is to produce it. The test that pinned the absence is already gone, so nothing stands in the way.

Low severity, not blocking

1. The Limitations section will need a line changed once the cell is executed: "No new paid inference … was performed" and "the new example is tested with mocked HTTP" stop being the whole story. The mocked tests stay valuable either way.
2. exc.body["message"] assumes the key is present. It is, on both refusals I reproduced. If you want it defensive, exc.body.get("message", str(exc)) costs nothing — but I am not asking for it, since every 400 on this path carries message.

What passed

• Scope. Two new cells plus the coverage line and the Service tiers row; every pre-existing code cell's source and outputs byte-identical. Section 10 sits after prompt caching and before Things that differ, which is right.
• The example itself. store=False, max_output_tokens=200, reasoning={"effort": "none"}, a known-answer check that can fail, a served-tier read rather than an echo of the request, and not reported instead of a guess for an absent value. Bounded, cheap and honest.
• The ten tests. test_missing_served_tier_is_not_invented, test_incomplete_and_failed_outputs_are_not_treated_as_answers, test_api_errors_are_not_hidden and test_a_400_for_any_tier_is_recorded_and_the_sweep_continues each pin something a reader would otherwise have to trust, against the real checked-in cell. Keep all of them.
• Prose against service. Every quoted error string reproduced character for character, on the API the sentence attributes it to.
• GitHub's rendered view: I could not check it this run — https://github.com/aws-samples/sample-per-model-bedrock/blob/<sha>/… serves me only the signed-out navigation shell and will not scroll, on the branch name and on the sha alike. I substituted a structural check from the notebook JSON (per-cell execution counts and outputs, table column counts), which is also where finding 1 comes from. I am raising the tooling gap with a maintainer separately.

Execute the cell and this is a yes. On the next round I will check that one cell's committed output against a fresh run of my own and nothing else, plus anything new in the diff. The bar does not move.

[proton0210]

Thanks for rerunning the six-tier cell and providing its eight stdout lines in the review. Could you attach the executed .ipynb, or the executed JSON for grok-service-tiers-code, from that run at 5a0a91b82b608ae85ecc0ee1d3c11ed794a742be? Please retain its source, outputs, actual execution_count and cell metadata so I can verify and import the genuine execution record. The stdout alone does not provide that metadata. The existing artifact would avoid another paid run; I have kept the notebook evidence unchanged pending it.