Skip to content

perf(eve): queue session stream writes off the pre-model path - #5287

Draft
chadhietala wants to merge 1 commit into
mainfrom
perf/session-stream-writes
Draft

chadhietala wants to merge 1 commit into
mainfrom
perf/session-stream-writes

Conversation

@chadhietala

@chadhietala chadhietala commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Every session lifecycle event waited for its durable stream write before the turn continued. A turn’s session.started, turn.started, message.received, and step.started writes therefore sat between prompt preparation and the model call. In the eve-e2e traces, workflow.stream.write and workflow.stream.flush cost 55–130 ms p50. Session stream writes now queue in order, and the step drains them before it commits. Turn-boundary events keep the old contract: their write settles before their hooks and instrumentation run. The event protocol and dispatcher order do not change. A failed write still fails the step at the drain, and later writes fail fast. Workflow step lifecycle writes are unchanged.

Validation

  • pnpm --filter eve exec vitest run --config vitest.unit.config.ts src/execution/publish-session-events.test.ts — 10 passed
  • pnpm --filter eve exec vitest run --config vitest.unit.config.ts src/execution/session/turn-step.test.ts — 68 passed
  • pnpm --filter eve exec vitest run --config vitest.integration.config.ts src/subagents/event-proxy-step.integration.test.ts — 12 passed
  • pnpm --filter eve exec vitest run --config vitest.integration.config.ts src/execution/session/turn-connection-approval.integration.test.ts — 11 passed
  • pnpm typecheck — 59 tasks passed
  • pnpm guard:invariants — passed
  • pnpm exec oxfmt and pnpm exec oxlint on the changed files — clean

Not run: hosted latency benchmark, full integration/scenario tiers, and e2e. The 55–130 ms figure is the existing OTel baseline, not a post-change measurement.

Checklist

  • I added tests and documentation where relevant
  • I added a changeset if this touches the published eve package
  • DCO sign-off passes for every commit (git commit --signoff)

@vercel

vercel Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
eve-docs Ready Ready Preview, v0 Oct 23, 2026 2:29am UTC
eve-pkg Ready Ready Preview, v0 Oct 23, 2026 2:29am UTC

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Bundle + Package Summary: apps/fixtures/weather-agent

Key takeaways

  • No notable deltas vs main (d5b9601).

Delta vs main (d5b9601)

Area Metric Baseline Current Delta
Package Packed tarball 7.77 MB 7.77 MB +763 B ⚠️
Package Unpacked publish size 27.27 MB 27.27 MB +2.0 kB ⚠️
Package Installed footprint 75.57 MB 75.57 MB +2.0 kB ⚠️
Package Published files 4215 4215 0
Package Installed files 8209 8209 0
Package Installed package instances 34 34 0
Package Distinct installed package names 32 32 0
Package Installed dependency edges 51 51 0
Package Installed optional peer edges 8 8 0
Runtime Unique function payloads 2 2 0
Runtime Total function bytes 20.87 MB 20.87 MB +2.5 kB ⚠️
Runtime Public routes 18 18 0
Changed function payloads vs main (d5b9601) (2)
Function Status Baseline Current Delta Route changes
functions/__server.func changed 10.44 MB 10.44 MB +1.2 kB ⚠️ none
functions/.well-known/workflow/v1/flow.func changed 10.44 MB 10.44 MB +1.2 kB ⚠️ none

eve init install

Metric Baseline Current Delta
Installed footprint 113.95 MB 113.95 MB +2.0 kB ⚠️
Installed packages 97 97 0
dependencies 4 4 0
devDependencies 2 2 0
Dependency package bytes 45.67 MB 45.67 MB +2.0 kB ⚠️
devDependency package bytes 5.11 MB 5.11 MB 0 B ➖
Build Metadata
  • Preset: vercel
  • Nitro: nitro@3.0.260903-beta
  • Output directory: apps/fixtures/weather-agent/.vercel/output
  • Build metadata timestamp: 2026-10-23T02:29:52.488Z
  • Route aliases: 18 public, 1 internal (19 total aliases)
  • Vercel routes in config: 21
  • Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small
Package Drill-Down

Package Details

  • Package: eve@0.83.0
  • Package directory: packages/eve
  • Tarball: 7.77 MB (eve-0.83.0.tgz)
  • Unpacked payload: 27.27 MB across 4215 published files
  • Installed footprint: 75.57 MB across 8209 installed files
  • Installed root package: 27.27 MB
  • Installed dependencies: 48.30 MB
  • Installed package instances: 34
  • Distinct installed package names: 32
  • Installed dependency edges: 51
  • Installed optional peer edges: 8
  • Runtime dependencies: 2
  • Peer dependencies: 6 (5 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.
Graph metrics read only package.json files in package directories directly beneath a node_modules boundary, including nested boundaries. Each directory is one package instance; distinct names come from those manifests. Dependency edges count each unique name in dependencies or optionalDependencies per instance; optional peer edges count peerDependencies marked optional.

Heavy installed dependencies

  • eve: 27.27 MB (36.1%)
  • @rolldown/binding-linux-x64-gnu: 19.89 MB (26.3%)
  • ai: 8.36 MB (11.1%)
  • zod: 6.15 MB (8.1%)
  • undici: 3.58 MB (4.7%)
Publish payload breakdown
Published file size
🟠 dist/src/compiled/shadcn-registry/index.js       [#####...................] 3.85 MB 14.1%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [###.....................] 2.29 MB 8.4%
🟠 dist/src/compiled/run/index.js                   [#.......................] 1.00 MB 3.7%
🟡 dist/src/compiled/@vercel/blob/index.js          [#.......................] 604.9 kB 2.2%
🟡 dist/src/compiled/_chunks/workflow/undici-D-I... [#.......................] 521.9 kB 1.9%
🔴 Other published files                            [########################] 19.01 MB 69.7%
Installed footprint breakdown
Installed package size
🔴 eve                             [########################] 27.27 MB 36.1%
🔴 @rolldown/binding-linux-x64-gnu [##################......] 19.89 MB 26.3%
🔴 ai                              [#######.................] 8.36 MB 11.1%
🔴 zod                             [#####...................] 6.15 MB 8.1%
🟠 undici                          [###.....................] 3.58 MB 4.7%
🟠 nitro                           [##......................] 1.89 MB 2.5%
🔴 Other installed packages        [#######.................] 8.42 MB 11.1%
Runtime dependencies (2)
Package Range Notes
nitro 3.0.260903-beta
undici 8.10.2
Peer dependencies (6)
Package Range Notes
@opentelemetry/api ^1.0.0 optional peer
ai catalog:
braintrust ^3.0.0 optional peer
chat ^4.42.0 optional peer
dd-trace ^6.13.0 optional peer
just-bash ^3.1.0 optional peer
eve init install drill-down

eve init install details

  • Command: eve init my-agent
  • Package manager: npm
  • Installed footprint: 113.95 MB across 10126 installed files
  • Installed packages: 97 total (91 transitive-only)
  • dependencies: 4 direct packages totaling 45.67 MB
  • devDependencies: 2 direct packages totaling 5.11 MB
  • Other transitive package files: 63.17 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

  • @typescript/typescript-linux-x64: 27.95 MB (24.5%)
  • eve: 27.27 MB (23.9%)
  • @rolldown/binding-linux-x64-gnu: 19.89 MB (17.5%)
  • zod: 9.76 MB (8.6%)
  • ai: 8.36 MB (7.3%)
Installed footprint breakdown
Installed package size
🔴 @typescript/typescript-linux-x64 [########################] 27.95 MB 24.5%
🔴 eve                              [#######################.] 27.27 MB 23.9%
🔴 @rolldown/binding-linux-x64-gnu  [#################.......] 19.89 MB 17.5%
🔴 zod                              [########................] 9.76 MB 8.6%
🔴 ai                               [#######.................] 8.36 MB 7.3%
🟠 undici                           [###.....................] 3.58 MB 3.1%
🔴 Other installed packages         [###############.........] 17.15 MB 15.0%
dependencies (4)
Package Range Installed size Share
@vercel/connect 2.5.2 276.3 kB 0.2%
ai ^7.0.137 8.36 MB 7.3%
eve file:eve-0.83.0.tgz 27.27 MB 23.9%
zod 4.5.4 9.76 MB 8.6%
devDependencies (2)
Package Range Installed size Share
@types/node 24.x 2.61 MB 2.3%
typescript 7.0.2 2.50 MB 2.2%
Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 10.44 MB 50.0%
🔴 functions/__server.func                         [########################] 10.44 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 10.44 MB
Metric Value
Public routes /.well-known/workflow/v1/flow
Runtime nodejs24.x
Handler index.mjs
Payload 10.44 MB
Function files 10.44 MB across 121 files
Traced dependencies 0 B
Signal 🟠 Bundled file _chunks/vercel.web.mjs is 2.47 MB (23.7%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 _chunks/vercel.web.mjs                         [#############...........] 2.47 MB 23.7%
🟡 _libs/undici.mjs                               [#####...................] 994.5 kB 9.5%
🟡 _chunks/sandbox2.mjs                           [####....................] 815.0 kB 7.8%
🟡 _chunks/compiled-artifacts-instrumentation.mjs [####....................] 732.3 kB 7.0%
🟡 _chunks/token-util-Bla5Z0G9.mjs                [####....................] 704.5 kB 6.7%
🔴 Other bundled files                            [########################] 4.72 MB 45.2%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 17 public routes, 1 internal alias • 10.44 MB
Metric Value
Public routes /
/.well-known/workflow/v1/webhook/[token]
/eve/v1/callback/[token]
/eve/v1/connections/[name]/callback/[attemptId]/[token]
/eve/v1/connections/[name]/callback/[token]
/eve/v1/health
/eve/v1/info
/eve/v1/session
/eve/v1/session/[parentSessionId]/subagents/[callId]/[childSessionId]/stream
/eve/v1/session/[sessionId]
/eve/v1/session/[sessionId]/cancel
/eve/v1/session/[sessionId]/clear
/eve/v1/session/[sessionId]/compact
/eve/v1/session/[sessionId]/reset
/eve/v1/session/[sessionId]/stream
/eve/v1/session/[sessionId]/stubs
/eve/v1/task-input/[token]
Internal aliases /__server
Runtime nodejs24.x
Handler index.mjs
Payload 10.44 MB
Function files 10.44 MB across 121 files
Traced dependencies 0 B
Signal 🟠 Bundled file _chunks/vercel.web.mjs is 2.47 MB (23.7%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 _chunks/vercel.web.mjs                         [#############...........] 2.47 MB 23.7%
🟡 _libs/undici.mjs                               [#####...................] 994.5 kB 9.5%
🟡 _chunks/sandbox2.mjs                           [####....................] 815.0 kB 7.8%
🟡 _chunks/compiled-artifacts-instrumentation.mjs [####....................] 732.3 kB 7.0%
🟡 _chunks/token-util-Bla5Z0G9.mjs                [####....................] 704.5 kB 6.8%
🔴 Other bundled files                            [########################] 4.72 MB 45.2%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: e2e/fixtures/agent-tools-sandbox

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

  • Build pipeline: 2.87 s -> 2.80 s (-72.7 ms) vs main (d5b9601).
  • Timing is informational: shared GitHub runners are too variable for a hard timing budget.
Detailed phase timings vs `main (d5b9601)`
Phase Baseline Current Delta
extension.check 0.4 ms 7.4 ms +7.0 ms
project.resolve 0.2 ms 0.3 ms +0.1 ms
workspace.create 0.4 ms 0.4 ms 0.0 ms
host.prepare 445.1 ms 403.4 ms -41.7 ms
vercel.service-prefix.resolve 1.1 ms 1.1 ms 0.0 ms
nitro.create 477.3 ms 453.9 ms -23.4 ms
sandbox.prewarm 477.1 ms 496.3 ms +19.2 ms
nitro.cache.prepare 0.2 ms 0.2 ms 0.0 ms
nitro.prepare 0.5 ms 0.5 ms 0.0 ms
nitro.public-assets 0.6 ms 0.6 ms 0.0 ms
nitro.prerender 0.4 ms 0.3 ms -0.1 ms
nitro.bundle 1.32 s 1.30 s -18.7 ms
nitro.cache.write 0.3 ms 0.3 ms 0.0 ms
vercel.workflow-function.materialize 83.7 ms 70.4 ms -13.3 ms
agent-summary.emit 0.7 ms 0.6 ms -0.1 ms
connect-manifest.emit 0.3 ms 0.2 ms -0.1 ms
nitro.close 0.1 ms 0.1 ms 0.0 ms
output.publish 2.9 ms 2.5 ms -0.4 ms
workspace.remove 1.6 ms 1.7 ms +0.1 ms

@chadhietala
chadhietala marked this pull request as ready for review October 23, 2026 02:18
@chadhietala
chadhietala requested a review from a team as a code owner October 23, 2026 02:18
Comment on lines 347 to 352
// `write` queues the next stream write and returns the stamped event, so publishing
// does not wait for the stream round trip. The step drains the queue before it
// returns; see `runSessionStep` and `commitSessionStep`.
const emit = async (event: UnstampedMessageStreamEvent): Promise<MessageStreamEvent> => {
const stamped = await writer.write(await dispatcher.deliver(event));
recordPublishedEvent(ctx, stamped);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Queuing here also moves the turn's terminal events ahead of their durability, not just the pre-model ones. emit returns as soon as the write is queued, so for turn.completed the instrumentation wrapper (instrumentation/native-events.ts:71-100) publishes channel.delivery.completed and clears ActiveChannelDeliveriesKey, and the stream-event hooks and memory lifecycle in turn-event-handler.ts:88-103 run, all before the write settles. If that write rejects, nothing emits again before drain() at turn-step.ts:614. That drain sits outside the catch that calls failChannelDeliveries, so the step fails and retries after telemetry already reported the delivery as completed and the terminal hooks already fired.

Before this PR the same failure threw from emit inside runModelCallBatch, so no hooks ran and the delivery was reported as channel.delivery.failed. The last event of a turn is the one most likely to hit this, since no later write is around to fail fast.

Suggestion: keep the fire-and-forget behavior for pre-model and streaming events, and have emit await writer.drain() before it returns for turn-boundary events (turn.completed / turn.failed / turn.cancelled / input.requested / session.*). That only adds one round trip at the end of the turn, off the path this PR is optimizing, and it keeps "after its write" in turn-event-handler.ts:86 true for the events whose consumers report outcomes. The write doc at line 295 ("returns the event as written") also needs to say "queued".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Fixed in 8e580b9: emit now drains for boundary events (turn.* terminal/park, session.*, input.requested, authorization.required), so their write settles before hooks and instrumentation run and a failure surfaces inside runModelCallBatch again. Pre-model and streaming events keep the queued path. Also updated the write doc to say queued.

Comment on lines +458 to +461
if (pending.size >= MAX_PENDING_SESSION_EVENT_WRITES) {
await new Promise<void>((resolve) => capacityWaiters.add(resolve));
if (failure !== undefined) throw failure;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 64-write cap is soft when emitters run concurrently (for example the Promise.allSettled over approved calls in harness/hitl/approved-calls.ts:115). Once one write settles, settleCapacity resolves every waiter, and none of them checks pending.size again after waking. So N waiters behind a full queue all enqueue together, and the queue grows to 63 + N. A write that arrives in the same tick also gets ahead of the waiters that were queued before it.

A while loop fixes the bound: while (pending.size >= MAX_PENDING_SESSION_EVENT_WRITES) { await new Promise(...); if (failure !== undefined) throw failure; }. Waking one waiter per freed slot would fix it too. Also worth thinking about whether the Set, the waiter Set, and the settleAll loop can collapse into one FIFO array. WritableStream already settles writes in order and rejects every later write once it errors, so the oldest entry is the only one capacity has to wait on, and the newest entry is the only one drain has to wait on.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. write now holds a FIFO enqueue lock while it waits for capacity and re-checks in a while loop, so wakeups cannot overshoot the cap and a later event cannot overtake a waiting one. drain and release take the same lock. capacityWaiter is a single callback because only the lock holder can wait. I kept the pending Set instead of collapsing to the newest write; it keeps the failure and backpressure bookkeeping simple.

Comment on lines +612 to +614
// The queued events must be durable before the step commits; publishing only
// waited for each write to be queued.
await publisher.writer.drain();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test covers this drain, and it is the only thing that turns a late write failure into a step failure. release() in the finally also waits for pending writes, but it uses allSettled and drops the error. So deleting this line (or the one at 590, or the one in publishInSessionScope) would let the step commit successfully after a stream write rejected, and every current test would still pass. The new writer tests only call drain() directly.

Could you add a turn-step test where the session writable rejects the final write (turn.completed) after a delay, and assert that the step rejects? It should also check that the result isn't returned before a slow write settles. A one-line note on release() saying it is cleanup only and never surfaces write failures would make that contract explicit too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a turn-step test that publishes a non-boundary step.started through HandleEventKey, holds its write, asserts the step has not settled, then rejects the write and asserts the step rejects. Also added publisher tests that a pre-model event returns while queued and a boundary event waits. release() now documents that it is cleanup only and never reports write failures.

Publish session lifecycle events without waiting for each durable stream round trip. The step drains the queued writes before it commits, so event order and failure handling stay the same.

Signed-off-by: Chad Hietala <chad.hietala@vercel.com>

This branch was successfully deployed

2 active deployments
Preview – eve-docs — 8e580b9a Deployed Oct 23, 2026 by vercel[bot]
Preview – eve-pkg — 8e580b9a Deployed Oct 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant