Skip to content

fix(voice): preserve thinking pauses and Android dictation - #705

Open
ludovicfourrage wants to merge 2 commits into
Ark0N:masterfrom
ludovicfourrage:fix/voice-silence-timeout
Open

ludovicfourrage wants to merge 2 commits into
Ark0N:masterfrom
ludovicfourrage:fix/voice-silence-timeout

Conversation

@ludovicfourrage

@ludovicfourrage ludovicfourrage commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Dictation should survive a thinking pause and keep the final words when the microphone is stopped. Chrome on Android also needs different native recognition handling: its continuous mode can turn growing partial phrases into repeated final results, producing a much longer transcript than was spoken.

This change adds Stop after silence in Voice settings: provider default, 3/6/15/30 seconds, or Manual. Provider default retains the 6-second Claude / 3-second Deepgram and Web Speech deadlines, but finalized phrases no longer stop Deepgram or Web Speech immediately.

  • Claude interims replace the current utterance; endpoint finals append once with a space. Endpoints keep recording until finalization is requested. Stop preserves the final transcript, with a 5-second client backstop for a dead connection.
  • Reconnect cleanup preserves available text, resets the UI, and releases capture. Pending microphone acquisition and connecting sockets are cancelled safely, including acquisition that resolves after cleanup.
  • Android uses single-utterance native recognition and restarts within the configured silence deadline. Web Speech rebuilds each complete result snapshot, preserving intentional repetitions without duplicating corrections. Stop accepts late final results before inserting once.
  • Browser speech activity animates the microphone bars without opening a second microphone capture. Streaming providers retain their audio-level meter. Repeated empty browser runs have a short backoff and a two-retry limit.
  • Settings share the allowed timeout values; the wiki, architecture guidance, and changeset describe the new behavior. Manual's overall duration policy is unchanged.

Validation:

  • 115 tests passed across voice result/lifecycle, silence settings, insertion targets, relay/routes, settings structure, and test-suite partition coverage.
  • Three real Chromium DOM/input fixtures passed at 390, 820, and 1292 pixels, with simulated Android, Android desktop-site, and Windows recognition events. They check animated/idle bars, five spoken words inserted once, intentional repeated phrases, and no competing microphone request.
  • npm run typecheck, npm run lint, npm run check:frontend-syntax (46 files), scoped formatting checks, and npm run build passed.
  • The initial regressions reproduced before the fixes; an independent review found no remaining blocking issues.
  • The reporter confirmed Chrome/Windows dictation worked and subsequently confirmed the deployed Android fix works. Native Android confirmation applies to the deployed Android changes; the additional Claude lifecycle fixes are covered by automated tests, not a live Anthropic transcription session.

@Ark0N

Ark0N commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Thank you for this, and for the thorough description. This PR adds a Stop after silence setting (provider default, 3/6/15/30 s or Manual) so dictation survives thinking pauses. It also fixes a real bug: on master, tapping the mic during Claude dictation dropped the transcript, and with this change the final frame is kept.

Two things on the Claude path need fixing before merge:

  1. Claude still ends at the first pause, whatever the setting (src/web/public/voice-input.js:744). The relay opens the upstream stream with endpointing_ms=300 / utterance_end_ms=1000 (src/web/voice-stream.ts:90-91) and turns every TranscriptEndpoint into a final: true frame, whether or not the client asked to finalize (src/web/voice-stream.ts:254). The Claude onResult calls this.stop(true) on any final, so with 15 s, 30 s or Manual a pause of about a second still ends the recording. A vm harness with silenceTimeout: 0 shows isRecording === false right after the first final. Interims carry the running text of the current utterance only, and the Claude Code CLI's own voice client appends each final with a space. Please handle Claude the way your Deepgram branch does: append on a final, replace _pendingInterim on an interim, and end the recording on a final only once ClaudeVoiceProvider._finalized is set. Please also update docs/architecture-invariants.md:701 and the comment at voice-input.js:742, which both say the Claude path only replaces. Then add a test that drives onResult(text, true) under 15 s / Manual, asserts the recording continues, and checks that two utterances plus a stop insert both.

  2. An SSE reconnect during Claude dictation leaves the mic stuck on (src/web/public/voice-input.js:1361). handleInit calls VoiceInput.cleanup() on every reconnect (src/web/public/app.js:4946). cleanup() calls stop(), which now takes the new wait-for-final branch (voice-input.js:866), and ClaudeVoiceProvider._cleanup() then nulls the socket and _onEnd, so the recording never completes: isRecording stays true, the button stays red, and the next tap only resets it. Master leaves it idle. if (this.isRecording) this.stop(true); in cleanup() fixes it. Since stop() now inserts accumulated text, decide whether a reconnect should salvage or discard the partial transcript, and clear the buffers first if it discards. Please add a test for cleanup() in the middle of Claude dictation.

Smaller items, fine in the same push:

  • voice-input.js:866: while waiting for Claude's final, nothing on the client bounds the wait, and further taps are ignored. The server's 3 s finalize timeout only helps on a live link. A client backstop of a few seconds that calls stop(true), or a second tap that forces it, covers a dead link; the test at test/voice-silence-timeout.test.ts:119 changes with it.
  • voice-input.js:926: Web Speech finals are joined with no separator. Please add the same space guard your Deepgram branch (:792) and the restart branch (:995) use.
  • voice-input.js:990: the extended/Manual restart in _onWebSpeechEnd has no cap or backoff, unlike the _retryCount < 2 retry above it, so a browser that ends recognition immediately without a result restarts in a tight loop. A short backoff or a cap on consecutive empty restarts bounds it.
  • docs/wiki/Input-And-Voice.md:85 and the changeset: under Provider default, a finalized phrase no longer ends a Deepgram or Web Speech recording; it now runs until 3 s of silence. Please say so in a sentence.
  • Nit: [0, 3, 6, 15, 30] appears in voice-input.js:612 and settings-ui.js:788; one constant on VoiceInput keeps them together.

One question is open, and it is not one for you to change in this PR: Manual has no upper bound on the Deepgram and Web Speech paths (Claude streams still stop at the relay's 5 minute cap, src/config/voice.ts:40), so a forgotten tab on Manual keeps streaming to Deepgram on the user's key. Whether Manual needs a ceiling there, and how long, is not settled.

Once the two Claude-path fixes are in, this goes back through review.

@ludovicfourrage ludovicfourrage changed the title fix(voice): allow thinking pauses and manual-stop dictation fix(voice): preserve thinking pauses and Android dictation Oct 24, 2026
@ludovicfourrage

Copy link
Copy Markdown
Contributor Author

Updated this branch with both requested Claude fixes and the smaller review items:

  • Endpoint finals append per utterance and continue recording under extended/Manual settings; finalization preserves both utterances.
  • SSE cleanup salvages partial text and returns the mic to idle. It also cancels pending microphone acquisition and connecting sockets.
  • A 5-second provider finalization backstop covers manual and silence-triggered stopping without extending on repeated taps.
  • Web Speech separates phrases, bounds consecutive empty restarts, and shares timeout choices with settings. The wiki and changeset explicitly describe provider-default final-event behavior.

The update also includes the reporter-tested Chrome/Android correction for repeated transcripts, missing late final text, and speech activity bars. The reporter confirmed the deployed Android fix works. Manual's overall duration ceiling remains unchanged, as requested.

Validation: 115 focused tests, three Chromium browser fixtures, typecheck, lint, frontend syntax, scoped formatting, and build passed. Independent review found no blocking issues. The browser speech service is simulated in automation; native Android confirmation comes from the reporter, and the additional Claude fixes have automated coverage.

@Ark0N

Ark0N commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Thank you for the second round, and for covering every item from the first review with a test. This PR adds a Stop after silence setting (provider default, 3/6/15/30 s or Manual), keeps Claude dictation going across utterance endpoints until Stop, and reworks Web Speech so Chrome on Android no longer repeats growing phrases.

Two things need fixing before merge:

  1. npm test fails: 14 tests in test/settings-admin-only-ui.test.ts and test/tab-cli-logos-setting.test.ts. src/web/public/settings-ui.js:788 reads VoiceInput.SILENCE_TIMEOUT_SECONDS.includes(...), and both files load settings-ui.js with a VoiceInput stub that has no such field (test/settings-admin-only-ui.test.ts:148, test/tab-cli-logos-setting.test.ts:128), so openAppSettings() throws Cannot read properties of undefined (reading 'includes'). Adding SILENCE_TIMEOUT_SECONDS: [0, 3, 6, 15, 30] to both stubs makes all 44 tests in those files pass. Please run the full npm test before pushing: while the branch conflicts, GitHub runs no CI on it, so nothing else will catch this.

  2. 30 s and Manual still end after about three silent native runs on Web Speech (src/web/public/voice-input.js:1033-1038). _onWebSpeechEnd counts every run that ended without a result toward _retryCount > 2, however long that run lasted. Under 30 s or Manual, a phrase followed by a long pause produces one silent run per no-speech window of the engine (several seconds on Chrome), so the third one ends the recording at roughly 15 to 25 s, before the deadline, and Manual stops on its own. With silent runs of 6 s, both settings stop at 18 s of silence; test/voice-silence-timeout.test.ts:244 pins the same thing under Manual. That contradicts the setting's description (src/web/public/index.html:3155, "Manual keeps listening until you tap the mic again") and docs/wiki/Input-And-Voice.md:84. The restart branch also runs in continuous mode, so it is not Android-only. The cap is only needed for a recognizer that ends immediately. Please stamp the run start at both recognition.start() sites (_startWebSpeech and the restart closure), count an empty run toward the cap only when it ended within about a second of starting, and reset the counter otherwise, keeping the 250 ms backoff. Then add a test: 30 s and Manual, one phrase, silent runs ending every 6 s, still recording after the third, with 30 s ending at its deadline. The existing cap test calls _onWebSpeechEnd() without a start stamp, so its runs need stamping as instant.

Also in the same push:

  • Rebase onto master. The branch conflicts in CLAUDE.md: master added an Agent inbox paragraph directly above the Voice dictation paragraph. Keep both, with your Voice text. While the branch conflicts, GitHub runs no CI on it.
  • docs/wiki/Settings-Reference.md:176: the Voice paragraph lists the controls; please add Stop after silence.
  • src/web/public/voice-input.js:703-705: the maxTouchPoints > 0 && /Linux/ fallback also matches ChromeOS and Linux laptops with a touchscreen, so they get single-utterance mode too. It works there, with a restart per phrase. Please narrow it or say so in the comment.

Nits, optional:

  • src/web/public/voice-input.js:955 and :1423: _webSpeechStream is no longer assigned now that the level-meter capture is gone, so both teardown blocks are dead code.
  • test/voice-input-results.test.ts and test/voice-input.browser.test.ts use node:assert; the rest of the suite uses vitest expect.

The question from the first review is still open, and it is not one for you to change in this PR: Manual has no upper bound on the Deepgram and Web Speech paths, so a forgotten tab on Manual keeps streaming to Deepgram on the user's key. Whether Manual needs a ceiling there, and how long, is not settled.

Once the test stubs and the empty-run cap are fixed and the branch is rebased, this goes back through review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants