Speech-to-Text

This chapter teaches you the gateway's transcription surface: how to declare speech models, how the interim and final roles work together, and how to use batch and Realtime transcription. Speech builds on local models, because speech models are provisioned and cached the same way.

Declare speech models

A speech-to-text model is a [[stt_model]] entry:

[[stt_model]]
name = "whisper-base-en"
role = "interim"
source = "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin"
sha256 = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
vram_gb = 1.0

Each entry has a name, a role of interim or final, a source, an optional sha256 pin, a vram_gb estimate, and an optional dominion binding. The interim role transcribes while a take is still recording. The final role crystallizes completed audio.

A profile may select at most one interim and one final STT model. A final model requires an interim partner. Interim-only is a supported degraded mode. You can restore a built-in recommended pair at any time: whisper-base-en for interim and whisper-small-en for final, both with canonical whisper.cpp URLs and SHA-256 pins.

Tune push-to-talk capture

Tune the pipeline in the optional [stt] section:

[stt]
window_seconds = 15
interval_ms = 500
vocabulary = ["MCP", "GGUF", "Lua"]

The window_seconds key sets the seconds of trailing audio transcribed per pass (default 15), and interval_ms sets the milliseconds between passes (default 500). Each must be at least 1; a zero value fails startup. The vocabulary lists domain terms that bias both transcription workers toward those terms. An empty list disables biasing. A vocabulary that exceeds the model's prompt budget is truncated, and a warning is logged. The whisper_backend key chooses the whisper.cpp runtime build: auto (the default), cpu, or cuda, as The runtime describes.

Version 2 accepts only the canonical [stt] section. Legacy [workshop.stt] input is rejected as an unknown workshop field whether it appears alone or beside [stt], and saved configuration uses only [stt].

Batch transcription

With the default-on stt feature the gateway serves OpenAI-compatible audio transcription at POST /v1/audio/transcriptions. The multipart form accepts file, model, language, prompt, temperature, response_format, and the repeated field timestamp_granularities[].

Uploads are capped at 25 MiB; an over-limit upload is answered with "audio file exceeds the 25 MiB limit". Only 16 kHz mono WAV audio is accepted. Other sample rates or channel counts are rejected with a message naming what was received.

Two response shapes are offered. The json shape returns text only. The verbose_json shape returns task, language, duration, text, segments, and words. Segment timestamps are on by default, and word timestamps are always empty. A transcription request for a model not loaded in the active profile is rejected as an unknown model. A caller-supplied temperature must be a finite non-negative number. The prompt hint is accepted but ignored by the current English whisper workers.

The runtime

Speech-to-text runs on a separately pinned whisper.cpp library bundle, b4938. A library that does not match the pinned layout fails to load, and only 64-bit targets are supported. The gateway serves first and loads speech second: after the listener is bound, the queued boot command downloads and verifies the model artifacts and the runtime into the configured cache directory, with progress on the status and progress endpoints. Each model file is prewarmed and then loaded, with progress per model. Speech routes answer as unavailable until the load completes, and the model catalog advertises speech models only once the speech engine is ready.

Windows x86-64 and Linux x86-64 each have two builds of the runtime, one for the CPU and one for CUDA. The [stt].whisper_backend setting chooses between them: auto, cpu, or cuda. The default auto downloads the CUDA build when nvidia-smi reports an NVIDIA GPU, and the CPU build when it reports none or cannot run. On Linux the CUDA build needs NVIDIA driver 570 or later, the floor for CUDA 12.8, so auto also takes the CPU build when nvidia-smi reports an older driver or a version it cannot read. The Linux CUDA build also needs a C++ runtime from GCC 12 or later: auto takes it only where the machine's libstdc++.so.6 defines GLIBCXX_3.4.30, so RHEL 9, its rebuilds, and Amazon Linux 2023 keep the CPU build, and an explicit cuda there fails to load. On both platforms nvidia-smi ignores CUDA_VISIBLE_DEVICES, so auto reads it as well: a value that hides every GPU from CUDA, such as -1, an empty value, or a device index past the last GPU, counts as no GPU and takes the CPU build. On Linux a first entry that is a GPU- or MIG- identifier counts as a visible GPU without being matched against the machine's GPUs, so an identifier that names no GPU, or an abbreviated one that names several, still takes the CUDA build. On Windows auto counts a first entry that is not a device index, a GPU- or MIG- identifier included, as hiding every GPU and takes the CPU build; set cuda to restore GPU decoding for a valid identifier. The cpu and cuda settings download their own build without probing. An explicit cuda is honored below the Linux floor, and loading that build can end the gateway. On Windows with every GPU hidden, an explicit cuda decodes on the CPU and ends the gateway at a graceful stop. Every other platform has one build and ignores the setting.

On Windows the CUDA build needs NVIDIA driver 580 or later, the floor for CUDA 13, and it carries native code only for GPUs of compute capability 8.6, 8.9, 12.0, and 12.1. So auto takes the CPU build when nvidia-smi reports an older driver, a version it cannot read, or any GPU of another compute capability. An explicit cuda is honored on such a machine, with two outcomes: a GPU without native code under a driver older than CUDA 13.3's ends the gateway at the first transcription, and where CUDA finds no usable device the build decodes on the CPU and a graceful stop then crashes the gateway instead of exiting cleanly. Set auto or cpu to recover from either. Where the driver can compile it, a GPU without native code, such as a Linux GPU at compute capability 7.5, 8.0, or 9.0, compiles the build's PTX at its first decode, which took about 20 seconds with an empty driver JIT cache.

Every x86-64 build, whatever the setting, uses the SSE4.2, AVX, AVX2, BMI2, FMA, and F16C instructions. On a CPU that lacks any of them, provisioning the whisper library fails with an error that names the build, the required extensions, and the missing ones, and speech stays unavailable while the gateway keeps serving.

STT startup failures are named by stage: opening the artifact store, provisioning the whisper library, provisioning a named model, a missing interim partner, an unsupported role, or speech engine load. Library load failures name the failing path or symbol in the logs. A failed boot load never stops the gateway and is never retried in-process: speech stays unavailable, the failed boot command shows on the queue and progress surfaces, and a restart is the recovery. A stop during the speech load cancels the whisper library and speech model downloads at their next chunk, and the next start resumes them; an extraction already under way finishes first. A graceful stop aborts running transcriptions after their current encoder pass or decoder step, their requests fail, and speech retires within the gateway's existing shutdown bounds. A stop while a speech model is still loading can outlast those bounds, and the gateway then exits without retiring speech.

How a take is transcribed

A recorded take is split into speech segments at silence boundaries. A segment closes only after 2 seconds of trailing silence, so sentence-internal pauses survive. Speech bursts shorter than 250 ms are discarded as clicks. Audio quieter than -60 dBFS is treated as silence and never sent to the model, and fragments shorter than half a second are gated out.

With a final model configured, completed speech segments are re-transcribed in the background while the take still records, and each segment's text is reported as it finishes. Without a final model, the stop falls back to the interim model. Silent or very short fragments are skipped so the model does not invent text for them. Transcription is pinned to English, and translation is disabled.

Realtime transcription

The gateway serves authenticated Realtime transcription at WS /v1/realtime?intent=transcription. The query is exact: missing, duplicate, malformed, unsupported, or additional parameters are rejected before upgrade. Native clients may omit Origin; browser clients must send an HTTP loopback Origin.

The server creates a transcription session for the logical model realtime-transcribe. Clients may send session.update, input_audio_buffer.append, input_audio_buffer.clear, and input_audio_buffer.commit. Audio appends are canonical Base64 containing signed little-endian mono PCM16 at 24 kHz. The gateway preserves an odd trailing byte across appends, continuously resamples to 16 kHz, flushes the resampler on commit, and resets the whole input on clear.

Only null noise reduction and turn detection are accepted. Session updates may change the transcription prompt and negotiate the Runvil extension item.input_audio_transcription.hypothesis. Standard clients receive OpenAI-shaped session, item, transcription delta, completed, failed, and error events. Extension clients also receive revisioned replacement snapshots with the complete transcript and its finalized, agreed, and tentative regions; completion remains authoritative.

A continuous Realtime recording remains one provisional item and one logical take for arbitrary duration. Continuous speech forces an accurate boundary every 10 seconds. Each successor repeats the preceding 8 seconds for text reconciliation, so every accurate decode contains at most 18 seconds. Finalized text and exact lifetime duration survive source-buffer compaction, and every hypothesis remains a complete replacement snapshot for that same item.

The 30-second limit is retained ownership, not recording duration. It includes resident, queued, and actively decoding 16 kHz PCM. Arbitrary-duration capture therefore requires steady-state final throughput at least equal to capture. If decoding falls behind until that retained budget is exhausted, the append receives too_much_unfinalized_audio; previously accepted audio and text remain valid and may still be committed. One append decodes to at most 15 MiB, committed audio must be at least 100 ms, one connection may have four committed items finalizing concurrently, and the service admits at most eight Realtime sessions. Queue and capacity overloads return explicit errors instead of waiting without limit.

The desktop Workshop exposes the same /v1/realtime path on its own origin. Its server authenticates the fixed upstream target and relays payloads without parsing them, so the webview never receives the gateway credential.

Speech loads exactly once per process, from the profile active at boot. Switching the active profile or applying a new configuration persists a changed speech selection but never loads, reloads, or unloads the running speech engine; the new selection takes effect on the next start. The configuration UI raises a restart toast when an apply changes the speech tuning, the speech model catalog, or the active profile's speech membership.