Cue AI Uses Gemma 4 for Faster Voice Dictation: 44% Latency Drop, 30% More Usage

Cue AI, a voice-activated desktop agent that executes dictation and agentic tasks via hotkey, replaced its cloud-based text polish step with Google DeepMind's Gemma 4 E4B running locally via Ollama. The switch cut median polish latency from 876ms to 488ms — a 44% reduction — and pushed the round-trip (speak, polish, insert) comfortably under 500ms, the perceptual budget for feeling faster than typing.

Key Results

• Latency: Polish step dropped from 876ms to 488ms median (measured on Apple Silicon M-series via Ollama, 227 real voice samples including mixed-language input).
• Usage: Dictation per user increased ~30% in the four weeks after the default switch. Users who previously dictated short messages began dictating longer ones, and voice-mode adoption grew for time-sensitive work.
• Cost: Running Gemma 4 E4B on-device drove marginal inference costs to zero, allowing unlimited dictation on the free tier without caps or paywall.

How It Works

The polish pipeline is intentionally simple: audio is transcribed via cloud STT, then sent to Gemma 4 E4B with a compact ~400-token system prompt that enforces formatting rules — restoring punctuation, segmenting sentences, removing filler words, correcting homophones, and adapting to the active input field (e.g., no trailing period for short commands, full punctuation for email). The polished text is pasted via native OS APIs.

Cue's deployment uses the base model with prompt engineering only. Active-app context (application name, field type, placeholder text) is injected into the prompt so the model knows whether the user is composing a Slack message, an email, or a terminal command. The cloud path remains as a fallback: if Ollama is not running, Cue routes to a cloud model without interruption.

Why Gemma 4 Won

The team originally planned to use Gemma only as an offline fallback, assuming a larger cloud model would provide better accuracy. Benchmarks on 227 real voice samples showed that Gemma 4 E4B has the right capacity to format and correct without over-editing — preserving the user's natural speech instead of smoothing into formal prose. This inversion — Gemma local as default, cloud as fallback — is now the operational backbone of every Cue dictation.