Developer Achieves Sub-Second STT/TTS Latency with Local Whisper and Coqui-TTS Servers
A developer has shared open-source server implementations that achieve sub-second latency for speech-to-text and text-to-speech in local AI agents, eliminating the conversational lag typically associated with cloud-based solutions.
Performance Benchmarks
The implementation achieves:
- ~0.2 seconds latency for speech-to-text (STT)
- ~250ms latency for text-to-speech (TTS)
This represents a significant improvement over the 2-3 second wait times mentioned as the previous bottleneck.
Technical Implementation
STT Server
- Built using Whisper large-v3-turbo
- Custom bridge implementation
- Hybrid thread-managed GPU architecture for concurrency without VRAM choking
TTS Server
- Uses Coqui-TTS running on a local server
- OpenAI-compatible API
- Optimized for low-latency synthesis
- Includes cloned Paul Bettany/Jarvis voice
Hardware Requirements
- Dedicated node with NVIDIA RTX GPU
- GPU acceleration is mandatory for these speeds
Open-Sourced Components
The developer has released two GitHub repositories:
These include server implementations and OpenClaw integration scripts for building local agents.
Results
The agent now exhibits truly conversational behavior with:
- Correct interruption handling
- Almost instant responses
- Zero audio data sent to external APIs
The developer is available to answer questions about server setup, VRAM management, and integration into other AI projects.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Exploring Clawe: Open-source Multi-agent Coordination System
Clawe is an open-source tool allowing for efficient multi-agent coordination, offering features like scheduling, task management, and real-time notifications.
Approval Boundary Tool for Claude Code Repository Work
A developer built an approval boundary tool that adds a review step before local execution when using Claude Code for repository work. The tool follows a loop: see the plan first, approve once, let the run happen locally, and keep proof afterward.
Adeu v1.4: Open-Source MCP for Track Changes in DOCX
Adeu v1.4 surgically injects native OOXML redlines into DOCX files, preserving formatting, numbering, and layouts. Adds footnotes/endnotes inline editing, defined term linting, cross-reference maps, and multi-level list round-tripping.
Wakehook: Trigger OpenClaw Morning Automations on Actual Wake Time, Not Cron
Wakehook reads Google Health/Fitbit sleep data and POSTs a user.awake event to OpenClaw when you actually wake up. Self-hostable, poll-based, no public URL needed.