boxBot Open-Source Smart Speaker: Claude + Hailo AI

A developer known as FunScore645 built boxBot, a smart speaker that uses Claude for agent-driven hardware control instead of rigid rules. The project is open-sourced at github.com/dv-hart/boxbot and is still a work in progress but already useful for daily family tasks.

Hardware and Build

The hardware stack includes:

Raspberry Pi (model not specified)
Hailo AI accelerator (for edge inference, presumably)
Respeaker microphone array
Raspberry Pi camera
Waveshare display
Speakers

The enclosure is handcrafted from walnut, with passive cooling—the developer used Claude to calculate thermal loads and dissipation rates for the design.

Agent-Driven Architecture

Instead of complex conversation rules, the agent (likely Claude-based) controls the hardware directly through a custom Python SDK. Key capabilities:

Dynamic muting: The agent decides when to mute the mic to ignore background noise or side conversations.
Barge-in handling: Agent-driven logic manages interruptions without explicit rules.
Multi-channel response: Can reply via voice or WhatsApp, chosen by the agent.
Widget-based display: The screen uses a JSON widget framework—the agent reads and manipulates widgets via JSON without screenshots.

Skill System and Flexibility

The agent uses a small set of tools, with most logic routed through Python scripts and the boxBot SDK. Skills are defined in Markdown files. Example: when the developer added a family calendar event, his wife wanted a text notification. He added one line in a .md file, and the agent updated the calendar skill to send a WhatsApp message automatically on new events.

Memory and Security

The developer mentions a custom memory system and security considerations (camera+mic in a living room) but deferred details to comments. The agent is tightly coupled with hardware for security.