Audio
VoiceMem
VoiceMem — streaming dual-brain memory for voice agents: a left brain for facts, a right brain for emotion
★ 2.4k starsApache-2.0
View guide →Self-hosted backend that gives ESP32 'Xiaozhi' devices streaming voice chat, memory and tool control.
xiaozhi-esp32-server is an open-source audio project by xinnan-tech (xinnan-tech/xiaozhi-esp32-server on GitHub). It has 10,756 GitHub stars and 3,670 forks, is released under the MIT license, and was last updated on GitHub on 8 October 2026.
Flash or buy a Xiaozhi ESP32 device and confirm it works against a public test backend, so you know the hardware side is fine before self-hosting.
Pick the minimal install (single agent, config-file storage, no database) or the full-module install (multi-user, multi-agent, web control console, database) - each is offered as Docker or source deployment in the docs.
Select ASR, LLM, vision and TTS providers in the config; the documented all-free starter setup uses local FunASR, EdgeTTS and free-tier cloud models.
Set your firmware to your server's OTA and WebSocket endpoints, then watch it answer, recognize speakers and run tools through the control console.
Streaming ASR, streaming TTS and voice-activity detection give low-latency, interruptible conversations in multiple languages.
Multiple users can register voiceprints; the server identifies who is speaking in parallel with recognition and passes that to the LLM for personalized replies.
LLMs connect via OpenAI-compatible APIs, Ollama, Dify, FastGPT, Coze, Xinference or Home Assistant; ASR, TTS and vision models include local options like FunASR, FishSpeech and GPT-SoVITS as well as major cloud providers.
Supports client IoT protocols, client and server MCP, and MCP access points, and can push MCP commands from the console down to ESP32 devices over MQTT.
Choose local short-term memory, mem0ai or PowerMem for conversation memory, and plug in a RAGFlow knowledge base the model can consult before answering.
The full-module install adds a web admin for users, system configuration and devices (in Simplified/Traditional Chinese and English), plus performance and audio test tools.
Get the server source; deployment itself is covered by the project's step-by-step deployment docs.
git clone https://github.com/xinnan-tech/xiaozhi-esp32-server.git
cd xiaozhi-esp32-serverFollow the deployment document for your tier - Docker or source, minimal or full-module - linked from the README's deployment section.
# see the deployment docs linked from the README for the exact commandsPoint your ESP32 firmware at the server's OTA/WebSocket endpoints and use the bundled test tools to check ASR, LLM and TTS latency.
# verify with the performance tester and audio test tool described in the READMEThe xinnan-tech team maintains it; the project is spearheaded by Professor Siyuan Liu's team at South China University of Technology and a large contributor community.
Yes. The server is MIT-licensed, and the README documents an all-free configuration (local FunASR recognition, EdgeTTS, free-tier models) for home use.
Not necessarily. You can run every module with local or free-tier components; paid cloud ASR, LLM or TTS providers are optional upgrades for speed and quality.
The public service is zero-setup but shared and third-party-run; self-hosting gives you your own data, your own model choices, multi-user management and no dependence on someone else's server.
Clone the repo, follow the deployment doc for the minimal Docker install, then point one ESP32 device at your server's WebSocket endpoint.
VoiceMem — streaming dual-brain memory for voice agents: a left brain for facts, a right brain for emotion
Self-host Chatterbox TTS with web UI, API, and voice cloning.
Open source voice AI platform. Self-hosted alternative to Vapi and Retell. On Prem, BYOK across Speech to Speech or LLM/STT/TTS, with a visual workflow build…
Self-hosted AI video analysis: summaries, chapters, transcripts.
Browse WhatsApp chat exports offline, with AI voice transcription.
CosyVoice: multilingual voice generation with zero-shot cloning, streaming and fine-tuning
The most-starred other Audio projects in the LibreAI catalogue — the realistic shortlist if xiaozhi-esp32-server is not the right fit.
利用 AI 大模型和自动化工作流，根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.
Robust Speech Recognition via Large-Scale Weak Supervision
1 min voice data can also be used to train a good TTS model! (few shot voice cloning)
Clone a voice from 5 seconds of audio and generate arbitrary speech in real time
Port of OpenAI's Whisper model in C/C++
Free and open — grab the source, star the repo, and start creating.