Audio
xiaozhi-esp32-server
Self-hosted backend that gives ESP32 'Xiaozhi' devices streaming voice chat, memory and tool control.
★ 10.8k starsMIT
View guide →VoiceMem — streaming dual-brain memory for voice agents: a left brain for facts, a right brain for emotion
VoiceMem is an open-source audio project by xzf-thu (xzf-thu/VoiceMem on GitHub). It has 2,402 GitHub stars and 192 forks, is released under the Apache-2.0 license, and was last updated on GitHub on 14 September 2026.
Decide your mode: full dual-brain (normal), leftbrain_only for factual memory, offline memory engine, or the streaming interface for a live voice agent.
Clone the repo and pip install voicemem (add the [slm] extra if you want the fine-tuned reply model).
Download the default model bundle from Hugging Face with hf download, and set your OpenAI API key used for fact extraction at write time.
Warm up the engine, ingest an audio file or a fact as text, then search it ('what are my dietary restrictions?') and read the left-brain and right-brain results.
Facts live in the left brain (schema/entity-organized); personality, emotion and relationship information lives in the right brain with cross-entity nodes, maintained jointly.
Audio segmentation, transcription, memory extraction and writing happen while the user is still speaking; queries route, rank, then inject only the Top-K memories into context.
Built-in components cover ASR, speaker voiceprint, scene and emotion perception and local embeddings — it can remember speech, speakers, sound events and multi-person conversations from real audio.
The architecture is fully decoupled: every component, including the underlying memory engine, can be replaced, and you can plug VoiceMem into your own voice model with a simple reply callback.
Open VoiceMem model families (based on Qwen2.5-Omni, Qwen3-Omni and Step-Audio2-Mini), the ChatMem-400K dataset, and adapter fine-tuning code are included.
The benchmark pipeline (LoCoMo and PersonaMem evaluations) is fully open and reproducible with a single command, including an --inspect mode that costs no API calls.
Clone VoiceMem and install the memory system exactly as the quick start shows (Python 3.10+).
git clone https://github.com/xzf-thu/VoiceMem.git
cd VoiceMemFetch the bundled local models (ASR/speaker/scene/emotion/embedding) from Hugging Face.
pip install -U huggingface_hub
hf download zhifeixie/VoiceMem_Default_Models_Env --local-dir ./modelsCreate the engine with your key, warm it up, ingest a fact or audio, and query both brains.
from voicemem import VoiceMem
vm = VoiceMem(mode="normal", openai_key="YOUR_KEY", top_k=5)
vm.warmup()
vm.ingest(audio="assets/input.wav")
result = vm.search("What are my dietary restrictions?")Zhifei Xie and co-authors (xzf-thu) maintain it, with a technical report on arXiv and model/dataset releases on Hugging Face.
Yes, Apache-2.0 licensed; the README pledges the repo will remain fully open source. Running it uses local models plus an LLM API for write-time extraction.
The examples use an OpenAI API key — the README notes it is used only when writing memories (fact extraction); memory retrieval runs entirely locally.
Those are general text/agent memory systems. VoiceMem is purpose-built for voice: streaming audio ingest with speaker/emotion perception, and a separate emotional 'right brain' alongside factual memory.
pip install voicemem, hf download the default models, then follow the offline-engine example: warmup, ingest, search. Or run the interactive demo with python web/run.py on port 8787.
Self-hosted backend that gives ESP32 'Xiaozhi' devices streaming voice chat, memory and tool control.
Open-source voice creation for desktop: local TTS, cloning, emotion.
AgentCall lets AI Agents join meetings with voice
CosyVoice: multilingual voice generation with zero-shot cloning, streaming and fine-tuning
Alibaba's open voice models: expressive, streaming-capable speech with free-form voice design.
MOSS-TTS family: open text-to-speech models for voice cloning, dialogue, voice design, and real-time agents
The most-starred other Audio projects in the LibreAI catalogue — the realistic shortlist if VoiceMem is not the right fit.
利用 AI 大模型和自动化工作流，根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.
Robust Speech Recognition via Large-Scale Weak Supervision
1 min voice data can also be used to train a good TTS model! (few shot voice cloning)
Clone a voice from 5 seconds of audio and generate arbitrary speech in real time
Port of OpenAI's Whisper model in C/C++
Free and open — grab the source, star the repo, and start creating.