Audio
xiaozhi-esp32-server
Self-hosted backend that gives ESP32 'Xiaozhi' devices streaming voice chat, memory and tool control.
★ 10,8 Tsd. SterneMIT
Anleitung ansehen →VoiceMem ist ein Open-Source-Projekt aus der Kategorie Audio von xzf-thu (xzf-thu/VoiceMem auf GitHub). Es hat 2.402 Sterne auf GitHub und 192 Forks, wird unter der Lizenz Apache-2.0 veröffentlicht und wurde zuletzt am 14. September 2026 auf GitHub aktualisiert.
VoiceMem — streaming dual-brain memory for voice agents: a left brain for facts, a right brain for emotion
Decide your mode: full dual-brain (normal), leftbrain_only for factual memory, offline memory engine, or the streaming interface for a live voice agent.
Clone the repo and pip install voicemem (add the [slm] extra if you want the fine-tuned reply model).
Download the default model bundle from Hugging Face with hf download, and set your OpenAI API key used for fact extraction at write time.
Warm up the engine, ingest an audio file or a fact as text, then search it ('what are my dietary restrictions?') and read the left-brain and right-brain results.
Facts live in the left brain (schema/entity-organized); personality, emotion and relationship information lives in the right brain with cross-entity nodes, maintained jointly.
Audio segmentation, transcription, memory extraction and writing happen while the user is still speaking; queries route, rank, then inject only the Top-K memories into context.
Built-in components cover ASR, speaker voiceprint, scene and emotion perception and local embeddings — it can remember speech, speakers, sound events and multi-person conversations from real audio.
The architecture is fully decoupled: every component, including the underlying memory engine, can be replaced, and you can plug VoiceMem into your own voice model with a simple reply callback.
Open VoiceMem model families (based on Qwen2.5-Omni, Qwen3-Omni and Step-Audio2-Mini), the ChatMem-400K dataset, and adapter fine-tuning code are included.
The benchmark pipeline (LoCoMo and PersonaMem evaluations) is fully open and reproducible with a single command, including an --inspect mode that costs no API calls.
Clone VoiceMem and install the memory system exactly as the quick start shows (Python 3.10+).
git clone https://github.com/xzf-thu/VoiceMem.git
cd VoiceMemFetch the bundled local models (ASR/speaker/scene/emotion/embedding) from Hugging Face.
pip install -U huggingface_hub
hf download zhifeixie/VoiceMem_Default_Models_Env --local-dir ./modelsCreate the engine with your key, warm it up, ingest a fact or audio, and query both brains.
from voicemem import VoiceMem
vm = VoiceMem(mode="normal", openai_key="YOUR_KEY", top_k=5)
vm.warmup()
vm.ingest(audio="assets/input.wav")
result = vm.search("What are my dietary restrictions?")Zhifei Xie and co-authors (xzf-thu) maintain it, with a technical report on arXiv and model/dataset releases on Hugging Face.
Yes, Apache-2.0 licensed; the README pledges the repo will remain fully open source. Running it uses local models plus an LLM API for write-time extraction.
The examples use an OpenAI API key — the README notes it is used only when writing memories (fact extraction); memory retrieval runs entirely locally.
Those are general text/agent memory systems. VoiceMem is purpose-built for voice: streaming audio ingest with speaker/emotion perception, and a separate emotional 'right brain' alongside factual memory.
pip install voicemem, hf download the default models, then follow the offline-engine example: warmup, ingest, search. Or run the interactive demo with python web/run.py on port 8787.
Self-hosted backend that gives ESP32 'Xiaozhi' devices streaming voice chat, memory and tool control.
Open-source voice creation for desktop: local TTS, cloning, emotion.
AgentCall lets AI Agents join meetings with voice
CosyVoice: multilingual voice generation with zero-shot cloning, streaming and fine-tuning
Alibaba's open voice models: expressive, streaming-capable speech with free-form voice design.
MOSS-TTS family: open text-to-speech models for voice cloning, dialogue, voice design, and real-time agents
Die anderen Projekte der Kategorie Audio mit den meisten Sternen im LibreAI-Katalog — die realistische Auswahl, falls VoiceMem nicht das Richtige ist.
利用 AI 大模型和自动化工作流，根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.
Robust Speech Recognition via Large-Scale Weak Supervision
1 min voice data can also be used to train a good TTS model! (few shot voice cloning)
Clone a voice from 5 seconds of audio and generate arbitrary speech in real time
Port of OpenAI's Whisper model in C/C++
Verwandte Projekte der Kategorie Audio im Katalog, die VoiceMem bei den Sternen am nächsten stehen — zum Weiterstöbern auf beiden Seiten.
Offline text-to-speech for Python — uses your operating system's own voices, no internet needed
GAN vocoder that turns mel-spectrograms into high-fidelity speech audio, fast enough for real-time use.
MaryTTS — open-source, multilingual text-to-speech platform written in pure Java
ToucanTTS: fast, controllable text-to-speech for over 7,000 languages, from Stuttgart's IMS.
gTTS — Python library and CLI that turns text into spoken mp3 via Google Translate's speech API
WaveRNN vocoder plus Tacotron TTS in PyTorch: train your own neural speech synthesis on LJSpeech.
Kostenlos und offen — hol dir den Quellcode, gib dem Repository einen Stern und leg los.