Audio
WaveRNN
WaveRNN vocoder plus Tacotron TTS in PyTorch: train your own neural speech synthesis on LJSpeech.
★ 2.2k starsMIT
View guide →Unofficial PyTorch implementation of VALL-E, the neural codec language model for zero-shot TTS
vall-e is an open-source audio project by enhuiz (enhuiz/vall-e on GitHub). It has 2,970 GitHub stars and 399 forks, is released under the MIT license, and was last updated on GitHub on 10 May 2023.
Understand this is a research implementation to train yourself: you will need your own audio/text data and a GPU setup DeepSpeed supports.
Clone with submodules, or pip install directly from GitHub as the README shows.
Install the package (tested under Python 3.10.7), put .wav + .normalized.txt files in a data folder, quantize them and generate phonemes.
Create AR/NAR configs, train both models, export checkpoints, then run the synthesis CLI with text and a reference audio path.
Implements the autoregressive model for the first quantizer and the non-autoregressive model for the remaining quantizers.
Built on Facebook's EnCodec for audio tokenization and decoding from tokens.
Data quantization (vall_e.emb.qnt), phoneme generation (vall_e.emb.g2p) and DeepSpeed-based trainers for both models.
After exporting AR and NAR checkpoints, synthesize speech from text plus a reference recording from the command line.
YAML configs per dataset with different model presets defined in the code.
A linked Google Colab notebook demonstrates the flow on a single utterance (the README notes it overfits and is not usable as a model).
Clone VALL-E with submodules as the README instructs.
git clone https://github.com/enhuiz/vall-e.git
cd vall-eAlternatively install directly from GitHub with pip (Python 3.10.7 tested; a DeepSpeed-supported GPU is needed for training).
pip install git+https://github.com/enhuiz/vall-eExport your trained AR/NAR checkpoints, then run the synthesis CLI with text, reference and output paths.
python -m vall_e <text> <ref_path> <out_path> --ar-ckpt zoo/ar.pt --nar-ckpt zoo/nar.ptenhuiz maintains this unofficial implementation; VALL-E itself is a Microsoft Research model described in the cited arXiv paper.
The code is MIT-licensed. Note the README's notice that EnCodec is CC-BY-NC 4.0, so non-commercial terms apply to the EnCodec parts.
No API key — it runs locally with PyTorch. But there is no pretrained checkpoint provided, so you must train the models yourself on GPU hardware.
Those synthesize speech immediately. This repo is a research training implementation of a specific paper, with no shipped pretrained model.
Clone with --recurse-submodules, install, prepare a dataset of .wav and .normalized.txt files, then follow the quantize → phoneme → train → export flow in the README.
WaveRNN vocoder plus Tacotron TTS in PyTorch: train your own neural speech synthesis on LJSpeech.
MetaVoice-1B: a 1.2B-parameter foundation model for emotional, zero-shot text-to-speech
1 min voice data can also be used to train a good TTS model! (few shot voice cloning)
IndexTTS: industrial-level, controllable zero-shot text-to-speech with voice cloning
Gradio WebUI for creators and developers, featuring key TTS (Edge-TTS, kokoro) and zero-shot Voice Cloning (E2 & F5-TTS, CosyVoice), with Whisper audio proce…
A tiny conversational TTS model with cloning, streaming, and CPU speed.
The most-starred other Audio projects in the LibreAI catalogue — the realistic shortlist if vall-e is not the right fit.
利用 AI 大模型和自动化工作流，根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.
Robust Speech Recognition via Large-Scale Weak Supervision
1 min voice data can also be used to train a good TTS model! (few shot voice cloning)
Clone a voice from 5 seconds of audio and generate arbitrary speech in real time
Port of OpenAI's Whisper model in C/C++
Free and open — grab the source, star the repo, and start creating.