Audio
fastrtc
FastRTC turns any Python function into a real-time audio or video stream over WebRTC or WebSockets
★ 4.6k starsMIT
View guide →GAN vocoder that turns mel-spectrograms into high-fidelity speech audio, fast enough for real-time use.
hifi-gan is an open-source audio project by jik876 (jik876/hifi-gan on GitHub). It has 2,373 GitHub stars and 560 forks, is released under the MIT license, and was last updated on GitHub on 27 July 2024.
Decide whether you need inference with a pretrained checkpoint, fine-tuning on your own voice data, or training from scratch.
Clone the repository and install the Python requirements; for training, download the LJ Speech dataset and place its wavs as documented.
Download a pretrained model folder that matches your use case, such as a universal model or one fine-tuned for Tacotron2 mel-spectrograms.
Run inference from wav files or mel-spectrograms, or start training with one of the bundled V1, V2, or V3 configurations.
A GAN generator designed around periodic patterns in audio, producing 22.05 kHz speech the paper reports as close to human quality in listening tests.
Checkpoints for LJSpeech, VCTK, and a universal model, including versions fine-tuned on Tacotron2-generated mel-spectrograms.
inference.py reconstructs audio from wav-derived features, while inference_e2e.py synthesises speech directly from mel-spectrogram files for end-to-end TTS.
Generate mel-spectrograms with a model such as Tacotron2, place them in the fine-tuning dataset folder, and continue training with the fine-tuning flag.
Bundled config_v1, config_v2, and config_v3 JSON files trade model size and speed against quality.
The universal model ships with discriminator weights intended as a starting point for adapting to new datasets.
Python 3.6 or newer is required; install the dependencies from requirements.txt.
git clone https://github.com/jik876/hifi-gan.git
cd hifi-gan
pip install -r requirements.txtTrain with a bundled config, or fine-tune a checkpoint on your own mel-spectrograms.
python train.py --config config_v1.json
python train.py --fine_tuning True --config config_v1.jsonPut wav files in test_files (or mel files in test_mel_files) and generate audio with a checkpoint.
python inference.py --checkpoint_file [generator checkpoint file path]
python inference_e2e.py --checkpoint_file [generator checkpoint file path]It is maintained by jik876 (Jungil Kong) and collaborators, as the official open-source release accompanying the HiFi-GAN paper.
Yes. It is MIT-licensed, including the provided pretrained models, so research and commercial use are permitted.
No. HiFi-GAN runs entirely locally in PyTorch; there are no API keys or hosted services involved.
WaveGlow and MelGAN are earlier neural vocoders the authors drew on; HiFi-GAN focuses on modelling periodic audio patterns to reach higher fidelity at much faster than real-time generation.
Clone the repo, pip install -r requirements.txt, download a pretrained model, and run inference.py or inference_e2e.py with its checkpoint.
FastRTC turns any Python function into a real-time audio or video stream over WebRTC or WebSockets
Clone a voice from 5 seconds of audio and generate arbitrary speech in real time
On-device streaming speech-to-text engine for real-time apps.
A macOS voice assistant that turns speech into actions and notes.
MOSS-TTS-Nano: a 0.1B open multilingual speech model that clones voices and streams in real time on CPU
gTTS — Python library and CLI that turns text into spoken mp3 via Google Translate's speech API
The most-starred other Audio projects in the LibreAI catalogue — the realistic shortlist if hifi-gan is not the right fit.
利用 AI 大模型和自动化工作流，根据主题或关键词一键生成高清短视频。Generate HD short videos from a topic or keyword with an automated AI workflow.
Robust Speech Recognition via Large-Scale Weak Supervision
1 min voice data can also be used to train a good TTS model! (few shot voice cloning)
Clone a voice from 5 seconds of audio and generate arbitrary speech in real time
Port of OpenAI's Whisper model in C/C++
Free and open — grab the source, star the repo, and start creating.