Local LLM
vLLM
A high-throughput and memory-efficient inference and serving engine for LLMs
★ 93.1k starsApache-2.0
View guide →vLLM-Omni — efficient inference for omni-modality models: image, audio, video, diffusion and policies.
vllm-omni is an open-source local llm project by vllm-project (vllm-project/vllm-omni on GitHub). It has 7,085 GitHub stars and 1,871 forks, is released under the Apache-2.0 license, and was last updated on GitHub on 8 October 2026.
Pick the model family you will serve (omni, TTS, diffusion or robot policy) and confirm your GPU capacity against that model's requirements.
Clone vllm-project/vllm-omni from GitHub (or use the install path in the run steps below) and skim its README for the parts relevant to your goal.
Follow the vLLM-Omni documentation installation guide for your CUDA environment, or use the nightly CUDA Docker images built from main.
Serve one supported model from Hugging Face and call it through the OpenAI-compatible API with a multimodal request.
One framework serves models across text, image, audio, video and action data, with heterogeneous outputs.
Extends vLLM's AR strengths to Diffusion Transformers and other parallel-generation models.
Stage execution overlaps for throughput, with full disaggregation via OmniConnector and dynamic resource allocation across stages.
State-of-the-art autoregressive support through vLLM's efficient KV-cache management, plus streaming outputs.
Tensor, pipeline, data and expert parallelism for serving large models across GPUs.
Serves through an OpenAI-compatible API server, with seamless integration for popular Hugging Face models.
Get the source from the official GitHub repository for vllm-omni.
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omniInstall following the official installation documentation for your environment, as the README's getting-started links direct.
# Follow https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/ — see READMEStart the OpenAI-compatible server for a supported model and send it a request, per the docs quickstart and deployment recipes.
# Launch a documented quickstart model server, then call its OpenAI-compatible endpoint — see READMEThe vllm-project, through the vllm-project/vllm-omni repository, with docs at docs.vllm.ai/projects/vllm-omni.
Yes — the project is open source under the Apache-2.0 license as recorded for this listing. See the license terms for any conditions on redistribution.
No API key is needed — it is self-hosted inference software under Apache-2.0; your costs are GPU hardware and the models you download.
Vanilla vLLM serves autoregressive text models superbly. vLLM-Omni keeps that core and adds omni-modality and diffusion models, heterogeneous outputs and staged pipelines — choose it when your models see, hear, generate media or act, not just chat.
Clone the repository (or use the documented installer), follow the install and configure steps on this page, and run the smallest real task first — serve one supported model from Hugging Face and call it through the OpenAI-compatible API with a multimodal request.
A high-throughput and memory-efficient inference and serving engine for LLMs
TokenSpeed — a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in productio…
Run and fine-tune LLMs and diffusion models locally — faster training, smaller memory footprint.
DeepSeek-Coder-V2 — DeepSeek's open code model family that challenged closed-source code models.
A lightweight vLLM implementation built from scratch in about 1,200 readable lines of Python.
The most-starred other Local LLM projects in the LibreAI catalogue — the realistic shortlist if vllm-omni is not the right fit.
The simplest way to run open language models on your own machine.
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
LLM inference in C/C++
A high-throughput and memory-efficient inference and serving engine for LLMs
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
Free and open — grab the source, star the repo, and start creating.