Lokale LLMs
vLLM
A high-throughput and memory-efficient inference and serving engine for LLMs
★ 93,1 Tsd. SterneApache-2.0
Anleitung ansehen →vllm-omni ist ein Open-Source-Projekt aus der Kategorie Lokale LLMs von vllm-project (vllm-project/vllm-omni auf GitHub). Es hat 7.085 Sterne auf GitHub und 1.871 Forks, wird unter der Lizenz Apache-2.0 veröffentlicht und wurde zuletzt am 8. Oktober 2026 auf GitHub aktualisiert.
vLLM-Omni — efficient inference for omni-modality models: image, audio, video, diffusion and policies.
Pick the model family you will serve (omni, TTS, diffusion or robot policy) and confirm your GPU capacity against that model's requirements.
Clone vllm-project/vllm-omni from GitHub (or use the install path in the run steps below) and skim its README for the parts relevant to your goal.
Follow the vLLM-Omni documentation installation guide for your CUDA environment, or use the nightly CUDA Docker images built from main.
Serve one supported model from Hugging Face and call it through the OpenAI-compatible API with a multimodal request.
One framework serves models across text, image, audio, video and action data, with heterogeneous outputs.
Extends vLLM's AR strengths to Diffusion Transformers and other parallel-generation models.
Stage execution overlaps for throughput, with full disaggregation via OmniConnector and dynamic resource allocation across stages.
State-of-the-art autoregressive support through vLLM's efficient KV-cache management, plus streaming outputs.
Tensor, pipeline, data and expert parallelism for serving large models across GPUs.
Serves through an OpenAI-compatible API server, with seamless integration for popular Hugging Face models.
Get the source from the official GitHub repository for vllm-omni.
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omniInstall following the official installation documentation for your environment, as the README's getting-started links direct.
# Follow https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/ — see READMEStart the OpenAI-compatible server for a supported model and send it a request, per the docs quickstart and deployment recipes.
# Launch a documented quickstart model server, then call its OpenAI-compatible endpoint — see READMEThe vllm-project, through the vllm-project/vllm-omni repository, with docs at docs.vllm.ai/projects/vllm-omni.
Yes — the project is open source under the Apache-2.0 license as recorded for this listing. See the license terms for any conditions on redistribution.
No API key is needed — it is self-hosted inference software under Apache-2.0; your costs are GPU hardware and the models you download.
Vanilla vLLM serves autoregressive text models superbly. vLLM-Omni keeps that core and adds omni-modality and diffusion models, heterogeneous outputs and staged pipelines — choose it when your models see, hear, generate media or act, not just chat.
Clone the repository (or use the documented installer), follow the install and configure steps on this page, and run the smallest real task first — serve one supported model from Hugging Face and call it through the OpenAI-compatible API with a multimodal request.
A high-throughput and memory-efficient inference and serving engine for LLMs
BitNet — Microsoft's official inference framework for 1-bit large language models, running fast on ordinary CPUs, MIT
TokenSpeed — a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in productio…
Run and fine-tune LLMs and diffusion models locally — faster training, smaller memory footprint.
DeepSeek-Coder-V2 — DeepSeek's open code model family that challenged closed-source code models.
Die anderen Projekte der Kategorie Lokale LLMs mit den meisten Sternen im LibreAI-Katalog — die realistische Auswahl, falls vllm-omni nicht das Richtige ist.
The simplest way to run open language models on your own machine.
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
LLM inference in C/C++
A high-throughput and memory-efficient inference and serving engine for LLMs
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
Verwandte Projekte der Kategorie Lokale LLMs im Katalog, die vllm-omni bei den Sternen am nächsten stehen — zum Weiterstöbern auf beiden Seiten.
Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
DeepSeek-Coder-V2 — DeepSeek's open code model family that challenged closed-source code models.
KoboldCpp - Run GGUF AI models locally with a KoboldAI UI. One File. Zero Install
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots
Typescript/React Library for AI Chat 💬🚀
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Kostenlos und offen — hol dir den Quellcode, gib dem Repository einen Stern und leg los.