Lokale LLMs
code-context-engine
Code-context-engine, in Python — built for save 94% on ai coding tokens. index your cod...
★ 425 SterneMIT
Anleitung ansehen →nano-vllm ist ein Open-Source-Projekt aus der Kategorie Lokale LLMs von GeeeekExplorer (GeeeekExplorer/nano-vllm auf GitHub). Es hat 15.743 Sterne auf GitHub und 2.666 Forks, wird unter der Lizenz MIT veröffentlicht und wurde zuletzt am 26. April 2026 auf GitHub aktualisiert.
A lightweight vLLM implementation built from scratch in about 1,200 readable lines of Python.
Choose nano-vllm to run offline inference with a codebase small enough to read fully, or to prototype engine ideas quickly.
Install nano-vllm directly from its GitHub repository with pip.
Fetch a model such as Qwen3-0.6B from Hugging Face to a local directory.
Load the model with the LLM class, set SamplingParams, and call generate on your prompts, as in example.py.
The whole implementation is about 1,200 lines of clean Python, so you can actually read how inference works.
The README benchmark shows output throughput comparable to vLLM on the tested setup (Qwen3-0.6B, RTX 4070 Laptop).
Reuses cached prefixes across requests to avoid recomputing shared prompt beginnings.
Splits a model across multiple GPUs with the tensor_parallel_size setting when one GPU is not enough.
Uses torch compilation and CUDA graphs as part of its optimization suite to cut inference overhead.
The LLM and SamplingParams interface mirrors vLLM, so existing vLLM-style code needs only minor changes.
Clone nano-vllm to read the source and run the included example and benchmark scripts.
git clone https://github.com/GeeeekExplorer/nano-vllm.git
cd nano-vllmInstall the package directly from the repository with pip.
pip install git+https://github.com/GeeeekExplorer/nano-vllm.gitDownload Qwen3-0.6B weights, then run example.py-style code with the LLM class and SamplingParams.
huggingface-cli download --resume-download Qwen/Qwen3-0.6B --local-dir ~/huggingface/Qwen3-0.6B/ --local-dir-use-symlinks FalseIt is a project by GeeeekExplorer with open-source contributors.
Yes. It is released under the MIT license.
No. It runs models locally on your GPU; you only download open model weights from Hugging Face.
Nano-vLLM trades vLLM's breadth (serving features, model coverage, production hardening) for a tiny, readable codebase with comparable offline throughput on the README's tested setup.
pip install it from GitHub, download Qwen3-0.6B, and run the example in example.py.
Code-context-engine, in Python — built for save 94% on ai coding tokens. index your cod...
Go-built AI gateway unifying many providers with routing and guardrails.
vLLM-Omni — efficient inference for omni-modality models: image, audio, video, diffusion and policies.
Early proposal and implementation for connecting LLMs to web pages.
Token efficient translation utility built on ChatGPT or any OpenAI compatible LLM.
Google DeepMind's Gemma open-weight LLM library — install, load and chat with Gemma models in Python.
Die anderen Projekte der Kategorie Lokale LLMs mit den meisten Sternen im LibreAI-Katalog — die realistische Auswahl, falls nano-vllm nicht das Richtige ist.
The simplest way to run open language models on your own machine.
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
LLM inference in C/C++
A high-throughput and memory-efficient inference and serving engine for LLMs
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
Verwandte Projekte der Kategorie Lokale LLMs im Katalog, die nano-vllm bei den Sternen am nächsten stehen — zum Weiterstöbern auf beiden Seiten.
Alibaba's lightweight, blazing-fast inference engine for on-device LLMs and edge AI.
Open bilingual (Chinese-English) 6B chat LLM with 32K-context variant and low-memory INT4 inference.
Qwen3-Coder - Alibaba's open code model series for agentic coding, up to 1M tokens of context
Access large language models from the command-line
High-performance In-browser LLM Inference Engine
Typescript/React Library for AI Chat 💬🚀
Kostenlos und offen — hol dir den Quellcode, gib dem Repository einen Stern und leg los.