Local LLM
code-context-engine
Code-context-engine, in Python — built for save 94% on ai coding tokens. index your cod...
★ 425 starsMIT
View guide →A lightweight vLLM implementation built from scratch in about 1,200 readable lines of Python.
nano-vllm is an open-source local llm project by GeeeekExplorer (GeeeekExplorer/nano-vllm on GitHub). It has 15,743 GitHub stars and 2,666 forks, is released under the MIT license, and was last updated on GitHub on 26 April 2026.
Choose nano-vllm to run offline inference with a codebase small enough to read fully, or to prototype engine ideas quickly.
Install nano-vllm directly from its GitHub repository with pip.
Fetch a model such as Qwen3-0.6B from Hugging Face to a local directory.
Load the model with the LLM class, set SamplingParams, and call generate on your prompts, as in example.py.
The whole implementation is about 1,200 lines of clean Python, so you can actually read how inference works.
The README benchmark shows output throughput comparable to vLLM on the tested setup (Qwen3-0.6B, RTX 4070 Laptop).
Reuses cached prefixes across requests to avoid recomputing shared prompt beginnings.
Splits a model across multiple GPUs with the tensor_parallel_size setting when one GPU is not enough.
Uses torch compilation and CUDA graphs as part of its optimization suite to cut inference overhead.
The LLM and SamplingParams interface mirrors vLLM, so existing vLLM-style code needs only minor changes.
Clone nano-vllm to read the source and run the included example and benchmark scripts.
git clone https://github.com/GeeeekExplorer/nano-vllm.git
cd nano-vllmInstall the package directly from the repository with pip.
pip install git+https://github.com/GeeeekExplorer/nano-vllm.gitDownload Qwen3-0.6B weights, then run example.py-style code with the LLM class and SamplingParams.
huggingface-cli download --resume-download Qwen/Qwen3-0.6B --local-dir ~/huggingface/Qwen3-0.6B/ --local-dir-use-symlinks FalseIt is a project by GeeeekExplorer with open-source contributors.
Yes. It is released under the MIT license.
No. It runs models locally on your GPU; you only download open model weights from Hugging Face.
Nano-vLLM trades vLLM's breadth (serving features, model coverage, production hardening) for a tiny, readable codebase with comparable offline throughput on the README's tested setup.
pip install it from GitHub, download Qwen3-0.6B, and run the example in example.py.
Code-context-engine, in Python — built for save 94% on ai coding tokens. index your cod...
Go-built AI gateway unifying many providers with routing and guardrails.
vLLM-Omni — efficient inference for omni-modality models: image, audio, video, diffusion and policies.
Early proposal and implementation for connecting LLMs to web pages.
Token efficient translation utility built on ChatGPT or any OpenAI compatible LLM.
Google DeepMind's Gemma open-weight LLM library — install, load and chat with Gemma models in Python.
The most-starred other Local LLM projects in the LibreAI catalogue — the realistic shortlist if nano-vllm is not the right fit.
The simplest way to run open language models on your own machine.
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
LLM inference in C/C++
A high-throughput and memory-efficient inference and serving engine for LLMs
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
Free and open — grab the source, star the repo, and start creating.