Local LLM
can-i-finetune-this
Check if your GPU can fine-tune an LLM — memory estimates and LoRA recipes.
★ 792 starsMIT
View guide →Run a 70B LLM on a single 4GB GPU — layer-by-layer inference with almost no memory tricks skipped
airllm is an open-source local llm project by lyogavin (lyogavin/airllm on GitHub). It has 35,511 GitHub stars and 3,753 forks, is released under the Apache-2.0 license, and was last updated on GitHub on 8 October 2026.
Decide what you want to achieve with airllm and check the README on GitHub for requirements before installing.
Clone the repository from GitHub (or take the packaged install route in step 3 if the README offers one).
Follow the project's documented install commands — the Run section below lists the exact ones — and add any required API keys or models.
Work through the README's quickstart example end to end, then adapt it to your own project or data.
Layer-by-layer loading lets models far larger than your VRAM still run, trading speed for reach.
Pass a Hugging Face repo id (or a local path) and AirLLM handles the rest — the quickstart is a few lines of Python.
Model compression via bitsandbytes can speed inference up to 3x, per the README.
Works with popular open model families listed in the README, including Llama and other Hugging Face models.
Includes a training path with dataset preparation and a Python API, not just inference.
Documents running on Apple hardware as well as CUDA GPUs.
Get the AirLLM source, examples and notebooks.
git clone https://github.com/lyogavin/airllm.git
cd airllmInstall AirLLM (and bitsandbytes if you want compression) from PyPI.
pip install airllm
pip install -U bitsandbytesLoad a Hugging Face model id through AirLLM's Python API as shown in the README quickstart.
# from airllm import AutoModel
# model = AutoModel.from_pretrained('meta-llama/Llama-2-70b-hf') # see README quickstartairllm is maintained by the lyogavin organization/account on GitHub, together with open-source contributors, in the public repository lyogavin/airllm.
Yes — the code is open source under the Apache-2.0 license. You can use, modify and redistribute it under that license's terms.
No API key and no paid service — models download from Hugging Face (some are gated and need a free Hugging Face account/token and license acceptance).
Those quantize models to fit memory and are much faster; AirLLM instead runs full-precision huge models layer-by-layer — slower, but it can run models that would not fit even quantized.
Clone the repository from GitHub, follow the install commands in the Run section (taken from the project's README), and complete the documented quickstart example before adapting it to your own use case.
Check if your GPU can fine-tune an LLM — memory estimates and LoRA recipes.
Open bilingual (Chinese-English) 6B chat LLM with 32K-context variant and low-memory INT4 inference.
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Distribute and run LLMs with a single file.
Compile and run LLMs anywhere - phones, laptops, browsers and GPUs - with one engine
TokenSpeed — a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability
The most-starred other Local LLM projects in the LibreAI catalogue — the realistic shortlist if airllm is not the right fit.
The simplest way to run open language models on your own machine.
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
LLM inference in C/C++
A high-throughput and memory-efficient inference and serving engine for LLMs
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
Free and open — grab the source, star the repo, and start creating.