Local LLM
airllm
Run a 70B LLM on a single 4GB GPU — layer-by-layer inference with almost no memory tricks skipped
★ 35.5k starsApache-2.0
View guide →Open bilingual (Chinese-English) 6B chat LLM with 32K-context variant and low-memory INT4 inference.
ChatGLM2-6B is an open-source local llm project by zai-org (zai-org/ChatGLM2-6B on GitHub). It has 15,501 GitHub stars and 1,780 forks, is released under the Apache-2.0 license, and was last updated on GitHub on 27 June 2024.
ChatGLM2-6B suits bilingual chat on modest hardware; if you need the strongest current model, the README points to the GLM-4 open series instead.
Get the code, demos and requirements file from GitHub; the weights are downloaded from Hugging Face (THUDM/chatglm2-6b).
Install the Python requirements, plus FastAPI and Uvicorn if you want the API demos.
Start the CLI or web demo for a first conversation, or the API server to integrate the model into your own app.
Trained for Chinese and English dialogue, with a smooth chat experience and a low deployment threshold inherited from the first ChatGLM.
FlashAttention-based training extends the base context from 2K to 32K, and a separate ChatGLM2-6B-32K variant targets long-text understanding.
Multi-Query Attention gives faster inference and lower memory use than the first generation - the README reports a 42% speed-up in the official implementation.
Runs quantized in as little as about 6GB of GPU memory, with the supported conversation length at INT4 raised from 1K to 8K tokens.
The repo includes cli_demo.py, web_demo.py, an api.py FastAPI server and an OpenAI-compatible API demo.
Supports splitting the model across several smaller GPUs with Accelerate when no single card can hold it.
Clone the ChatGLM2-6B code and demo scripts.
git clone https://github.com/zai-org/ChatGLM2-6B.git
cd ChatGLM2-6BInstall the Python requirements from the repo.
pip install -r requirements.txtStart a terminal conversation with the model (weights download from Hugging Face on first load).
python cli_demo.pyThe repo is under zai-org, from the team behind the GLM model family (Zhipu AI and Tsinghua KEG). The README now directs users to the newer GLM-4 open models.
The code is Apache-2.0. The weights are free for academic research, and free for commercial use after completing the project's registration form, under its Model License.
No. You download the open weights and run them on your own hardware.
It was a leading 6B bilingual model of its generation, but newer models such as the GLM-4 series and current Qwen releases are stronger; ChatGLM2-6B remains attractive for its small footprint and bilingual tuning.
Clone the repo, run pip install -r requirements.txt, then python cli_demo.py or python web_demo.py for the browser UI.
Run a 70B LLM on a single 4GB GPU — layer-by-layer inference with almost no memory tricks skipped
Check if your GPU can fine-tune an LLM — memory estimates and LoRA recipes.
A fast LLM inference server for Apple Silicon, managed from the macOS menu bar
Google DeepMind's Gemma open-weight LLM library — install, load and chat with Gemma models in Python.
A local-first context and memory layer for your AI agents.
TokenSpeed — a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability
The most-starred other Local LLM projects in the LibreAI catalogue — the realistic shortlist if ChatGLM2-6B is not the right fit.
The simplest way to run open language models on your own machine.
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
LLM inference in C/C++
A high-throughput and memory-efficient inference and serving engine for LLMs
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
Free and open — grab the source, star the repo, and start creating.