Local LLM
nobodywho
Lightweight inference engine for running LLMs efficiently on any device.
★ 1.5k starsEUPL-1.2
View guide →Alibaba's lightweight, blazing-fast inference engine for on-device LLMs and edge AI.
MNN is an open-source local llm project by alibaba (alibaba/MNN on GitHub). It has 16,192 GitHub stars and 2,467 forks, is released under the Apache-2.0 license, and was last updated on GitHub on 8 October 2026.
Decide whether you are targeting Android, iOS, embedded Linux or desktop, and choose a model to convert, such as a Qwen-series LLM for on-device chat.
Get the MNN source, which includes the converter, runtime and sample apps including the MNN LLM chat apps.
Convert your TensorFlow, ONNX, Caffe or TorchScript model to MNN format and build the library for your platform, choosing CPU or GPU backends.
Integrate the runtime into your app or run the included chat app, then tune quantization (FP16/INT8) for size and speed.
Optimized for devices with no dependencies: the Android core library is about 800KB, and FP16/INT8 quantization can cut model size by 50-70%.
The converter supports models from TensorFlow, Caffe, ONNX and TorchScript, covering CNN, RNN, GAN and Transformer networks.
Runs on ARM and x64 CPUs with optimized assembly, plus Metal, OpenCL and Vulkan on mobile GPUs, CUDA on NVIDIA, and a Hexagon DSP backend.
Powers local LLM apps, with recent releases adding support for Qwen3 and Qwen3-VL series models and multimodal chat on Android and iOS.
An MNN Python API lets ML engineers run inference, training and image processing without writing C++.
MNN supports training models on PC or mobile in addition to inference, plus a lightweight OpenCV-like image processing module.
Clone the MNN source, including the converter, runtimes and sample apps.
git clone https://github.com/alibaba/MNN.git
cd MNNConvert an ONNX, TensorFlow, Caffe or TorchScript model into MNN format with the included converter before deploying it.
# Convert your model with the MNN converter - see README for the exact commandBuild MNN for Android, iOS or desktop following the platform guide, or open one of the included LLM chat sample apps.
# Build for your target platform - see README for the exact commandMNN is developed by Alibaba and the open-source community; it is battle-tested in Alibaba's own products.
Yes. It is released under the Apache-2.0 license, which allows commercial use.
No. MNN runs models locally on your device, so there is no cloud API or key involved.
MNN is a lightweight C++ engine focused on fast on-device inference, with its own converter for TensorFlow, Caffe, ONNX and TorchScript models and strong mobile GPU and LLM support; the best choice depends on your model and target hardware.
Clone the repo, convert a model with the MNN converter, and try one of the included sample apps, such as the Android or iOS LLM chat app.
Lightweight inference engine for running LLMs efficiently on any device.
A high-throughput and memory-efficient inference and serving engine for LLMs
Offline inference engine for art.
Fast Rust scraper that turns web pages into clean data for LLMs.
Compile and run LLMs anywhere - phones, laptops, browsers and GPUs - with one engine
A fast LLM inference server for Apple Silicon, managed from the macOS menu bar
The most-starred other Local LLM projects in the LibreAI catalogue — the realistic shortlist if MNN is not the right fit.
The simplest way to run open language models on your own machine.
User-friendly AI Interface (Supports Ollama, OpenAI API, ...)
LLM inference in C/C++
A high-throughput and memory-efficient inference and serving engine for LLMs
🤯 LobeHub is your Chief Agent Operator, organizing your agents into 7×24 operations by hiring, scheduling, and reporting on your entire AI team.
Free and open — grab the source, star the repo, and start creating.