AirLLM is a Python inference library that runs very large language models on a single small GPU by loading the model's layers from disk one at a time. It is aimed at machine-learning engineers, researchers and hobbyists who want to put a 70B-class open model (or something far bigger) on a desktop card instead of renting a multi-GPU server. The project is Apache-2.0 licensed, has roughly 36,000 stars and 3,800 forks, and was last pushed in October 2026.
What it does
The headline claim, made by the project in its own README, is that a 70B model runs inference on a single 4GB GPU card with no quantization, no distillation and no pruning: the weights are used as they ship. The README goes further and lists several records, again the authors' own figures rather than an independent benchmark:
- Kimi K3 (2.8T parameters), which the README calls the largest open-source model released to date, under 4GB
- Qwen3.8-Flash-Next (125B) on 6GB
- DeepSeek-V3 (671B) on about 12GB
- training, not only inference, of the 125B model under 6GB
Loading a model takes one line of code, and the library covers several families including Qwen, DeepSeek and Llama. The repository's main language is Jupyter Notebook, and the README points to example notebooks, a Configurations section, a macOS section and an FAQ.
How it works
A transformer runs its layers in sequence, so AirLLM keeps only the layer it needs in VRAM. It streams the layers off disk one at a time, uses each one, then releases it. The ceiling on GPU memory becomes roughly the size of a single layer rather than the size of the whole model, which is why a 671B model can be quoted at around 12GB of VRAM.
The consequence is that nothing is compressed. Compared with a quantized build, there is no change to the numbers inside the model, because the full weights are still the ones being used; the saving comes from where they sit between uses. The corresponding cost moves to storage and to reading from it.
Getting started
The code lives at github.com/lyogavin/airllm and is published on PyPI as airllm, so the standard path is a package install (pip install airllm) rather than hunting for a build in Releases. There is no API key and nothing to subscribe to: the library is free under Apache-2.0, and the models are open weights you download yourself.
What you pay for is hardware and storage. You need a GPU, and the project's numbers are all quoted in VRAM, from 4GB up to about 12GB for DeepSeek-V3. You also need room on disk for the full, unshrunk weights, because disk is where the model actually lives between layer loads: the largest configuration in the material asks for 360GB of free disk space. The README has a dedicated macOS section, so Apple Silicon is addressed, though the material does not spell out what differs there.
The record configurations need extra setup beyond a plain install. Kimi K3 requires flash-attn and a CUDA 12 build, and Qwen3.8-Flash-Next requires transformers installed from source. Beyond that, the project does not list minimum CPU, system RAM or operating-system requirements, so assume some trial and error.
When to use it / when not
The main catch is the trade being made: memory savings come from disk, so weights have to be read off storage while the model runs, and nothing in the project's material here attaches a tokens-per-second figure to that. Read the 4GB claim as a statement about what fits, not about how quickly it answers. That makes AirLLM a tool for batch jobs, offline evaluation and experiments you can leave running, rather than a chat backend with someone waiting on it.
Use it instead of a quantized build, a 4-bit GGUF in llama.cpp for example, when you specifically need the unmodified weights: AirLLM shrinks nothing, while quantization alters the model's numbers to make it fit.
Don't use it if you want interactive latency, if you cannot spare hundreds of gigabytes of free disk for the largest models, or if you are not willing to build flash-attn and a CUDA 12 stack for the headline configurations. Renting a cloud GPU for an hour is still the simpler route when you just need one 70B answer quickly.
Anyone who has concluded that big open models are out of reach on the hardware they own should take this repository seriously, because it attacks that assumption from an unusual angle: not a smaller model, but the same model paged through a small card. The claims are the authors' own and the speed question is left open, so treat the numbers as an invitation to test rather than a measured result. Given a desktop NVIDIA GPU, a large disk and work that tolerates waiting, it is one of the few credible ways to run 70B and up without a server.