Colibri is an inference engine written in pure C that runs frontier Mixture-of-Experts language models on hardware you already own, by treating disk, RAM and VRAM as one continuous memory hierarchy. It is aimed at people who want to run very large models locally — machine-learning engineers, systems programmers and self-hosting enthusiasts who have a fast drive and a modest GPU rather than access to a cluster.
What it does
The project ships a small engine, built without engine dependencies, that loads and serves Mixture-of-Experts models far larger than the memory of the machine running them. Eight model families run today, each implemented as a single C file:
- GLM-5.2 and GLM-5.3, at 744 billion parameters
- GLM-5.3-Flash, 321 billion parameters, with vision
- Inkling, 975 billion parameters
- Kimi K3, 2.8 trillion parameters
- DeepSeek V4 Flash, 284 billion parameters
- Qwen3.8-Flash-Next, 125 billion parameters plus a 51 billion n-gram component
- Qwen3.6, a 35B-A3B model
- OLMoE, 7 billion parameters
All of them sit behind the same front end: coli chat for an interactive session, coli serve for a server, and coli web for a browser interface. The web view includes a live dashboard of expert activity; in the demo that accompanies this entry, a 744-billion-parameter model shows roughly nineteen thousand experts firing in real time as it generates.
The repository is Apache-2.0 licensed and, despite being young — first published in July 2026 — has already gathered more than twenty-seven thousand stars.
How it works
A Mixture-of-Experts model is mostly weights that are not needed for any particular token. Colibri leans on that: instead of demanding that the whole model fit in VRAM, or even in RAM, it keeps experts on storage and streams them in as they are required, with RAM acting as a cache layer between disk and the GPU. The authors call this AI memory multitiering, and it is what turns "you need a hyperscaler's GPU cluster" into "you need a big SSD".
The practical result shown in the video is a 744-billion-parameter model that becomes ready to answer in 32 seconds while staying under 10 GB of RAM. That is one machine and one configuration, not a published benchmark, but it illustrates the shape of the trade: startup and memory cost collapse, and the burden moves to storage bandwidth.
Beyond the engine itself, the project describes itself as an open research platform. The stated scope is inference-side performance across the whole software and hardware boundary — model formats, memory hierarchy, storage I/O, placement, scheduling, kernels, speculation, and CPU/GPU overlap — with the explicit aim of making large models depend less on scarce hardware. The maintainers are blunt about what that costs: there is no service-level agreement on speed, and a hard guarantee on semantics instead. Aggressive experiments are welcome, but they must not change what the model says.
Getting started
Tagged builds are published on the repository's releases page, and the project has its own website with documentation. Once installed, the three commands above are the entire surface: pick a model family, then chat, serve or open the web UI. Each supported family is one C file, so reading the implementation for the model you care about is realistic in an afternoon — an unusual property for an inference stack. The README is also maintained in several translations, and there is a Discord for questions.
When to use it / when not
Reach for Colibri when the model you want is far larger than your VRAM and you would rather spend disk than money; when you want a frontier-class model to stay on your own machine for privacy or cost reasons; or when you want to study how multitier inference actually behaves and have a single-file C implementation to instrument.
Look elsewhere when you need predictable throughput under load. The project says plainly that it makes no promises about speed, and streaming experts from storage means latency tracks your drive rather than your GPU. It is also a very new codebase with a fast-moving commit history, so treating it as settled production infrastructure is premature. And if a small model that fits comfortably in memory already answers your questions, the whole mechanism buys you nothing.
Anyone who has looked at a 744-billion-parameter release and assumed it was out of reach should take this repository seriously. Colibri is a working argument that the ceiling on local inference is a memory-management problem rather than a hardware budget, and it makes that argument in code small enough to read. Systems-minded engineers will find it the most interesting thing to study; everyone else gets a way to run models locally that, until recently, only rented clusters could serve.