FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine for running frontier-scale open-weight models on personal hardware. It is written for developers and ML engineers who already own a gaming PC or a workstation with one NVIDIA RTX card and plenty of system memory, and who would rather keep a very large model on their own machine than call it through somebody else's API. The code is Python under Apache 2.0, and the repository has roughly 15,200 stars and 1,500 forks since it first appeared in July 2026.
What it does
FreeToken takes the parts of a desktop that normally work separately (the GPU, the CPU, host memory and the links between them) and treats them as one elastic inference platform. On top of that pool it serves a model over an API that Claude Code and Codex can be pointed at, so a locally hosted model can sit behind tools that were built for hosted ones.
The README's headline claim is 290B+ parameter MoE models running locally at interactive speeds. That is the authors' own claim, not an independent benchmark. The repository's topics name the model families it targets: DeepSeek V4, GLM, MiniMax and Qwen.
How it works
An MoE model only activates a small share of its experts for any given token, which is what makes a 290B+ model thinkable on a desktop at all: most weights can sit in host memory and move to the GPU when they are needed. FreeToken's listed features are mostly about making that movement cheap:
- A global LRU cache for experts, so the experts just used are kept instead of being reloaded.
- Bandwidth-adaptive CPU and GPU co-execution (the README calls this its q-star policy), full-layer double-buffered prefill streaming, and graph-compatible execution.
- A fast weight format the project calls FTW.
- Semantic anchor checkpoints for recurrent state and KV caches, so agentic context edits such as tool calls and thinking blocks do not force the whole context to be recomputed.
- Elastic memory management: VRAM can be re-divided between expert caches and KV memory while the server is running.
The project's release notes report decode 8.7% faster with 24GB of free VRAM. That figure is the project measuring itself, and nobody here has re-run it. There is also a paper on arXiv (2608.16157) for readers who want the method rather than the changelog.
Getting started
The engine itself is free: Apache 2.0, no key, no account, and nothing in the material that bills you. The intelligence is yours to supply too, since you download open-weight models and run them on hardware you own. That means no API fee, but it also means you pay in VRAM, RAM and electricity.
Hardware is where the honesty matters. NVIDIA RTX GPUs are the native path and AMD support is experimental. The 24GB free-VRAM number in the release notes hints at the class of card the authors work with, but the project does not publish a minimum specification in the material available here, and it says nothing about Apple Silicon or about which operating systems are supported. A 290B-class model also has to live somewhere, which in practice means a large pool of system memory; no figure is given for that either.
The code is at github.com/FlashML-org/FreeToken, and the README's first link is a download at flashml.ai rather than a build-from-source recipe, so that page is the entry point and the repository is where the flags are documented. No install command appears in the material, so do not expect a published one-liner. Help lives in a developer Slack, a community Discord, and a WeChat group coordinated through issue 482 in the tracker.
When to use it / when not
Use it instead of a hosted frontier model when the work is steady, private or high volume, when the API bill is the thing you want gone, and when you have an RTX card with memory to spare. Use it instead of a general-purpose local runtime such as Ollama or llama.cpp when the model you want is a frontier MoE rather than a mid-sized dense model, since expert caching and CPU and GPU co-execution are the whole point of this engine.
Don't use it if your GPU is AMD or if you are on a Mac: the first path is explicitly experimental and the second is not addressed at all. Don't use it if you need a settled dependency, either. The repository was created in July 2026 and was still being pushed at the end of October 2026, so this is a fast-moving project only a few months old. And don't plan on serving a team from it: the stated goal is interactive use on one person's machine.
Anyone who has looked at a 290B-parameter open-weight model and assumed it needed a rack should take this repository seriously. FreeToken builds on the one structural fact that makes local serving plausible, that an MoE touches few of its experts per token, and spends its engineering on caching, streaming and runtime memory re-allocation rather than on new model claims. If you have an RTX card, a lot of RAM and a use case where sending text to a vendor is awkward, it is worth an evening. If you are on AMD or Apple hardware, watch the repository rather than planning around it.