Needle is an on-device foundation model for tool calling, structured extraction and text embeddings, shipped as a single 8-29 MB binary. It is aimed at developers who build for phones, wearables, smart-home devices, robots, cars and microcontrollers, where a cloud model is unreachable, too slow or too expensive to call on every user action. The project comes from Cactus Compute, is written mostly in Python, ships under Apache-2.0, and has gathered roughly 11,500 stars since it first appeared in February 2026.
What it does
Needle deliberately gives up general chat ability in exchange for being very good at three narrow jobs that applications actually need:
- Tool calls. You hand it the functions your app exposes; it picks the right ones and fills in every argument from what the user said. Ask for two things and you get two calls in order. Ask for something none of the tools cover and you get an empty list rather than an invented call — the behaviour the video demonstrates by asking it, in one plain sentence, to call a weather function.
- Structured extraction. You declare a shape and pass in messy text: an invoice, a booking, a notification, a form. What comes back is typed fields. The decode grammar guarantees the output parses, so there is no salvage step around malformed output, and the same mechanism generalises to classification.
- Text embeddings. The same model returns a vector for a sentence, so search, matching and routing can all happen locally instead of being split between an embedded model and a remote one.
The project's own claim is that it beats models ten times its size on mobile tool calls and matches models two to three times bigger on extraction. The benchmark chart in the repository reports tool calling as exact-match accuracy and extraction as field micro-F1, both on full test splits rather than sampled subsets, which is the honest way to publish those numbers.
How it works
Needle 3 is what the authors call a Laddered Simple Attention Network. The feed-forward block is replaced by a Monarch Hadamard MLP; attention is grouped-query with causal convolution taps; there is an engram n-gram memory read by gather; and the layers are wired with multi-lane hyper-connections. The ladder part matters for deployment: the model is trained so that every depth from 2 to 20 layers is itself a shippable model, which is how one project covers the range from a microcontroller to a phone. Most of the parameters live in the engram, so the 121M-parameter version does roughly the arithmetic of a 50M one. Weights are quantised to 2 bits, which is how the whole thing fits in tens of megabytes.
Two details are worth noting for anyone wiring this into production code. A byte-level grammar is compiled from your schemas and constrains every generated token, so structurally invalid output is not merely unlikely but excluded. And every response carries a calibrated confidence score from a learned head, which gives your application a number to threshold on before it acts on a tool call.
Getting started
Installation is a single Python package:
pip install cactus-needle
The README stops shortly after that line in the copy available here, so treat the usage examples, the interactive frontier plot, the architecture diagram and the fine-tuning results as living on the project's release page rather than in the repository text. The repository is active — the last push was in September 2026, about seven months after it was created — so the published docs are the place to check exact API shapes before you build against them.
When to use it / when not
Reach for Needle when the job is turning a user's sentence into a function call or a typed record, on hardware that has to work offline, cheaply, or without sending user text anywhere. Battery-powered and memory-constrained targets are the whole point: an 8-29 MB binary sits comfortably inside an app bundle, and there is no per-request cost or network latency.
Do not reach for it when you want a conversational assistant, long-form writing, reasoning over long documents or broad world knowledge. The general chat capacity was traded away on purpose, and no amount of prompting brings it back. It is also worth checking that your task genuinely fits one of the three supported shapes before adopting it — this is a component with a defined job, not a drop-in replacement for a large language model.
Anyone shipping an app that needs natural-language control of its own features should take Needle seriously, especially on mobile and embedded targets where the alternative is a network call on every interaction. The permissive licence, the grammar-constrained output and the confidence head make it practical rather than merely impressive, and the narrow scope is a feature: it is small because it does not try to be everything.