post4 minlocal-aisearchtoolingmeasurement
The 35 MB router: what a small model is actually good at
My note search used to require a compiler. QMD’s package pulls in
node-llama-cpp and a cmake native build, so any host that wanted to search a
folder of markdown needed a C++ toolchain — for a retrieval job that a BM25
implementation in the standard library answers in milliseconds.
The fallback was already in the repo, and the measurement that settled it was
not about speed. The fallback was the only engine that ran everywhere. So the
delegation was deleted rather than maintained, and search became BM25, full
stop.
What moved up a level instead of disappearing: a ~35 MB on-device model
(needle3) reads find my notes about german word order, picks the search_notes
tool, fills the argument, and runs the BM25 command. Understanding on top, exact
retrieval underneath.
One detail worth copying: the output contract was left alone. --json still
emits qmd://vault/<path> URIs, because a parser downstream consumes that as a
wire format. The URI outlived the engine. Renaming it to tidy a string would have
broken the layer above to make a path look tidier.
The obvious next question#
If a small model can choose between four tools, why not give it all 140 commands in the taskfile? The schemas fit. This is where the measurements start being useful, because the answer is no, and it fails in three separate places.
| catalogue | init | routing |
|---|---|---|
| 4 curated tools | ~1 s | correct tool and arguments, confidence 0.52 – 0.74 |
| 130 verbose schemas | fits | — |
| 131 verbose schemas | fails | needle_init failed |
| 140 schemas, 80-char descriptions | 63.7 s | 2/5 exact, confidence 0.13 – 0.26 |
| 140 schemas, 40-char descriptions, no parameters | 33.3 s | 2/5 |
The static prefix has a hard ceiling. System prompt plus tool schemas must
fit the window, and needle_init refuses rather than trimming. Shortening
descriptions moved the ceiling past 140 and did not move the accuracy at all.
Accuracy degrades long before the ceiling. At 140 tools the model returned
search-console-test for search my notes, test-modular for run the test
suite, fix-frontmatter for rebuild the search index. Those are plausible
neighbours, which is worse than nonsense — they are commands that do something.
Init cost is its own wall. Twenty-seven to sixty-four seconds per process. Even if accuracy had held, a CLI that spends a minute loading a tool list loses to typing the command.
Confidence collapsed to 0.13 – 0.26 with all of it, i.e. under any sensible floor, so the gate then declines nearly every request. The big-catalogue configuration buys nothing over not having a router.
Split the catalogue, then measure the split#
The obvious remedy is a first hop over a handful of groups and a second hop inside the chosen group. I built it and measured it, because the prediction was the model’s own advice and advice is not evidence.
| configuration | right group | exact recipe | cost |
|---|---|---|---|
| flat, 140 schemas | — | 2/5 | 27–64 s init |
| two-hop, model on both hops | 26/28 | 8/28 | ~2 s + ~2 s |
| two-hop, lexical on both hops | 26/28 | 18/28 | 1 ms |
Splitting worked where it was supposed to: the right group is chosen 26 times out of 28, so narrowing a catalogue is a real lever. But the second hop reproduced the flat failure almost exactly — 8/28 — because 37 near-identical sibling names is the same disease as 140.
And on that second hop the model lost to BM25, 8/28 against 18/28, at roughly eight thousand times the cost. Where they disagreed, the lexical matcher was usually right for a boring reason: recipe names are descriptive prose.
The model’s value is concentrated in the first hop, where a group description is long and written like a sentence. Below that granularity, BM25 on the command names is both better and cheaper.
The hazard the split created#
Read/write sibling pairs are one token apart: format and format-apply,
backlinks and backlinks-apply. The longer name matches first more often than
not. A router that prefers -apply because it is longer will do the thing when
the user asked to look.
The matcher now prefers the read-only sibling and prints the resolved command before running it. That is the cheapest guard I know of that survives a wrong hop, and it costs one line of output.
A second trap applies to any conversational reuse: turns accumulate, so unrelated queries left in the same context lose accuracy and confidence. Construct the agent stateless, or reset between independent requests, or you are measuring your own history.
What I actually ship#
Both modes, four curated tools by default, and a router whose job is to reach the right group and admit when it cannot. The model’s confidence is a threshold that turns a wrong answer into a visible refusal, which is a different thing from making the answer reliable — that distinction got its own post.
The general lesson is not “small models are bad at tools”. It is that a catalogue has a measured ceiling that sits far below its fitted ceiling, and the only way to find yours is to count the exact hits instead of admiring that it loaded.