post3 minembeddingssearchlocal-aimeasurement
Measure the vectors before you build the index
The model advertises three jobs: tool calls, structured extraction, and embedding
text. The embedding head is real. needle_embed exists in the C API, returns
3072 floats, and the same weights do the tool-ranking that makes my router work.
So the natural move is to embed the vault and build a vector index. I did the cheaper thing first and measured whether the vectors separate documents, and the answer was no:
| measurement | value |
|---|---|
| pairwise cosine range | 0.935 – 0.990 |
| pairwise cosine stdev | 0.008 |
| “german word order” vs “bake sourdough bread” | 0.948 |
| embedding cost | 543 ms per note (151 s for the vault) |
A standard deviation of 0.008 across 278 notes is not a ranking signal. Nearest neighbours confirm it: the closest note to hybrid search is a robust default was one about irregular verbs, and any phrase returned a different arbitrary five.
The explanation is the training objective, not a bug. That head exists to rank tool schemas against a request — a corpus of dozens of short strings. Document retrieval wants a space where topical distance dominates; this one collapsed it.
The cost is the second half of the argument. At 543 ms per note, reindexing a growing vault is minutes of CPU for results worse than a keyword index answering in milliseconds.
A vector index is justified by measured separation — a query landing measurably nearer its own document — never by the presence of an
embedmethod.
The same lesson, in a classifier#
A lexical classifier is a different tool with the same kind of trap. Mine scores each option’s description against the situation text and reports the softmax separation as confidence. No model, no network, microseconds per decision. A self-test over twelve labelled cases splits the results in half:
| group | accuracy | coverage | accuracy when it answers |
|---|---|---|---|
| vocabulary (which topic, which MOC) | 1.00 | 1.00 | 1.00 |
| judgement (roll back or wait, offload or retry) | 0.50 | 0.75 | 0.67 |
The vocabulary group is genuinely solved. Given irregular verbs cluster into ablaut families and three described topics, it picks the German one every time, for free.
The judgement group is a coin flip with a confident voice. A build fails only on the small host with 8 GB of memory chose retry on the small host, because that option contains “build”, “small” and “host”. Lexical evidence pointed the wrong way, and no threshold fixes a wrong ranking.
Needle3 hit the same wall from another direction: asked to choose among three
customer-retention options, it answered offer_discount for a question about
orphaned notes — deterministically, three times — and returned nothing for
another. Its enum is honoured for schema-shaped extraction, not for judgement.
Confidence is a gate#
The model returns a calibrated confidence with every call, and it separates outcomes cleanly on real requests:
| request | routed tool | confidence |
|---|---|---|
| “find my notes about german word order” | search_notes (correct) |
0.523 |
| “search the vault for sourdough” | search_notes (correct) |
0.385 |
| “what links to the Kelly criterion zettel” | search_notes (wrong) |
0.199 |
| an ambiguous statement, one tool offered | withheld | 0.032 – 0.090 |
The third row is the useful one. The model was wrong, and the score said so
before anyone looked at the result. A router that ignores it runs the wrong
command; one that gates prints confidence 0.199 below threshold 0.25 and stops.
Two properties make that work, and both belong to the caller:
- The floor is set per tool set. A correct call scored 0.52 here. A threshold copied from another project would either pass everything or refuse everything.
- A withheld call is a decline, not a failure. Low-confidence calls go into a
suppressed_callslist instead of disappearing, so the tool and its arguments can be shown to a human before they are ignored.
What confidence does not buy is correctness. A gate reduces the damage of a weak engine; it does not make it strong. A destructive step still needs a human, never a score.
The line, written down#
The honest split on this vault is: classification is automated, judgement is not. Topic tagging, MOC placement and routing gates run on the envelope. A decision that turns on what a failure means stays with the agent or the person.
Writing the limit down is what makes the tool usable. --self-test prints both
numbers — 1.00 for vocabulary, 0.50 for judgement — so nobody has to guess which
group their case resembles, and the next person to reach for the model has the
receipt already in front of them.