~jisifu

notebook · envs.net
no framework, no fonts

post3 minembeddingssearchlocal-aimeasurement

Measure the vectors before you build the index

The model advertises three jobs: tool calls, structured extraction, and embedding text. The embedding head is real. needle_embed exists in the C API, returns 3072 floats, and the same weights do the tool-ranking that makes my router work.

So the natural move is to embed the vault and build a vector index. I did the cheaper thing first and measured whether the vectors separate documents, and the answer was no:

measurement value
pairwise cosine range 0.935 – 0.990
pairwise cosine stdev 0.008
“german word order” vs “bake sourdough bread” 0.948
embedding cost 543 ms per note (151 s for the vault)

A standard deviation of 0.008 across 278 notes is not a ranking signal. Nearest neighbours confirm it: the closest note to hybrid search is a robust default was one about irregular verbs, and any phrase returned a different arbitrary five.

The explanation is the training objective, not a bug. That head exists to rank tool schemas against a request — a corpus of dozens of short strings. Document retrieval wants a space where topical distance dominates; this one collapsed it.

The cost is the second half of the argument. At 543 ms per note, reindexing a growing vault is minutes of CPU for results worse than a keyword index answering in milliseconds.

A vector index is justified by measured separation — a query landing measurably nearer its own document — never by the presence of an embed method.

The same lesson, in a classifier#

A lexical classifier is a different tool with the same kind of trap. Mine scores each option’s description against the situation text and reports the softmax separation as confidence. No model, no network, microseconds per decision. A self-test over twelve labelled cases splits the results in half:

group accuracy coverage accuracy when it answers
vocabulary (which topic, which MOC) 1.00 1.00 1.00
judgement (roll back or wait, offload or retry) 0.50 0.75 0.67

The vocabulary group is genuinely solved. Given irregular verbs cluster into ablaut families and three described topics, it picks the German one every time, for free.

The judgement group is a coin flip with a confident voice. A build fails only on the small host with 8 GB of memory chose retry on the small host, because that option contains “build”, “small” and “host”. Lexical evidence pointed the wrong way, and no threshold fixes a wrong ranking.

Needle3 hit the same wall from another direction: asked to choose among three customer-retention options, it answered offer_discount for a question about orphaned notes — deterministically, three times — and returned nothing for another. Its enum is honoured for schema-shaped extraction, not for judgement.

Confidence is a gate#

The model returns a calibrated confidence with every call, and it separates outcomes cleanly on real requests:

request routed tool confidence
“find my notes about german word order” search_notes (correct) 0.523
“search the vault for sourdough” search_notes (correct) 0.385
“what links to the Kelly criterion zettel” search_notes (wrong) 0.199
an ambiguous statement, one tool offered withheld 0.032 – 0.090

The third row is the useful one. The model was wrong, and the score said so before anyone looked at the result. A router that ignores it runs the wrong command; one that gates prints confidence 0.199 below threshold 0.25 and stops.

Two properties make that work, and both belong to the caller:

  1. The floor is set per tool set. A correct call scored 0.52 here. A threshold copied from another project would either pass everything or refuse everything.
  2. A withheld call is a decline, not a failure. Low-confidence calls go into a suppressed_calls list instead of disappearing, so the tool and its arguments can be shown to a human before they are ignored.

What confidence does not buy is correctness. A gate reduces the damage of a weak engine; it does not make it strong. A destructive step still needs a human, never a score.

The line, written down#

The honest split on this vault is: classification is automated, judgement is not. Topic tagging, MOC placement and routing gates run on the envelope. A decision that turns on what a failure means stays with the agent or the person.

Writing the limit down is what makes the tool usable. --self-test prints both numbers — 1.00 for vocabulary, 0.50 for judgement — so nobody has to guess which group their case resembles, and the next person to reach for the model has the receipt already in front of them.