Morphology Review Pilot — cost and design

Date: 2026-08-15 Question: what would it cost to pilot an LLM review of words.root, the way the 7 LLM norms were built? Short answer: the API spend is negligible (≈$2–6 for the entire 20K population, ~$0.10 for a pilot). The cost that matters is building the gold set to validate against — roughly a day of agent time, or two to three hours of yours if you label the gold yourself. Recommend piloting.

Driver: PHON-231 — root carries hallucinated lemmas (sheep→shoop, informal→cocky, wild→wildcat, ad→ave). Since PHON-205 removed image inheritance, root's only functional consumer is the "Base forms only" filter, so every bad root silently hides a real word.

Why this is cheaper than a norm build

The 7 norms (research/2026-04-30-llm-word-features/build_<norm>.py) ask for a rating — a subjective 1–7 judgment with no ground truth, validated by correlating against a human oracle (Glasgow, Spearman 0.868 for AoA). Morphology is a verification task with three advantages:

1. Binary/categorical, not continuous. "Is sheep an inflection of shoop?" has a right answer. No calibration curve, no scale anchoring, no E[rating] integration — just P(yes) over logprobs, using the same top_logprobs=20 machinery already in the harness.
2. Evidence can be supplied in-prompt. The norms ask the model to introspect. Here we hand it the word, the candidate root, both WordNet gloss sets, and the POS pair — the same evidence that made today's PHON-216 curation tractable.
3. It only has to beat a very low bar. The current pipeline emits sheep→shoop. We are not chasing linguistic perfection; we are removing assignments a competent reader can see are wrong.

Cost arithmetic

Population: 20,108 canonical words with a non-null root. Verified against data/runtime/words.parquet — and note that every one of the 20,108 has root != word, so this is not a population to sample from, it is exactly the set the "Base forms only" filter hides. Reviewing all of it and reviewing the blast radius are the same job. The whole lexicon only adds 1,218 more (21,326 total), so there is no meaningful saving in staying canonical-scoped; review all 21,326.

Prompt: word + root + POS pair + up to 3 glosses each ≈ 250 input tokens (the AoA prompt is ~110; glosses roughly double it). Output with the norms' max_completion_tokens=4 ≈ 4 tokens.

	tokens	gpt-4.1-mini rate	cost
input	21,326 × 250 = 5.33M	~$0.40 / 1M	$2.13
output	21,326 × 4 = 85K	~$1.60 / 1M	$0.14
full run (whole lexicon)			≈ $2.27
pilot (n=400)			≈ $0.04
3× headroom (richer prompt, a retry, a second pass)			≈ $6.80

Verify current per-token pricing before quoting these; the arithmetic is what matters, not the rate I assumed.

Wall clock at the norms' --concurrency 6: ~21K calls ≈ 1.5–2 hours. Higher concurrency shortens it; the harness is already resumable via append-mode TSV, so an interrupted run costs nothing.

The API spend is not the decision. At $2–6 the run is free in every sense that matters. The decision is whether we can tell whether it worked.

What actually costs: the gold set

This is the whole pilot. Without ground truth, a run produces 21,326 confident verdicts and no way to know if they're better than what we have.

The trap, learned today (PHON-216): the same signal scored precision 1.00 on a curated 40-word test set and 0.48 on a random 150-word sample. The curated set was built from known examples, so it measured performance on cases the method was designed around. The gold set must be a uniform random sample of the 21,326, labeled blind — no convenience sampling, no starting from the words we already know are broken.

Recommended gold set: n=400, uniform random, each labeled correct / wrong / unsure against the question "is word an inflected form of root?" At the ~5% error rate hinted by the stem-mismatch signature, n=400 gives roughly ±2% on the error rate and enough wrong cases (~20) to characterize failure modes. Label it blind to the model's verdicts, or the labels drift toward them.

Cost of the gold set: ~1 day of agent time (a labeling pass plus an independent adversarial pass, exactly the two-pass shape used for PHON-216's 3,124 items), or 2–3 hours of your time if you want human-authored ground truth for a linguistic claim. Given that root is linguistic data others may cite, your labels are worth more than mine — but agent labels with an adversarial second pass are defensible, and that's the precedent the norms set.

Evidence we already have that this works

Today's PHON-216 curation was this method at small scale — LLM classification of 3,124 root-bearing forms with gloss evidence, followed by an independent adversarial pass:

• The adversarial pass disagreed with 20 of 317 keeps (6.3%) and 2 of 120 sampled hides.
• Every disagreement was a judgment call (rare-sense-vs-common-sense), not a factual error about morphology.
• All ten seeded control cases landed correctly (rights, glasses, arms, customs kept; judges, queens, years, rates, bowls hidden).

That's an observed ~94% inter-pass agreement on a harder, more subjective question than "is X an inflection of Y." It's a reason for optimism, not a substitute for the gold set.

Pilot design

Phase 1 — gold set (the expensive part). 400 uniform-random draws from the 21,326. Two independent labeling passes, disagreements adjudicated. Output: data/norms/_gold/root_gold.tsv (word, root, label, rationale). Keep it — it becomes the regression fixture for any future root change, including PHON-231's own fix.

Phase 2 — the run (cheap). Adapt build_<norm>.py: same client, concurrency, resume, and top_logprobs machinery; swap E[rating] for P(correct). Run over the same 400. Cost ≈ $0.04.

Phase 3 — read the result. Compare against gold at several P(correct) thresholds: - Agreement / precision / recall on wrong (the class we care about — a missed bad root keeps hiding a real word). - Where it sits on the cost curve. A high-precision, low-recall setting that flags 500 words for review is useful (that's the PHON-216 shortlist posture). A low-precision setting is not shippable as an automatic correction, exactly as PHON-216 concluded. - The decision that follows: if precision on wrong is ≥0.9 at a usable recall, run the full 21,326 for ~$2 and treat the flags as a review queue. If it's ~0.5, we've learned that cheaply and PHON-231 falls back to auditing the 173 stem-mismatch cases by hand.

Explicitly out of scope for the pilot: changing root in the pipeline. The pilot produces a measurement and a decision, not a data change. Any correction ships behind the same reseed gating as PHON-216's column.

Recommendation

Run the pilot. The API cost is a rounding error, the harness already exists, the gold set has standalone value as a regression fixture, and the alternative — hand-auditing 21,326 assignments — is not something anyone will do. The honest risk is that it lands at 0.5 precision like the PHON-216 heuristic did, and the pilot is designed to surface that for ~$0.04 plus a day rather than after a full run and a reseed.

One caveat worth stating: an LLM asked "is X an inflection of Y" will be fluent about morphology whether or not it is right, and the failure mode will be confident agreement with the existing bad root. The gold set is the only defense, which is why it's the part worth spending on.