Skip to content

Norms Recompute + Canonical Denylist — v7.0.2 Design

Date: 2026-07-21 Branch: fix/norms-recompute-7-0-2 (worktree .claude/worktrees/norms-7-0-2, off develop) Tickets: PHON-197 (canonical denylist) → PHON-196 (norm recompute) — one seed regen Target release: v7.0.2

Problem

PHON-193 (v6.4.0) redefined is_canonical as a two-sided rule (content-POS mass + WordNet-common-lemma realness). The canonical set grew to 48,314, but the in-house LLM-cloze norms were computed on the prior ~47.7K non-PROPN content vocabulary. Result: ~4,163 canonical words carry NULL for every LLM norm — they appear in Word Lists / Lookup / Contrast Sets but return nothing for norm-based percentile filters or Lookup norm readouts. This is the norm/canonical decoupling flagged in the v6.4.0 CHANGELOG.

Two coupled fixes, in order:

  1. PHON-197 — denylist. The gap set contains junk that shouldn't be canonical or normed: abbreviations (aaa, abc), genericized brands (pepsi, uzi), proper-dominant homographs (warner, hays). Drop these from canonical first so we don't spend LLM calls norming them and don't ship junk in generated candidate sets.
  2. PHON-196 — norm recompute. Run the existing gpt-4.1-mini cloze pipeline over the remaining real gap words for all 7 shipped LLM norms, merge into data/norms/phonolex_*.tsv, rebuild → reseed. Restores canonical ⊆ normed (modulo denylisted drops).

Key facts (verified against current code)

  • The norms are already LLM estimates (gpt-4.1-mini, logprobs=true, top_logprobs=20, max_completion_tokens=4, Brysbaert expected-value method). Filling the gap uses the identical method that produced the existing column — no provenance/explainability mismatch, and no mixing of human-rated and estimated values. This is why the OpenAI key is the right tool.
  • Column homogeneity requirement: the gap words MUST be rated with the same model (gpt-4.1-mini) and same prompt template as the existing rows. Re-rating the whole 48K set with a newer model is a separate, larger effort and explicitly out of scope here.
  • 7 shipped LLM norms in scope: aoa, concreteness, valence, arousal, familiarity, iconicity, boi. socialness is excluded (built but unsurfaced/empty). Corpus-derived norms (frequency, semantic_diversity, grade bands) and CMU-deterministic columns are not in scope — they already cover the full vocabulary or auto-compute.
  • The build scripts compute their vocabulary internally as CMU ∩ non-PROPN freq (build_aoa.py:load_content_freq_words filters Dom_PoS == 'PROPN'). This is the old scope and would exclude the newly-canonical PROPN-dominant kingdom-class words we need. The scripts must be fed an explicit target-word list instead of recomputing the old vocab.
  • Build scripts already support --resume (append + skip-done, flush every 50-word chunk) — this satisfies the checkpoint policy for a long job.

Design

Part A — PHON-197 canonical denylist

Insertion point: packages/data/src/phonolex_data/pipeline/canonical.py. The realness gate (_passes_realness / is_canonical) is where a denylist check belongs. Add a denylist: set[str] parameter (threaded from the pipeline caller) and short-circuit to non-canonical when a word is denylisted.

Three drop classes, each with its own mechanism:

  1. Abbreviations / initialisms — cheap heuristic. A word is treated as an abbreviation if it clears a shape test: length ≤ 3 with no vowel letter (ws, tlc), OR its CMU pronunciation is letter-spelled (each phoneme sequence maps to a letter name, e.g. abc → EY BIY SIY). Implement as is_abbreviation(record) in a new pipeline/denylist.py. Guardrail: only applies to short forms; real short words with vowels (ace, elf) are untouched.
  2. Genericized brands — no clean heuristic; curated list. New file data/vocab/brand-denylist.txt (one lowercased word per line, mirroring the data/vocab/profanity-list pattern). Seed from the PHON-197 examples + a review pass over the gap set's suspected brands.
  3. Proper-dominant homographs — signal + curated override. Signal: PROPN frequency-mass share from all_pos/all_freqs above a high threshold (start ~0.90, tuned against the QA sample). A word above the threshold is dropped unless it appears in a curated allow-list (data/vocab/canonical-allow.txt) protecting legit occupational/common nouns that are also common surnames (baker, mason, turner, carter, cook). A small curated deny-list (data/vocab/canonical-deny.txt) can force-drop edge cases the signal misses. The high threshold + allow-list is deliberately conservative — over-dropping real words is worse than leaking a few names, so QA prioritizes zero false-drops of the leave-alone classes.

Leave-alone classes (must NOT drop): real loanwords (signor, gateau, pilsner), legit demonyms (arabian, bostonian), rare-but-real content words (freq_floor=0 keeps them), interjections (hello).

QA: re-run the PHON-193 probe-style ADDED/DROPPED sampling (research/2026-07-19-canonical-redesign/ pattern) → new research/2026-07-21-canonical-denylist/. Report: count dropped per class, a spot-check sample of dropped words, and explicit confirmation the leave-alone classes survive. Denylist decisions + false-positive spot-check recorded per the ticket acceptance criteria.

Part B — PHON-196 norm recompute

Step 1 — target set. After the denylist is wired and words.parquet rebuilt, compute the gap set from the current build: is_canonical == 1 AND NULL across all 7 LLM norms. Write to research/2026-07-21-norms-recompute/gap_words.txt (regenerate, never hardcode). Expected ~3,500–4,000 after denylist drops.

Step 2 — target-list injection. Add a --words-file <path> argument to each of the 7 build_*.py scripts. When supplied, it replaces the internal CMU ∩ non-PROPN freq vocabulary computation with the explicit list (still intersected with CMU for pronunciation availability). This is the load-bearing change — without it the scripts re-exclude the PROPN-dominant words. Keep --resume append semantics pointed at the existing data/norms/phonolex_<norm>.tsv so new rows append and existing rows are skipped.

Step 3 — run. Run all 7 build scripts over the gap set (--resume, --concurrency 6, gpt-4.1-mini). Each run is resumable (append + flush every 50). Log [ckpt]-style progress. Record cost + coverage (fail count from NaN EVs). Estimated ~$3–5 total, ~1–2 h wall clock.

Step 4 — validate. Sanity-check the new rows against the existing normed population: distribution overlap per norm (the new words should not be wildly out-of-distribution), and where any gap word appears in a _oracles/ dataset, spot-check the estimate correlation via the existing validate_*.py scripts. This is a lightweight regression check, not a full re-validation.

Step 5 — rebuild + reseed. - build_runtime_parquet.py → norms attach to the gap words; percentiles recompute per-norm-population (bisect_right cumulative) so the newly-normed words are included. - export-to-d1.py → d1-seed.sql. - chunk-seed-sql.py + apply to local D1; verify gap-word norms resolve via /api/words/:word. - upload-seed-to-r2.py → commit updated d1-seed.manifest.json (the git pointer; .sql stays gitignored). - Deploy to staging via develop (manifest change triggers the CI reseed), verify, then promote to main after review.

Acceptance criteria

  • Canonical words missing core norms drop to ~0 (modulo deliberate denylist drops).
  • The 3 junk classes are measurably reduced in the ADDED/DROPPED QA sample; leave-alone classes preserved (zero false-drops in the sample).
  • New norm values are homogeneous with the existing column (same model + prompt); distributions in-range.
  • Percentiles recomputed to cover the newly-normed words.
  • Cost + coverage recorded; denylist decisions + false-positive spot-check documented.
  • One seed regen; staging verified before prod promotion.

Out of scope

  • Re-rating existing normed words with a newer model (separate effort).
  • socialness (unsurfaced), corpus-derived norms, grade bands.
  • Any frontend change — norms are already surfaced; this only fills coverage.

Risks

  • Over-dropping in the proper-dominant class. Mitigated by high PROPN-share threshold + curated allow-list + QA gate that fails on any leave-alone false-drop.
  • Build-script vocab regression. The --words-file injection must fully bypass the non-PROPN filter; a unit/smoke check on a 5-word kingdom-class sample (kingdom, march, frank) confirms PROPN-dominant words are actually rated.
  • Seed size / reseed. ~474 MB seed; reseed is gated on the manifest path. workflow_dispatch on deploy-staging reseeds unconditionally — trigger via the branch push, not dispatch.