Skip to content

Canonical Word List Redesign

Date: 2026-07-19 Status: Design approved, pending spec review Jira: PHON-193 (to be created) Predecessor: PHON-186 (Done) — diagnosed the root cause and deferred the rule change to this ticket

Problem

The is_canonical flag defines the user-facing vocabulary (Word Lists, Contrast Sets, Lookup browse). It is currently produced by a winner-take-all dominant-POS gate in packages/data/src/phonolex_data/pipeline/words.py (lines 270–279):

record.is_canonical = (record.pos in _CONTENT_POS and len(record.word) > 1)

where record.pos is the single most-frequent POS for the word from the FineWeb-Edu spaCy parse. This is flawed in two directions:

  1. False negatives (the documented kingdom class). A common content word whose corpus occurrences are dominated by proper-noun contexts gets the wrong dominant POS and is excluded. kingdom → PROPN (via "United Kingdom", "Kingdom of God"); baker → PROPN (the surname); downtown, etc. PHON-186 enumerated 1,030 such inversions (research/2026-07-13-phon-186-morphynet-root/canonicality_inversions.parquet), 890 of them the PROPN class. A band-aid repair pass (lines 392–412) only rescues a base form when one of its inflections survived as canonical — so kingdom returns only because kingdoms happened to be tagged NOUN. Fragile and incomplete.

  2. False positives (garbage the user sees). Being tagged with a content-POS dominant label is too weak a positive signal. Foreign words, proper-noun junk tagged as common NOUN, abbreviations/initialisms/OCR fragments, and legitimately real-but-far-too-rare words all pass the gate and flood the outputs.

Additionally, canonical is not applied consistently across the app. Sound similarity and word associations run over the full ~125K vocabulary, so their result sets are flooded with the same non-canonical junk — even though there is no reason those surfaces should differ from Word Lists.

Goals

  • Redefine is_canonical so it is a positive statement — "this is a real, reasonably common English content word" — that fixes both failure directions at the pipeline root.
  • Make canonical the vocabulary scope for every generated result set across the app, with sentence retrieval the single deliberate exception.
  • Pick thresholds from data via a probe, and QA the resulting set against ground truth — no blind magic numbers.

Non-Goals

  • Redistributing any external lexical resource. WordNet is used at build time to derive a boolean; it never ships to D1 or the client (same posture as PHON-186's MorphyNet use).
  • Changing the similarity algorithm, the edges/associations graph construction, or the sentence-retrieval ranking. Only the candidate scope changes.
  • Removing any word from the words table. Non-canonical words stay fully present (searchable by exact lookup, usable in similarity/sentence internals); they are just not offered as browse/generate candidates.

Design

Two independent components. Component 1 is a data-pipeline change requiring a seed regen; Component 2 is a Worker query-time change that ships independently.

Component 1 — Two-sided canonical definition (data pipeline)

Replace the dominant-POS gate and delete the inversion-repair band-aid (lines 392–412 become unnecessary — the inversion class is now natively canonical). Extract the logic into a single focused, testable module (packages/data/src/phonolex_data/pipeline/canonical.py) that build_words() calls after norms are merged.

A word is_canonical iff both conditions hold:

(A) Content-POS presence — replaces winner-take-all with the full POS distribution the word already carries (all_pos / all_freqs, populated by load_phonolex_frequency):

content_mass = sum(all_freqs[i] for i where all_pos[i] in {NOUN, VERB, ADJ, ADV})
total_mass   = sum(all_freqs)
present = (total_mass > 0 and content_mass / total_mass >= TAU)
          or (max content-POS absolute count >= FLOOR_ABS)

A word with no FineWeb-Edu frequency row (all_pos empty) has no content-POS mass, so it is not canonical — and the realness floor below independently excludes it (no frequency → below the floor). WordNet membership alone is never sufficient; there is no separate no-frequency fallback (WordNet lemmas are inherently open-class, so a synset-POS fallback would add nothing the realness gate doesn't already require).

This makes kingdom's real NOUN mass count even though PROPN "wins" the single-label argmax, and it does so uniformly rather than via inflection-dependent rescue.

(B) Realness floor — the positive signal that excludes the four garbage classes:

real = (word in REAL_WORDS) and (frequency_signal >= FREQ_FLOOR)

REAL_WORDS is the subset of our vocabulary that is a genuine English open-class word, precomputed at build time from WordNet. The 2026-07-19 threshold probe (research/2026-07-19-canonical-redesign/) showed a naive word in all_lemma_names() check fails in both directions, so REAL_WORDS is built with two corrections:

  • Lemma-normalize (fixes false drops). WordNet stores only base lemmas, so a raw surface-form check drops inflected content words (flew, mountains, toddlers, represents) — ~80% of the probe's dropped sample was legitimate inflections. A word is real if word or its nltk.corpus.wordnet.morphy(word) base form is a WordNet common lemma. (morphy returns a base lemma only when it exists in WordNet, so mountains → mountain, flew → fly.)
  • Exclude proper-noun instances (fixes false adds). WordNet carries named entities as instance synsets (england, venice, aristotle are lemmas), so plain membership leaks proper nouns — and the probe showed a frequency floor makes this worse (frequent names like brussels, pepsi clear any floor). WORDNET_COMMON_LEMMAS therefore keeps only lemmas with at least one non-instance synset (syn.instance_hypernyms() empty), dropping pure named entities while keeping words that also have a common sense (china the porcelain, mercury the metal).
  • Do NOT reuse the wordnet column in research/2026-07-13-phon-186-morphynet-root/lemma_comparison.parquet — that column is a lemmatizer output (null when unresolvable), not clean membership.
  • frequency_signal = the on-record frequency / contextual_diversity (CD_pct) fields already populated on every WordRecord from the in-house FineWeb-Edu corpus (no SUBTLEX re-load). Its job is only to trim the real-but-obscure tail — NOT to filter proper nouns (WordNet-instance exclusion does that). The probe found cd_floor=0.5 collapses the set and is not a usable lever at that value; floors are chosen by the re-run probe.

canonical.py stays pure: it receives the precomputed real_words set and checks simple membership. The morphy-normalization and instance-exclusion live in the WordNet loader (build_real_word_set(vocab, common_lemmas) + load_wordnet_common_lemmas()).

The len(word) > 1 single-character exclusion is retained (folded into the module).

Retained downstream shape: is_canonical stays a boolean column on words and a derived flag on pairs (a pair is canonical iff both members are). No schema change; the existing indexes and compileWordFilter default (w.is_canonical = 1) are untouched.

Dependency: add nltk + the WordNet corpus as a developer-local build dependency only. The build is developer-local (CI consumes the LFS seed and runs no Python pipeline), so this never enters CI or the runtime. WordNet 3.0's license permits commercial use; build-time boolean derivation is not redistribution.

Component 2 — Canonical as the universal candidate scope (Worker)

The governing rule:

  • Generated candidate sets are canonical-scoped — anything that produces a set of suggested words (similarity neighbors, association targets, Word Lists, Contrast Sets).
  • Exact-word resolution resolves any word — endpoints that look up a specific typed word by key (/api/words/:word, the /api/audio/* target lookup) are not candidate sets and stay full-vocabulary.
  • Sentence retrieval is the single candidate-generating exception — it stays full-vocabulary for the reasons below.

Expressed as an explicit per-endpoint policy (the scope decision belongs at the endpoint — this is the root expression of the policy, not a leaf patch). Query-time filtering; no table re-emit, so this ships independently of the Component 1 seed regen and is fully reversible.

Endpoint Change
/api/similarity/search (routes/similarity.ts) Restrict the neighbor candidate pool to is_canonical = 1. The route currently loads the entire word_syllables table as its search corpus with no canonical filter (line 54). Arbitrary query words still resolve their own phoneme decomposition on the fly (from words.phonemes); only the returned neighbors are canonical-scoped.
/api/associations/* (routes/associations.ts) Return only canonical targets — join returned edge targets against words.is_canonical = 1.
/api/words/:word (routes/words.ts) Unchanged. Direct exact-word lookup still resolves any typed word (line 34), so a user can look up a proper noun or rare word. Its similar-words / associations panels inherit canonical scoping from the two rows above.
/api/audio/* (routes/audio.ts) Unchanged. The audio target word is resolved by exact key (SELECT phonemes FROM words WHERE word = ?, line 345) — a clinician may score a production against any target (proper noun, rare word). Like /api/words/:word this is exact-word resolution, not a generated candidate pool, so it must stay full-vocabulary. Do not wrap a canonical filter around the target lookup.
/api/sentences (routes/sentences.ts) Unchanged — the single deliberate candidate-generating exception. For sentence retrieval, attestation and sentence naturalness outrank word canonicality; a natural attested sentence is the unit of value, not its constituent words. Non-canonical junk also self-filters here because it rarely appears in corpus sentences.
Contrast Sets / pairs (routes/contrastive.ts) Unchanged — already is_canonical = 1 filtered.

Threshold selection + validation (probe)

Thresholds (TAU, FLOOR_ABS, the frequency signal + FREQ_FLOOR) are not hardcoded blind. A probe script (research/2026-07-19-canonical-redesign/) sweeps candidate values and reports, for each configuration:

  • "Should be included" recall: fraction of the 1,030-row canonicality_inversions.parquet worklist (the known kingdom class) that the new rule makes canonical. Target: the PROPN class (890) should be recovered.
  • "Should be excluded" precision: a random sample of words the new rule adds vs. today's 47K, and a random sample of words it drops, hand-inspected for the four garbage classes (foreign / proper-noun / abbreviation / obscure). Report counts and set sizes so any coverage change is explicit (per "no silent drops").
  • Set-size delta: total canonical count before/after, and the churn (added / dropped).

The probe output picks the thresholds; the chosen values and the QA sample are recorded in the research notes and referenced from the ticket before the production build runs.

Build & rollout sequence

  1. Land Component 2 (Worker query-time canonical scoping) + tests — ships independently, no seed change.
  2. Build the probe, pick thresholds, QA the delta.
  3. Implement canonical.py, wire into build_words(), delete the repair pass, add the nltk/WordNet dev dependency, add Python tests.
  4. Local pipeline regen → export-to-d1.py → d1-seed.sql (LFS) → deploy applies it. Coordinate with any other pending seed change so it remains one regen.
  5. Update CLAUDE.md: the canonical-scope invariant changes from "sound similarity and corpus retrieval use the full vocabulary" → generated candidate sets are canonical-scoped; sentence retrieval is the only full-vocabulary candidate-generating surface; exact-word resolution (/api/words/:word, audio target lookup) resolves any word. Update the lexicon-scope paragraph and the compileWordFilter note accordingly.

Testing

  • Python (packages/data/tests/): unit tests for canonical.py — the kingdom case (PROPN dominant, real NOUN mass → canonical), a proper-noun-only word (no WordNet lemma → not canonical), an abbreviation (not a WordNet lemma → not canonical), an obscure WordNet word below the frequency floor (→ not canonical), and the single-char exclusion.
  • Worker (packages/web/workers/, cloudflare:test + SELF.fetch): similarity and associations responses contain only is_canonical = 1 words; /api/words/:word still resolves a non-canonical word directly; /api/sentences still returns sentences containing non-canonical surfaces.

Risks / open questions

  • WordNet recall gaps. WordNet lacks some legitimate content words (very new coinages, some compounds). The realness floor is WordNet AND frequency (per the approved "strict" option); the probe's drop-sample QA quantifies how many real words this costs. If it is material, revisit toward "WordNet OR high-frequency."
  • Coordination with pending seed regens. Component 1 requires a seed regen; batch it with any other pending data change to keep it a single regen (per project convention).