Skip to content

Canonical Word List Redesign Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the flawed winner-take-all dominant-POS is_canonical gate with a two-sided rule (content-POS mass + a WordNet/frequency realness floor), and make canonical the universal candidate scope everywhere except sentence retrieval and exact-word resolution.

Architecture: Two independent components. Component 2 (Worker, Tasks 1–2) canonical-scopes the similarity and associations result sets at query time — no data change, ships first. Component 1 (data pipeline, Tasks 3–7) redefines is_canonical at the build root, validates thresholds with a probe, regenerates the D1 seed.

Tech Stack: TypeScript / Hono / Cloudflare Workers / D1 (cloudflare:test + SELF.fetch); Python 3 / Polars / nltk-WordNet; pytest.

Global Constraints

  • Spec: docs/superpowers/specs/2026-07-19-canonical-word-list-redesign-design.md. Jira: PHON-193.
  • is_canonical stays a boolean column on words and a derived flag on pairs. No schema change; existing indexes and the compileWordFilter w.is_canonical = 1 default are untouched.
  • No word is removed from the words table. Non-canonical words remain searchable by exact key and usable in similarity/sentence internals.
  • nltk + WordNet are developer-local build dependencies only. They never enter CI (CI consumes the LFS seed and runs no Python pipeline) and never reach the runtime/D1/client.
  • Terminology: "feature vectors," never "embeddings."
  • Content-POS set: {"NOUN", "VERB", "ADJ", "ADV"} — the single source of truth is CONTENT_POS in pipeline/canonical.py (Task 4); pipeline/words.py imports it, does not redefine it.
  • Governing scope rule: generated candidate sets are canonical-scoped; exact-word resolution (/api/words/:word, /api/audio/* target lookup) resolves any word; /api/sentences is the single full-vocabulary candidate-generating exception.
  • Root-cause discipline: fix at the pipeline/endpoint root, never patch a downstream leaf.
  • Commit after every task. Run the exact test command shown before committing. Branch: feature/phon-193-canonical-redesign (already created off develop).

File Structure

Component 2 (Worker): - Modify packages/web/workers/src/routes/similarity.ts — canonical-filter returned neighbors. - Modify packages/web/workers/src/routes/associations.ts — canonical-scope surfaced target lists. - Modify packages/web/workers/src/types.ts — declare is_canonical on WordRow. - Test packages/web/workers/test/similarity.spec.ts, packages/web/workers/test/associations.spec.ts (match existing test dir; see Task 1 for the exact path check).

Component 1 (data): - Create packages/data/src/phonolex_data/loaders/wordnet.py — load_wordnet_lemmas(). - Create packages/data/src/phonolex_data/pipeline/canonical.py — the rule + threshold constants. - Modify packages/data/src/phonolex_data/pipeline/words.py:270-279 (replace gate) and delete :392-412 (repair pass). - Modify packages/data/pyproject.toml — add nltk dev dependency. - Create research/2026-07-19-canonical-redesign/probe.py + README.md — threshold sweep + QA. - Test packages/data/tests/test_canonical.py, packages/data/tests/test_wordnet_loader.py.

Task 1: Canonical-scope similarity neighbors

Files: - Modify: packages/web/workers/src/types.ts:49-68 (add is_canonical to WordRow) - Modify: packages/web/workers/src/routes/similarity.ts:143-160 - Test: packages/web/workers/test/similarity.spec.ts (create if absent — first confirm the test dir)

Interfaces: - Consumes: fetchMergedWordRows(db, words, {requirePhonology}) → Map<string, WordRow> (unchanged); WordRow.is_canonical: number | null. - Produces: /api/similarity/search responses contain only is_canonical === 1 words. scoreSimilarityScan is unchanged (its other caller, the words/search intersection path, canonicalizes separately).

  • [ ] Step 1: Confirm the worker test directory + harness

Run: ls packages/web/workers/test/ 2>/dev/null || ls packages/web/workers/src/**/*.spec.ts Expected: a test/ dir with *.spec.ts files using cloudflare:test + SELF.fetch. Use the same path/pattern for the new spec. If similarity has an existing spec, add the test there instead of creating a new file.

  • [ ] Step 2: Write the failing test

In packages/web/workers/test/similarity.spec.ts (or the existing similarity spec), add:

import { SELF } from 'cloudflare:test';
import { describe, it, expect } from 'vitest';

describe('similarity canonical scope', () => {
  it('returns only canonical words as neighbors', async () => {
    const res = await SELF.fetch('https://example.com/api/similarity/search', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ word: 'cat', threshold: 0.5, limit: 50 }),
    });
    expect(res.status).toBe(200);
    const neighbors = await res.json<Array<{ word: { word: string; is_canonical?: number } }>>();
    expect(neighbors.length).toBeGreaterThan(0);
    // Every returned neighbor must be canonical. rowToWordResponse carries is_canonical
    // through (Step 4 ensures this); assert none are the known non-canonical junk class.
    for (const n of neighbors) {
      expect(n.word.is_canonical).toBe(1);
    }
  });
});

Note: if the seeded test D1 fixture has no non-canonical near-neighbors of cat, this test can pass vacuously. To make it meaningful, the fixture must include at least one non-canonical word phonologically near the anchor. Check packages/web/workers/test/ fixtures/seed; if none exists, add a non-canonical word (is_canonical=0) near cat (e.g. a proper-noun-tagged kat) to the fixture in this step.

  • [ ] Step 3: Run the test to verify it fails

Run: cd packages/web/workers && npm test -- similarity Expected: FAIL — a non-canonical neighbor appears (or is_canonical is undefined on the response).

  • [ ] Step 4: Declare is_canonical on WordRow and carry it through the response

In packages/web/workers/src/types.ts, inside interface WordRow (after has_image, line ~61) add:

  is_canonical: number | null;   // 0/1 — content-POS scope flag

Confirm rowToWordResponse (in lib/wordResponse.ts) includes is_canonical in its output; if it does not, add is_canonical: row.is_canonical to the returned object so the test can assert on it.

  • [ ] Step 5: Filter non-canonical candidates out of the returned list

In packages/web/workers/src/routes/similarity.ts, replace the comment + loop at lines 143–160. Change the availability skip to also drop non-canonical candidates:

  // Filter availability + canonical scope BEFORE taking `limit`: an unavailable
  // OR non-canonical candidate must not consume a slot — lower-ranked canonical
  // candidates fill the list. Sound similarity is a generated candidate set, so
  // it is canonical-scoped (PHON-193); the anchor word itself is resolved
  // separately and may be non-canonical.
  const out: Array<{ word: ReturnType<typeof rowToWordResponse>; similarity: number }> = [];
  const BATCH = 160;
  for (let i = 0; i < ranked.length && out.length < limit; i += BATCH) {
    const batch = ranked.slice(i, i + BATCH);
    const rowMap = await fetchMergedWordRows(c.env.DB, batch.map(([w]) => w), {
      requirePhonology: true,
    });
    for (const [w, sim] of batch) {
      const row = rowMap.get(w);
      if (!row || row.is_canonical !== 1) continue;
      out.push({ word: rowToWordResponse(row), similarity: sim });
      if (out.length >= limit) break;
    }
  }
  • [ ] Step 6: Run the test to verify it passes

Run: cd packages/web/workers && npm test -- similarity Expected: PASS.

  • [ ] Step 7: Run type-check + full worker suite

Run: cd packages/web/workers && npm run type-check && npm test Expected: PASS (no regressions).

  • [ ] Step 8: Commit
git add packages/web/workers/src/routes/similarity.ts packages/web/workers/src/types.ts packages/web/workers/src/lib/wordResponse.ts packages/web/workers/test/
git commit -m "feat(similarity): canonical-scope returned neighbors (PHON-193)"

Task 2: Canonical-scope association targets

Files: - Modify: packages/web/workers/src/routes/associations.ts (helper checkVocabulary → canonical semantics; all three handlers) - Test: packages/web/workers/test/associations.spec.ts

Interfaces: - Consumes: edges table, words.is_canonical. - Produces: /api/associations/:word, /api/associations/:word/confusability, and the shared list of /api/associations/compare contain only canonical target words. /compare's jaccard, word1_degree, word2_degree remain computed over the full target sets (graph metrics, not surfaced word lists).

  • [ ] Step 1: Write the failing test

In packages/web/workers/test/associations.spec.ts:

import { SELF } from 'cloudflare:test';
import { describe, it, expect } from 'vitest';

describe('associations canonical scope', () => {
  it('returns only canonical association targets', async () => {
    const res = await SELF.fetch('https://example.com/api/associations/cat');
    expect(res.status).toBe(200);
    const body = await res.json<{ associations: Array<{ target: string; in_vocabulary: boolean }> }>();
    // Every surfaced target must be a canonical vocabulary word.
    for (const a of body.associations) {
      expect(a.in_vocabulary).toBe(true);
    }
  });
});

Fixture note: as in Task 1, the seeded edges fixture for cat must include at least one non-canonical target for this test to be non-vacuous. Add one (edge cat → <non-canonical word>) to the fixture in this step if absent.

  • [ ] Step 2: Run the test to verify it fails

Run: cd packages/web/workers && npm test -- associations Expected: FAIL — a non-canonical target is returned (in_vocabulary false or a junk target present).

  • [ ] Step 3: Make checkVocabulary canonical-scoped

In packages/web/workers/src/routes/associations.ts, change the query in checkVocabulary (line 37-39) to restrict to canonical words, and rename for clarity:

/** Return which of `targets` are canonical vocabulary words (is_canonical=1). */
async function canonicalVocabulary(db: D1Database, targets: string[]): Promise<Set<string>> {
  if (!targets.length) return new Set();
  const vocabSet = new Set<string>();
  for (let i = 0; i < targets.length; i += 80) {
    const chunk = targets.slice(i, i + 80);
    const placeholders = chunk.map(() => '?').join(',');
    const { results } = await db.prepare(
      `SELECT word FROM words WHERE word IN (${placeholders}) AND is_canonical = 1`
    ).bind(...chunk).all<{ word: string }>();
    for (const r of results) vocabSet.add(r.word);
  }
  return vocabSet;
}
  • [ ] Step 4: Filter /:word results to canonical targets before pagination

In the /:word handler (lines 154-198), canonical-filter the full edge list before computing total and paginating so counts and pages reflect the canonical scope. Replace lines 173-190 with:

  const { results: allEdges } = await c.env.DB.prepare(sql).bind(...params).all<EdgeRow>();

  // Canonical-scope: association results are a generated candidate set (PHON-193).
  // Filter to canonical targets BEFORE pagination so total + pages reflect scope.
  const vocabSet = await canonicalVocabulary(c.env.DB, allEdges.map((r) => r.target));
  const canonicalEdges = allEdges.filter((r) => vocabSet.has(r.target));
  const total = canonicalEdges.length;
  const page = canonicalEdges.slice(offset, offset + limit);

  const typeCounts: Record<string, number> = {};
  const edgeResults: EdgeResponse[] = [];
  for (const row of page) {
    const resp = rowToEdgeResponse(row, vocabSet);   // in_vocabulary === true for all
    edgeResults.push(resp);
    for (const src of resp.edge_sources) {
      typeCounts[src] = (typeCounts[src] ?? 0) + 1;
    }
  }
  • [ ] Step 5: Filter /:word/confusability partners to canonical

In the confusability handler, after the seen map is fully built (line 137) and before sorting (line 139), replace the results line (139-141) with a canonical filter. Rename the existing vocabSet call at line 104 to use canonicalVocabulary:

  const vocabSet = await canonicalVocabulary(c.env.DB, allTargets);

then:

  // Sort by phoneme distance (lower = more clinically relevant), canonical only.
  const results = [...seen.values()]
    .filter((r) => vocabSet.has(r.target))
    .sort((a, b) => (a.eccc_phoneme_distance ?? 999) - (b.eccc_phoneme_distance ?? 999));
  • [ ] Step 6: Filter /compare surfaced shared list, keep metrics full-scope

In the /compare handler, the jaccard / word1_degree / word2_degree computations (lines 62-64, 74-75) stay over the full targets1/targets2 sets. Only the surfaced shared list is canonical-scoped. The existing vocabSet (line 67) already covers shared; rename its call to canonicalVocabulary and drop non-canonical shared words:

  // Metrics (jaccard, degrees) use the full association sets; only the surfaced
  // word list is canonical-scoped (PHON-193).
  const canonicalShared = await canonicalVocabulary(c.env.DB, shared);

  return c.json({
    word1,
    word2,
    shared: shared.filter((w) => canonicalShared.has(w)).sort()
      .map((w) => ({ word: w, in_vocabulary: true })),
    jaccard: Math.round(jaccard * 10000) / 10000,
    word1_degree: targets1.size,
    word2_degree: targets2.size,
  });
  • [ ] Step 7: Run the test to verify it passes

Run: cd packages/web/workers && npm test -- associations Expected: PASS.

  • [ ] Step 8: Type-check + full worker suite

Run: cd packages/web/workers && npm run type-check && npm test Expected: PASS.

  • [ ] Step 9: Commit
git add packages/web/workers/src/routes/associations.ts packages/web/workers/test/
git commit -m "feat(associations): canonical-scope surfaced targets, keep graph metrics full (PHON-193)"

Task 3: WordNet lemma loader + nltk dev dependency

Files: - Modify: packages/data/pyproject.toml (add nltk) - Create: packages/data/src/phonolex_data/loaders/wordnet.py - Test: packages/data/tests/test_wordnet_loader.py

Interfaces: - Produces: load_wordnet_lemmas() -> set[str] — the set of WordNet 3.0 lemma names, lowercased. (WordNet contains only open-class words, so membership already implies a content-POS reading.) Consumed by pipeline/canonical.py (Task 4) and the probe (Task 5).

  • [ ] Step 1: Add nltk to the data package dev dependencies

In packages/data/pyproject.toml, add nltk to the dependency list (match the existing formatting — it sits alongside lemminflect, simplemma, polars). Then:

Run: uv sync --all-packages && uv run python -m nltk.downloader wordnet omw-1.4 Expected: nltk installed; WordNet corpus downloaded to the local nltk_data dir.

  • [ ] Step 2: Write the failing test

packages/data/tests/test_wordnet_loader.py:

"""WordNet lemma-set loader — build-time realness signal for is_canonical (PHON-193)."""
import pytest


def test_load_wordnet_lemmas_returns_common_words():
    from phonolex_data.loaders.wordnet import load_wordnet_lemmas
    lemmas = load_wordnet_lemmas()
    assert isinstance(lemmas, set)
    assert len(lemmas) > 100_000          # WordNet 3.0 has ~155k lemma names
    assert "kingdom" in lemmas
    assert "cat" in lemmas
    assert "run" in lemmas


def test_load_wordnet_lemmas_is_lowercased():
    from phonolex_data.loaders.wordnet import load_wordnet_lemmas
    lemmas = load_wordnet_lemmas()
    assert all(x == x.lower() for x in list(lemmas)[:1000])
  • [ ] Step 3: Run the test to verify it fails

Run: uv run python -m pytest packages/data/tests/test_wordnet_loader.py -v Expected: FAIL — module phonolex_data.loaders.wordnet does not exist.

  • [ ] Step 4: Implement the loader

packages/data/src/phonolex_data/loaders/wordnet.py:

"""WordNet lemma-set loader.

Build-time only (nltk + WordNet are developer-local dependencies; nothing from
WordNet ships to D1 or the client). Provides the realness signal for the
is_canonical rule (PHON-193): a word that is a WordNet lemma is a real English
open-class word, which excludes foreign words, proper-noun junk, abbreviations,
and OCR fragments. WordNet contains only noun/verb/adj/adv synsets, so lemma
membership already implies a content-POS reading exists.
"""
from __future__ import annotations

from functools import lru_cache


@lru_cache(maxsize=1)
def load_wordnet_lemmas() -> set[str]:
    """Return all WordNet 3.0 lemma names, lowercased.

    Raises a clear error if the WordNet corpus is not downloaded.
    Run: python -m nltk.downloader wordnet omw-1.4
    """
    try:
        from nltk.corpus import wordnet as wn
        # Force corpus load so a missing download errors here, not lazily later.
        wn.ensure_loaded()
    except LookupError as e:  # corpus not downloaded
        raise RuntimeError(
            "WordNet corpus not found. Run: "
            "python -m nltk.downloader wordnet omw-1.4"
        ) from e
    return {name.lower() for name in wn.all_lemma_names()}
  • [ ] Step 5: Run the test to verify it passes

Run: uv run python -m pytest packages/data/tests/test_wordnet_loader.py -v Expected: PASS.

  • [ ] Step 6: Commit
git add packages/data/pyproject.toml packages/data/src/phonolex_data/loaders/wordnet.py packages/data/tests/test_wordnet_loader.py
git commit -m "feat(data): WordNet lemma-set loader + nltk dev dep (PHON-193)"

Task 4: canonical.py — the two-sided rule (TDD, injected thresholds)

Files: - Create: packages/data/src/phonolex_data/pipeline/canonical.py - Test: packages/data/tests/test_canonical.py

Interfaces: - Consumes: a record with attributes word: str, all_pos: list[str], all_freqs: list[int], frequency: float | None, contextual_diversity: float | None (the WordRecord fields); a set[str] of WordNet lemmas. - Produces: - CONTENT_POS: frozenset[str] — the content-POS source of truth. - CanonicalThresholds(tau: float, floor_abs: int, freq_floor: float, cd_floor: float) — frozen dataclass. - is_canonical(record, wordnet_lemmas: set[str], thr: CanonicalThresholds) -> bool. - assign_canonicality(words: dict[str, Any], wordnet_lemmas: set[str], thr: CanonicalThresholds) -> int — sets record.is_canonical on every record, returns the canonical count. - PRODUCTION_THRESHOLDS: CanonicalThresholds — filled by Task 5 (placeholder values here, overwritten after the probe).

Unit tests inject thresholds and a small lemma set; no nltk/model load needed (mirrors the FakeToken duck-typing in test_canonical_spacy.py).

  • [ ] Step 1: Write the failing tests

packages/data/tests/test_canonical.py:

"""Two-sided is_canonical rule (PHON-193): content-POS mass + WordNet/freq realness."""
from dataclasses import dataclass, field

import pytest

from phonolex_data.pipeline.canonical import (
    CONTENT_POS,
    CanonicalThresholds,
    is_canonical,
    assign_canonicality,
)


@dataclass
class Rec:
    """Duck-typed WordRecord stand-in (avoids building the full pipeline)."""
    word: str
    all_pos: list = field(default_factory=list)
    all_freqs: list = field(default_factory=list)
    frequency: float | None = None
    contextual_diversity: float | None = None
    is_canonical: bool = False


# Permissive thresholds for the unit tests; production values come from the probe.
THR = CanonicalThresholds(tau=0.10, floor_abs=50, freq_floor=1.0, cd_floor=0.0)
LEMMAS = {"kingdom", "cat", "run", "acrobat"}


def test_kingdom_canonical_despite_propn_dominance():
    """The core fix: real NOUN mass counts even though PROPN wins the argmax."""
    # 70% PROPN ("United Kingdom"), 30% NOUN — dominant POS is PROPN, but NOUN
    # mass share (0.30) clears tau (0.10) and the word is a WordNet lemma.
    r = Rec("kingdom", ["PROPN", "NOUN"], [700, 300], frequency=50.0,
            contextual_diversity=40.0)
    assert is_canonical(r, LEMMAS, THR) is True


def test_proper_noun_junk_excluded():
    """A name-only word absent from WordNet fails the realness gate."""
    r = Rec("zylthorpe", ["PROPN"], [900], frequency=50.0, contextual_diversity=40.0)
    assert is_canonical(r, LEMMAS, THR) is False


def test_abbreviation_excluded():
    """An abbreviation with content-POS mass but not a WordNet lemma is excluded."""
    r = Rec("aaa", ["NOUN"], [900], frequency=50.0, contextual_diversity=40.0)
    assert is_canonical(r, LEMMAS, THR) is False


def test_obscure_wordnet_word_below_freq_floor_excluded():
    """A real WordNet word too rare to be useful fails the frequency floor."""
    r = Rec("acrobat", ["NOUN"], [3], frequency=0.2, contextual_diversity=0.1)
    assert is_canonical(r, LEMMAS, THR) is False


def test_no_content_pos_mass_excluded():
    """A WordNet lemma whose corpus use is all function-word POS is excluded."""
    r = Rec("cat", ["DET", "ADP"], [500, 500], frequency=50.0, contextual_diversity=40.0)
    assert is_canonical(r, LEMMAS, THR) is False


def test_single_char_excluded():
    r = Rec("a", ["NOUN"], [9999], frequency=999.0, contextual_diversity=99.0)
    assert is_canonical(r, LEMMAS, THR) is False


def test_absolute_floor_path():
    """content-POS mass share below tau still qualifies via the absolute-count floor."""
    # 5% NOUN share (< tau 0.10) but 300 absolute NOUN count (>= floor_abs 50).
    r = Rec("run", ["PROPN", "NOUN"], [5700, 300], frequency=80.0,
            contextual_diversity=60.0)
    assert is_canonical(r, LEMMAS, THR) is True


def test_no_frequency_row_not_canonical():
    """A word with no corpus frequency fails realness (no freq → below floor)."""
    r = Rec("kingdom", [], [], frequency=None, contextual_diversity=None)
    assert is_canonical(r, LEMMAS, THR) is False


def test_assign_canonicality_sets_flag_and_counts():
    words = {
        "kingdom": Rec("kingdom", ["PROPN", "NOUN"], [700, 300], 50.0, 40.0),
        "zylthorpe": Rec("zylthorpe", ["PROPN"], [900], 50.0, 40.0),
    }
    n = assign_canonicality(words, LEMMAS, THR)
    assert n == 1
    assert words["kingdom"].is_canonical is True
    assert words["zylthorpe"].is_canonical is False


def test_content_pos_constant():
    assert CONTENT_POS == frozenset({"NOUN", "VERB", "ADJ", "ADV"})
  • [ ] Step 2: Run the tests to verify they fail

Run: uv run python -m pytest packages/data/tests/test_canonical.py -v Expected: FAIL — module phonolex_data.pipeline.canonical does not exist.

  • [ ] Step 3: Implement the module

packages/data/src/phonolex_data/pipeline/canonical.py:

"""Two-sided is_canonical rule (PHON-193).

Replaces the winner-take-all dominant-POS gate. A word is canonical iff BOTH:

  (A) Content-POS presence — its content-POS frequency MASS is meaningful, using
      the full all_pos/all_freqs distribution rather than the single dominant
      label. This fixes the `kingdom` class: a real NOUN reading counts even when
      PROPN ("United Kingdom") wins the argmax.

  (B) Realness floor — the word is a WordNet lemma (excludes foreign / proper-noun
      junk / abbreviations / OCR fragments) AND clears a corpus frequency floor
      (excludes the real-but-too-rare tail).

Single-character words are excluded regardless (CMU letter names).

A word with no corpus frequency row (all_freqs empty) has no content-POS mass and
no frequency, so it is non-canonical by both gates — WordNet membership alone is
never sufficient. Thresholds are chosen by the probe in
research/2026-07-19-canonical-redesign/ and pinned in PRODUCTION_THRESHOLDS.
"""
from __future__ import annotations

from dataclasses import dataclass
from typing import Any

CONTENT_POS: frozenset[str] = frozenset({"NOUN", "VERB", "ADJ", "ADV"})


@dataclass(frozen=True)
class CanonicalThresholds:
    tau: float        # min content-POS frequency-mass share (0-1)
    floor_abs: int    # OR-path: min absolute content-POS count
    freq_floor: float # min record.frequency
    cd_floor: float   # min record.contextual_diversity (CD_pct); 0 disables


def _content_pos_present(record: Any, thr: CanonicalThresholds) -> bool:
    total = sum(record.all_freqs)
    if total <= 0:
        return False
    content_counts = [
        f for p, f in zip(record.all_pos, record.all_freqs) if p in CONTENT_POS
    ]
    content_mass = sum(content_counts)
    if content_mass / total >= thr.tau:
        return True
    return bool(content_counts) and max(content_counts) >= thr.floor_abs


def _passes_realness(record: Any, wordnet_lemmas: set[str],
                     thr: CanonicalThresholds) -> bool:
    if record.word not in wordnet_lemmas:
        return False
    if (record.frequency or 0.0) < thr.freq_floor:
        return False
    if thr.cd_floor > 0.0 and (record.contextual_diversity or 0.0) < thr.cd_floor:
        return False
    return True


def is_canonical(record: Any, wordnet_lemmas: set[str],
                 thr: CanonicalThresholds) -> bool:
    return (
        len(record.word) > 1
        and _content_pos_present(record, thr)
        and _passes_realness(record, wordnet_lemmas, thr)
    )


def assign_canonicality(words: dict[str, Any], wordnet_lemmas: set[str],
                        thr: CanonicalThresholds) -> int:
    n = 0
    for record in words.values():
        record.is_canonical = is_canonical(record, wordnet_lemmas, thr)
        if record.is_canonical:
            n += 1
    return n


# Pinned by Task 5 (probe). Placeholder until then — DO NOT ship a build with
# these values; the probe overwrites them with data-chosen thresholds.
PRODUCTION_THRESHOLDS = CanonicalThresholds(
    tau=0.10, floor_abs=50, freq_floor=1.0, cd_floor=0.0,
)
  • [ ] Step 4: Run the tests to verify they pass

Run: uv run python -m pytest packages/data/tests/test_canonical.py -v Expected: PASS (all 10 tests).

  • [ ] Step 5: Commit
git add packages/data/src/phonolex_data/pipeline/canonical.py packages/data/tests/test_canonical.py
git commit -m "feat(data): two-sided is_canonical rule module + tests (PHON-193)"

Task 5: Threshold probe + human checkpoint

Files: - Create: research/2026-07-19-canonical-redesign/probe.py - Create: research/2026-07-19-canonical-redesign/README.md - Modify: packages/data/src/phonolex_data/pipeline/canonical.py (pin PRODUCTION_THRESHOLDS)

Interfaces: - Consumes: build_lexical_database() word records (or the current data/runtime/words.parquet for POS/freq columns), load_wordnet_lemmas(), is_canonical, and the ground-truth worklist research/2026-07-13-phon-186-morphynet-root/canonicality_inversions.parquet (columns: word, root, word_pos, root_pos; the root column is the should-be-canonical set, 890 rows with root_pos == "PROPN"). - Produces: a printed/CSV report of recall + precision + set-size delta per threshold config, and (after human review) the pinned PRODUCTION_THRESHOLDS.

This task is exploratory + human-gated, not a red/green TDD cycle. The deliverable is a recorded decision, not a passing assertion.

  • [ ] Step 1: Write the probe script

research/2026-07-19-canonical-redesign/probe.py:

"""Threshold sweep + QA for the is_canonical redesign (PHON-193).

Runs the candidate rule over the current lexicon at a small grid of thresholds
and reports, per config:
  - inversion recall: fraction of the 890 PROPN-class `root` words (the kingdom
    class) the rule makes canonical — should be high.
  - set size + churn vs today's is_canonical.
  - random samples of ADDED and DROPPED words for hand-QA of the four garbage
    classes (foreign / proper-noun / abbreviation / obscure).

Run:  uv run python research/2026-07-19-canonical-redesign/probe.py
"""
from itertools import product
from pathlib import Path

import polars as pl

from phonolex_data.loaders.wordnet import load_wordnet_lemmas
from phonolex_data.pipeline import build_lexical_database
from phonolex_data.pipeline.canonical import CanonicalThresholds, is_canonical

ROOT = Path(__file__).resolve().parents[2]
INVERSIONS = (ROOT / "research/2026-07-13-phon-186-morphynet-root"
              / "canonicality_inversions.parquet")


def main():
    print("Building lexical database ...")
    db = build_lexical_database()
    words = db.words
    lemmas = load_wordnet_lemmas()

    # Ground truth: PROPN-class base words that SHOULD become canonical.
    inv = pl.read_parquet(INVERSIONS)
    should_include = set(
        inv.filter(pl.col("root_pos") == "PROPN")["root"].to_list()
    )
    print(f"  ground-truth PROPN bases: {len(should_include)}")

    old_canonical = {w for w, r in words.items() if r.is_canonical}
    print(f"  current is_canonical count: {len(old_canonical):,}")

    # Grid — widen/narrow after the first pass.
    taus = [0.05, 0.10, 0.20]
    floors_abs = [25, 50, 100]
    freq_floors = [0.5, 1.0, 3.0]
    cd_floors = [0.0, 0.5]

    for tau, fa, ff, cd in product(taus, floors_abs, freq_floors, cd_floors):
        thr = CanonicalThresholds(tau=tau, floor_abs=fa, freq_floor=ff, cd_floor=cd)
        new_canonical = {
            w for w, r in words.items() if is_canonical(r, lemmas, thr)
        }
        recalled = len(should_include & new_canonical)
        added = new_canonical - old_canonical
        dropped = old_canonical - new_canonical
        print(
            f"tau={tau} floor_abs={fa} freq={ff} cd={cd} | "
            f"n={len(new_canonical):6d} "
            f"recall={recalled}/{len(should_include)} "
            f"(+{len(added)} / -{len(dropped)})"
        )

    # After picking a config, re-run this block with it to dump QA samples:
    #   chosen = CanonicalThresholds(tau=..., floor_abs=..., freq_floor=..., cd_floor=...)
    #   new = {w for w,r in words.items() if is_canonical(r, lemmas, chosen)}
    #   import random; random.seed(0)   # note: pipeline randomness is fine here
    #   print("ADDED sample:", random.sample(sorted(new - old_canonical), 40))
    #   print("DROPPED sample:", random.sample(sorted(old_canonical - new), 40))


if __name__ == "__main__":
    main()
  • [ ] Step 2: Run the sweep

Run: uv run python research/2026-07-19-canonical-redesign/probe.py Expected: a table of configs with recall + set-size + churn. Pick the config with high PROPN recall (target ≥ 85% of the 890) and a set size in a sane range.

  • [ ] Step 3: Dump QA samples for the leading config

Uncomment/adapt the QA block for the chosen config, re-run, and hand-inspect the ADDED sample (should be real content words, not the four garbage classes) and the DROPPED sample (should be junk/obscure, not useful words). Record counts.

  • [ ] Step 4: HUMAN CHECKPOINT — record the decision

Write research/2026-07-19-canonical-redesign/README.md documenting: the sweep table, the chosen thresholds, the recall achieved, the ADDED/DROPPED sample QA verdict, and the final canonical set-size delta vs the current 47K. This is the "no silent drops" record. Do not proceed until a human has reviewed the ADDED/DROPPED samples and approved the thresholds.

  • [ ] Step 5: Pin the chosen thresholds

Update PRODUCTION_THRESHOLDS in packages/data/src/phonolex_data/pipeline/canonical.py with the approved values (replace the placeholder). Re-run the unit tests to confirm nothing broke:

Run: uv run python -m pytest packages/data/tests/test_canonical.py -v Expected: PASS (tests inject their own thresholds; this change is data-only).

  • [ ] Step 6: Commit
git add research/2026-07-19-canonical-redesign/ packages/data/src/phonolex_data/pipeline/canonical.py
git commit -m "feat(data): pin is_canonical thresholds from probe + QA record (PHON-193)"

Task 6: Wire the new rule into build_words(), delete the repair pass

Files: - Modify: packages/data/src/phonolex_data/pipeline/words.py (replace :270-279, delete :392-412, remove the local _CONTENT_POS if now unused) - Test: packages/data/tests/test_canonical.py (add an integration test) OR packages/data/tests/test_pipeline.py

Interfaces: - Consumes: assign_canonicality, PRODUCTION_THRESHOLDS, load_wordnet_lemmas. - Produces: build_words() sets is_canonical via the new rule; the inversion-repair pass is gone.

  • [ ] Step 1: Write the failing integration test

Add to packages/data/tests/test_canonical.py:

def test_build_words_sets_canonical_via_new_rule(monkeypatch):
    """build_words wires assign_canonicality, not the old dominant-POS gate."""
    import phonolex_data.pipeline.words as words_mod
    # Sanity: the new symbols are imported into the module namespace.
    assert hasattr(words_mod, "assign_canonicality")
    assert hasattr(words_mod, "load_wordnet_lemmas")
    # The old repair-pass marker string must be gone (root fix, not band-aid).
    import inspect
    src = inspect.getsource(words_mod.build_words)
    assert "Repairing canonicality inversions" not in src
    assert "canonicality-inversion repair" not in src.lower()
  • [ ] Step 2: Run it to verify it fails

Run: uv run python -m pytest packages/data/tests/test_canonical.py::test_build_words_sets_canonical_via_new_rule -v Expected: FAIL — repair-pass text still present / symbols not imported.

  • [ ] Step 3: Replace the gate

In packages/data/src/phonolex_data/pipeline/words.py, add imports near the top (with the other pipeline imports):

from phonolex_data.loaders.wordnet import load_wordnet_lemmas
from phonolex_data.pipeline.canonical import (
    assign_canonicality,
    PRODUCTION_THRESHOLDS,
)

Replace the is_canonical block (lines 270-279) with:

    # Set is_canonical via the two-sided rule (PHON-193): content-POS MASS from
    # the full all_pos/all_freqs distribution + a WordNet/frequency realness
    # floor. Replaces the winner-take-all dominant-POS gate that excluded the
    # `kingdom` class and admitted foreign/proper/abbreviation/obscure junk.
    print("Setting is_canonical (content-POS mass + WordNet/freq realness) ...")
    wordnet_lemmas = load_wordnet_lemmas()
    n_canonical = assign_canonicality(words, wordnet_lemmas, PRODUCTION_THRESHOLDS)
    print(f"  canonical: {n_canonical:,} / {len(words):,} words")
  • [ ] Step 4: Delete the repair pass

Delete the entire "canonicality-inversion repair" block (lines 392-412, from the # PHON-186 canonicality-inversion repair. comment through the print(f" promoted {n_promoted:,} base forms ...")). The inversion class is now natively canonical, so the band-aid is dead code.

Remove the module-level _CONTENT_POS definition (lines 152-155) if no other code in the file references it (grep first: grep -n "_CONTENT_POS" packages/data/src/phonolex_data/pipeline/words.py). Note root population (lines 333-390) uses _CONTENT_UPOS, a separate local — leave it.

  • [ ] Step 5: Run the integration test + the canonical unit tests

Run: uv run python -m pytest packages/data/tests/test_canonical.py -v Expected: PASS.

  • [ ] Step 6: Run the broader data test suite (excluding the CI-ignored heavy tests)

Run: uv run python -m pytest packages/data/tests/ --ignore=packages/data/tests/test_datasets.py --ignore=packages/data/tests/test_new_loaders.py Expected: PASS (no regressions from removing the repair pass / _CONTENT_POS).

  • [ ] Step 7: Commit
git add packages/data/src/phonolex_data/pipeline/words.py packages/data/tests/test_canonical.py
git commit -m "feat(data): wire two-sided is_canonical, delete inversion-repair band-aid (PHON-193)"

Task 7: Regenerate the D1 seed + update CLAUDE.md invariant

Files: - Regenerate: data/runtime/*.parquet (gitignored cache) → packages/web/workers/scripts/d1-seed.sql (LFS) - Modify: CLAUDE.md (lexicon-scope paragraph + the is_canonical filter / compileWordFilter note)

Interfaces: - Produces: a seed whose words.is_canonical / pairs.is_canonical reflect the new rule. Deploy applies it (per the CI/deploy paradigm — CI does not regenerate).

  • [ ] Step 1: Rebuild the runtime parquets

Run: uv run python packages/data/scripts/build_runtime_parquet.py Expected: completes; prints the new canonical count (matches the probe's chosen set size). Confirm kingdom is now canonical:

Run: uv run python -c "import polars as pl; df=pl.read_parquet('data/runtime/words.parquet'); print(df.filter(pl.col('word')=='kingdom').select(['word','is_canonical']))" Expected: kingdom → is_canonical = true.

  • [ ] Step 2: Emit the D1 seed

Run: uv run python packages/web/workers/scripts/export-to-d1.py Expected: packages/web/workers/scripts/d1-seed.sql regenerated.

  • [ ] Step 3: Apply to local D1 and spot-check

Run (re-chunk + apply per CLAUDE.md dev-setup): 

uv run python packages/web/workers/scripts/chunk-seed-sql.py
cd packages/web/workers && for f in scripts/d1-chunks/chunk_*.sql; do npx wrangler d1 execute phonolex --local --file "$f"; done
Then verify against local D1: 
npx wrangler d1 execute phonolex --local --command "SELECT is_canonical FROM words WHERE word='kingdom'"
Expected: is_canonical = 1.

  • [ ] Step 4: Update the CLAUDE.md scope invariant

In CLAUDE.md, update the lexicon-scope wording so it no longer says similarity/corpus retrieval use the full vocabulary. Replace the relevant sentence in the opening scope paragraph and the is_canonical filter gotcha note with:

Generated candidate sets (Word Lists, Custom Word Lists, Contrast Sets, sound similarity, associations) filter to is_canonical=1. Sentence retrieval is the only full-vocabulary candidate-generating surface. Exact-word resolution (/api/words/:word, the /api/audio/* target lookup) resolves any word regardless of canonical status.

Also update the is_canonical filter pattern note to reflect that /api/similarity/search and /api/associations/* are now canonical-scoped (they no longer "bypass the helper").

  • [ ] Step 5: Commit the seed + docs (LFS)
git add packages/web/workers/scripts/d1-seed.sql CLAUDE.md
git commit -m "feat(data): regen D1 seed with new is_canonical + update scope invariant (PHON-193)"

Note: d1-seed.sql is the only LFS artifact; confirm git status shows it staged as LFS. Coordinate this regen with any other pending seed change so it remains one regen (per project convention).

Self-Review

Spec coverage: - Component 1 two-sided definition → Tasks 3 (WordNet), 4 (rule module), 6 (wiring). ✓ - Delete repair band-aid → Task 6 Step 4. ✓ - WordNet computed directly from all_lemma_names(), not the lemma_comparison column → Task 3 Step 4. ✓ - Content-POS mass from all_pos/all_freqs with absolute-floor OR-path → Task 4 _content_pos_present. ✓ - Realness = WordNet ∧ frequency floor (CD optional) → Task 4 _passes_realness. ✓ - nltk dev-only dependency → Task 3 Step 1 + Global Constraints. ✓ - Probe-chosen thresholds + QA against the 1,030-row worklist + add/drop sampling + set-size delta → Task 5. ✓ - Component 2 similarity scope → Task 1; associations scope with metrics preserved → Task 2. ✓ - /api/words/:word + /api/audio/* unchanged (exact-word resolution) → not modified by any task; called out in Global Constraints. ✓ - Sentence retrieval unchanged → not modified by any task. ✓ - CLAUDE.md invariant update → Task 7 Step 4. ✓ - Seed regen → Task 7. ✓

Refinement vs spec: the spec's "WordNet noun/verb/adj/adv synset fallback for no-frequency words" is intentionally dropped — WordNet lemmas are inherently open-class, and the realness floor already makes any no-frequency word non-canonical (Task 4 test_no_frequency_row_not_canonical locks this). The realness frequency signal uses the on-record frequency / contextual_diversity fields (from the in-house FineWeb-Edu corpus) rather than re-loading SUBTLEX, since those fields already carry CD_pct. Both refinements are recorded here and should be reflected in a one-line spec note.

Placeholder scan: PRODUCTION_THRESHOLDS is a deliberate, labeled placeholder pinned by Task 5 with a human checkpoint — not an unfilled plan step. No other placeholders.

Type consistency: CanonicalThresholds(tau, floor_abs, freq_floor, cd_floor), is_canonical(record, wordnet_lemmas, thr), assign_canonicality(words, wordnet_lemmas, thr), load_wordnet_lemmas() -> set[str] used consistently across Tasks 3–6. Worker: WordRow.is_canonical: number | null, canonicalVocabulary(db, targets) used consistently across Tasks 1–2.