Phonemic Stress Separation Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Map AH/ER to one IPA symbol each, carry stress in a parallel index-aligned column, and delete the three places that currently have to know ə~ʌ is not a contrast.

Architecture: Stress stops being encoded in vowel identity. arpa_to_ipa collapses AH*→ə and ER*→ɚ, dropping the search inventory from 41 to 39. Two new columns carry stress beside an untouched phonemes_str: phonemes_stress_str fuses the stress digit onto each nucleus (|ə0|), and syllable_stress_str holds one slot per syllable. So schwa stays searchable as unstressed /ə/ and stress position becomes queryable for the first time. ipa still renders the stressed allophone (/kʌp/, /bɝd/) from syllable stress, so nothing a reader sees regresses. The 181 false pairs then cannot exist, so the suppression rule and its two mirrors are removed rather than kept.

Tech Stack: Python 3.12 + Polars (packages/data), pytest; Hono + TypeScript + D1 (packages/web/workers), vitest with cloudflare:test; React + MUI (packages/web/frontend).

Spec: docs/superpowers/specs/2026-09-13-phonemic-stress-separation-design.md

Global Constraints

• Terminology: "feature vectors," never "embeddings."
• ə and ɚ are the phonemic symbols. ʌ and ɝ survive only as render-time allophones in ipa and as entries in the learned feature vectors. They must not appear in phonemes, phonemes_str, variants_str, pairs, or the phonemes table.
• ipa is display-only — verified in PR #283: every consumer wraps it in slashes, matching runs on phonemes_str. Its rendered form must not regress.
• Do NOT annotate phonemes_str in place. Every pattern query is a LIKE against it; %|æ|% becoming |æ1| would silently break every vowel search. Stress goes in a parallel column.
• syllable_count stays max-across-variants (words.py:203). Unrelated to this change, in the same file, do not touch.
• D3 is measured, not decided (Task 6). Shipping an unmeasured change to similarity ranking is out of bounds.
• The learned vectors keep both ə and ʌ. They are distinct realizations (z = 16.8); only the phonemic layer collapses.
• D1: 100 bind params/query, 100 columns/table. words gains exactly one column.
• Branch: feature/phonemic-stress-separation off develop; PR targets develop.
• Don't commit d1-seed.sql, data/runtime/*.parquet, or scripts/d1-chunks/.

File Structure

file	responsibility	change
data/mappings/arpa_to_ipa.json	ARPAbet → IPA	AH*→ə, ER*→ɚ
phonology/stress.py	new — phonemes_stress_str + syllable_stress_str	create
phonology/ipa_render.py	render stress marks and the stressed allophone	extend
runtime/emit_parquet.py	emit both stress columns	extend
runtime/emit_d1_sql.py	both columns + schema + DDL	extend
phonology/allophones.py	—	delete
config/allophones.ts	—	delete
pipeline/derived.py	drop the suppression call	edit
lib/wordFilter.ts	stress-aware pattern predicate	extend
research/2026-09-13-similarity-baseline/	D3 measurement harness	create

---

Task 1: Capture the similarity baseline before anything changes

This is the only chance to record "before". Task 2 changes phonemes_str for 12,259 words, which moves similarity scores; without a committed baseline, D3 cannot be measured.

Files: - Create: research/2026-09-13-similarity-baseline/capture_baseline.py - Create: research/2026-09-13-similarity-baseline/baseline.json (committed)

Interfaces: - Consumes: POST /api/similarity/search on a running dev worker against the seeded local D1. - Produces: baseline.json — { [word]: [[neighbour, score], ...] }, top 20 per probe, scores at 4dp. - Consumed by: Task 6, which re-runs the identical probes and diffs.

EXECUTED 2026-09-13. The planned vitest approach was abandoned for cause.

vitest-pool-workers supplies an ephemeral, empty D1 and silently ignores d1Persist — it manages isolated storage per test file by design. Verified empirically: a probe test pointed at a copy of the seeded database still reported no such table: words, and running against an empty persist directory wrote nothing into it. So an in-pool baseline could only have been assembled from fixture rows, which cannot measure ranking movement across 48,782 canonical words — the thing the baseline exists for.

Captured through the live dev worker over HTTP instead, which uses the real scoreSimilarityScan against the real seeded data with no divergence risk:

cd packages/web/workers && npm run dev          # :8787, seeded local D1
uv run python research/2026-09-13-similarity-baseline/capture_baseline.py

Result: 24 probes × 20 neighbours = 480 ranked rows. The probes that will move are already legible in it — cup→cop is a ʌ~ɑ nucleus contrast today and becomes ə~ɑ after the collapse. The script refuses to write a hollow baseline: any probe returning zero neighbours aborts with a non-zero exit rather than recording an empty list.

Task 2: Collapse AH/ER to one symbol, render the allophone

Files: - Modify: data/mappings/arpa_to_ipa.json - Modify: packages/data/src/phonolex_data/phonology/ipa_render.py - Test: packages/data/tests/test_ipa_render.py (extend), packages/data/tests/test_collapse.py (new)

Interfaces: - Produces: render_ipa(syllables) unchanged in signature, but now substitutes the stressed allophone — ə→ʌ and ɚ→ɝ in syllables whose stress is 1 or 2. - Consumed by: _build_phonological_record, unchanged at the call site.

Rule: the map emits only ə and ɚ. render_ipa reintroduces ʌ/ɝ for display, keyed on syllable stress — the single place the allophony lives.

• [ ] Step 1: Write the failing tests

# packages/data/tests/test_collapse.py
"""AH/ER collapse to one phoneme each (spec 2026-09-13 §3.1)."""
from phonolex_data.loaders.cmudict import cmudict_to_phono
from phonolex_data.mappings import load_arpa_to_ipa
from phonolex_data.pipeline.words import _build_phonological_record


def test_arpa_map_emits_one_symbol_per_ah_and_er():
    m = load_arpa_to_ipa()
    assert {m[k] for k in ("AH", "AH0", "AH1", "AH2")} == {"ə"}
    assert {m[k] for k in ("ER", "ER0", "ER1", "ER2")} == {"ɚ"}


def test_no_wedge_or_stressed_rhotic_in_phonemes():
    """The phonemic layer carries ə/ɚ only; ʌ/ɝ are display allophones."""
    phono = cmudict_to_phono()
    for word in ("cup", "bus", "bird", "work", "abduction", "ablate"):
        ph = phono[word]["phonemes"]
        assert "ʌ" not in ph, f"{word}: {ph}"
        assert "ɝ" not in ph, f"{word}: {ph}"


def test_abduction_has_the_same_phoneme_twice():
    """The word that proved the double encoding: one CMU phone AH, twice."""
    phono = cmudict_to_phono()
    ph = phono["abduction"]["phonemes"]
    assert ph.count("ə") == 2, ph


def test_display_still_shows_the_stressed_allophone():
    phono = cmudict_to_phono()
    cases = {"cup": "kʌp", "bird": "bɝd", "about": "əˈbaʊt", "letter": "ˈlɛtɚ"}
    for word, want in cases.items():
        rec = _build_phonological_record({**phono[word], "word": word})
        assert rec.ipa == want, f"{word}: {rec.ipa}"


def test_ablate_vowel_is_now_correct():
    """PR #283 rendered /ˌʌˈbleɪt/; the wrong vowel was our AH2 mapping, not CMU's.
    CMU's spurious secondary stress remains as the ˌ — that is out of scope."""
    phono = cmudict_to_phono()
    rec = _build_phonological_record({**phono["ablate"], "word": "ablate"})
    assert "ʌ" not in rec.phonemes
    assert rec.ipa == "ˌəˈbleɪt", rec.ipa

• [ ] Step 2: Run to verify they fail

Run: uv run python -m pytest packages/data/tests/test_collapse.py -v Expected: FAIL — the map still returns ʌ for AH1.

• [ ] Step 3: Collapse the map

uv run python - <<'PY'
import json
from pathlib import Path
p = Path('data/mappings/arpa_to_ipa.json')
m = json.loads(p.read_text(encoding='utf-8'))
for k in ("AH", "AH0", "AH1", "AH2"):
    m[k] = "ə"
for k in ("ER", "ER0", "ER1", "ER2"):
    m[k] = "ɚ"
p.write_text(json.dumps(m, ensure_ascii=False, indent=2) + "\n", encoding='utf-8')
print("collapsed:", {k: m[k] for k in ("AH0","AH1","AH2","ER0","ER1","ER2")})
PY

• [ ] Step 4: Render the allophone in ipa_render.py

Add above render_ipa:

#: Stressed realizations of the two reduced vowels. The phonemic layer carries
#: ə and ɚ only (spec §3.1); the stressed allophones exist for DISPLAY, so a
#: reader still sees /kʌp/ and /bɝd/ rather than /kəp/ and /bɚd/. This is the
#: single place the allophony lives — it used to be baked into arpa_to_ipa,
#: which put stress into vowel identity and manufactured 181 false contrasts.
STRESSED_ALLOPHONE = {"ə": "ʌ", "ɚ": "ɝ"}

In render_ipa, replace the nucleus append with a stress-conditioned one:

        stressed = stress in (1, 2)
        parts.append("".join(syl.get("onset") or []))
        nucleus = str(syl.get("nucleus") or "")
        parts.append(STRESSED_ALLOPHONE.get(nucleus, nucleus) if stressed else nucleus)
        parts.append("".join(syl.get("coda") or []))

Note the mark_at_all guard applies only to the ˈ/ˌ marks — a monosyllable takes no mark but does take the allophone, so cup renders kʌp, not kəp. Confirm test_display_still_shows_the_stressed_allophone covers that (it does: cup).

• [ ] Step 5: Run both suites

Run: uv run python -m pytest packages/data/tests/test_collapse.py packages/data/tests/test_ipa_render.py -v Expected: PASS. The PR #283 invariant test test_ipa_minus_stress_marks_equals_the_phoneme_sequence will now fail — ipa minus stress marks contains ʌ where phonemes has ə. That invariant is superseded; replace it with:

def test_ipa_minus_stress_and_allophony_equals_the_phoneme_sequence():
    """Superseded form of the PR #283 invariant. `ipa` now differs from
    `phonemes` by stress marks AND the stressed allophone, both derived from
    syllable stress — so undo both before comparing."""
    from phonolex_data.phonology.ipa_render import STRESSED_ALLOPHONE
    inverse = {v: k for k, v in STRESSED_ALLOPHONE.items()}
    phono = cmudict_to_phono()
    mismatches = []
    for word in list(phono)[:4000]:
        rec = _build_phonological_record({**phono[word], "word": word})
        bare = rec.ipa
        for m in "ˈˌ":
            bare = bare.replace(m, "")
        for a, b in inverse.items():
            bare = bare.replace(a, b)
        if bare != "".join(rec.phonemes):
            mismatches.append((word, bare, "".join(rec.phonemes)))
    assert not mismatches, mismatches[:10]

• [ ] Step 6: Commit

git add data/mappings/arpa_to_ipa.json \
        packages/data/src/phonolex_data/phonology/ipa_render.py \
        packages/data/tests/test_collapse.py packages/data/tests/test_ipa_render.py
git commit -m "fix(phonology): one phoneme for AH and ER; allophone is display-only

CMU has a single AH phone and the digit marks stress, but arpa_to_ipa split it
into ə and ʌ — putting stress into vowel identity. abduction was
|æ|b|d|ʌ|k|ʃ|ə|n|: the same phone twice, as two phonemes.

AH* -> ə and ER* -> ɚ. ʌ and ɝ become render-time allophones in ipa_render,
keyed on syllable stress, so /kʌp/ and /bɝd/ read unchanged. Inventory 41 -> 39.

ə/ɚ chosen over ʌ/ɝ on diff size: 12,259 words rewritten rather than 70,758,
and it matches the only label packages/audio ever emits.

ablate is now /ˌəˈbleɪt/ — PR #283 blamed CMU for the wrong vowel, but AH2->ʌ
was ours. The spurious ˌ is CMU's and stays out of scope.

Spec: docs/superpowers/specs/2026-09-13-phonemic-stress-separation-design.md §3.1, §3.3"

---

Task 3: Emit the stress columns — EXECUTED, format corrected

Files: - Create: packages/data/src/phonolex_data/phonology/stress.py - Modify: runtime/emit_parquet.py, runtime/schema.py, runtime/emit_d1_sql.py - Test: packages/data/tests/test_stress_str.py

Interfaces: - Produces: phonemes_stress_str(syllables, phonemes) -> str | None — |æ0|b|d|ə1|k|ʃ|ə0|n|, the stress digit fused onto each nucleus, consonants bare. - Produces: syllable_stress_str(syllables) -> str | None — |0|1|0|, one slot per syllable. - Both raise ValueError on misalignment rather than fusing stress onto the wrong segment.

EXECUTED 2026-09-14. The planned format was wrong and was replaced before Task 5 built on it.

This task originally specified a parallel index-aligned column, |2|-|-|1|-|, one slot per phoneme. Task 5 revealed it is unqueryable: pattern matching is LIKE on a pipe-delimited string with no positional indexing (patterns.ts expresses position only as STARTS_WITH / ENDS_WITH / CONTAINS, plus the indexed initial_phoneme / final_phoneme columns). Pinning a phoneme and a stress value to the same slot would have needed two independent LIKEs that can match at different positions — returning words that look right and are not.

Delivered instead:

phonemes_str          |æ|b|d|ə|k|ʃ|ə|n|        untouched — every pattern query LIKEs this
phonemes_stress_str   |æ0|b|d|ə1|k|ʃ|ə0|n|    one LIKE gives sound + stress
syllable_stress_str   |0|1|0|                  stress POSITION needs syllable boundaries

abduction's two schwas — the pair that proved the double encoding — are now told apart by the digit, where they used to be told apart by being written ʌ and ə.

The lexicon-wide alignment test runs over live pipeline output, not data/runtime/words.parquet: reading the parquet would skip whenever the build cache predates the columns, and a test that usually skips is not a guarantee.

Task 4: Delete the three allophone special cases

The pairs can no longer exist, so the rules that suppressed them are dead weight. Deleting them is the point of the branch — the suppression was a patch on a symptom.

Files: - Delete: packages/data/src/phonolex_data/phonology/allophones.py, packages/data/tests/test_allophones.py - Delete: packages/web/workers/src/config/allophones.ts, packages/web/workers/src/__tests__/allophoneContrast.test.ts - Modify: packages/data/src/phonolex_data/pipeline/derived.py, packages/data/tests/test_derived.py, packages/web/workers/src/routes/contrastive.ts

Interfaces: removes is_stress_allophone_pair, STRESS_ALLOPHONES, isStressAllophonePair, notAContrastBody. No replacement — nothing downstream needs to know.

• [ ] Step 1: Write the test that proves the rules are unnecessary

# packages/data/tests/test_collapse.py — append
def test_no_allophone_pair_can_be_generated_without_a_rule():
    """The point of the branch: ə~ʌ pairs are impossible by construction, so no
    suppression rule is needed. ʌ is not a phoneme, so no word contains it."""
    import numpy as np
    from phonolex_data.pipeline.derived import _compute_minimal_pairs
    from phonolex_data.pipeline.schema import WordRecord

    words = {
        name: WordRecord(word=name, has_phonology=True, phonemes=list(ph),
                         phoneme_count=len(ph), syllable_count=1)
        for name, ph in {"aaa": ["k", "ə", "t"], "bbb": ["k", "ɪ", "t"]}.items()
    }
    inv = {p for r in words.values() for p in r.phonemes}
    rows = _compute_minimal_pairs(
        words, vectors={p: np.array([0.0]) for p in inv}, feature_names=["sonorant"],
    )
    subs = [(r[2], r[3]) for r in rows if r[8] == "substitution"]
    assert any({a, b} == {"ə", "ɪ"} for a, b in subs), subs
    assert not any("ʌ" in (a, b) or "ɝ" in (a, b) for a, b in subs), subs

• [ ] Step 2: Run it — it should pass before the deletion too

Run: uv run python -m pytest packages/data/tests/test_collapse.py::test_no_allophone_pair_can_be_generated_without_a_rule -v Expected: PASS. That is the signal the rules are now dead — the guarantee holds structurally.

• [ ] Step 3: Delete the Python side

git rm packages/data/src/phonolex_data/phonology/allophones.py \
       packages/data/tests/test_allophones.py

In pipeline/derived.py, remove the import and the three-line suppression block, restoring the docstring's "One rule, one pass, no exceptions."

In tests/test_derived.py, revert the oracle to its pre-exception form: drop the is_stress_allophone_pair import and its guard, delete test_generator_equals_brute_force_with_stress_allophones, and restore test_one_phoneme_words_pair_like_any_other to expect ("a", "uh", ...) only if uh still differs from a by one phoneme — under the collapse both are ["ə"], so they are now homophones and the pair is a presence/identity case, not a substitution. Check the actual phonemes before rewriting the assertion.

• [ ] Step 4: Delete the TypeScript side

git rm packages/web/workers/src/config/allophones.ts \
       packages/web/workers/src/__tests__/allophoneContrast.test.ts

In routes/contrastive.ts, remove the import and the guard block. A request for ə~ʌ now falls through to a normal query; since ʌ is not in the inventory it returns an empty list. That is a silent drop — the thing PR #283 added the 422 for. So it needs the generic replacement, not nothing:

  // ʌ and ɝ are no longer phonemes — they are display allophones of ə and ɚ
  // (spec 2026-09-13 §3.1). A request naming one is a request for a symbol
  // outside the 39-phoneme inventory, so answer as such rather than returning
  // an empty list. This replaces the bespoke stress-allophone guard, which is
  // no longer needed now that the pairs cannot exist.
  for (const [label, ph] of [['phoneme1', p1], ['phoneme2', p2]] as const) {
    if (!(await isKnownPhoneme(c.env.DB, ph))) {
      return c.json({
        error: 'unknown_phoneme',
        detail: `/${ph}/ is not one of the 39 phonemes PhonoLex indexes. `
          + (ALLOPHONE_OF[ph]
            ? `It is how /${ALLOPHONE_OF[ph]}/ is pronounced in a stressed syllable, `
              + `so search for /${ALLOPHONE_OF[ph]}/ instead.`
            : 'Check the symbol against the phoneme keyboard.'),
        [label]: ph,
      }, 422);
    }
  }

ALLOPHONE_OF is { 'ʌ': 'ə', 'ɝ': 'ɚ' } — a two-entry display hint, not a contrast rule. Put it and isKnownPhoneme in lib/phonemeInventory.ts, and write its vitest cover before wiring (same shape as Task 1's tests).

• [ ] Step 5: Run both suites

uv run python -m pytest packages/data/tests/ --ignore=packages/data/tests/test_datasets.py --ignore=packages/data/tests/test_new_loaders.py
cd packages/web/workers && npm test && npx tsc --noEmit

Expected: green. Any residual reference to the deleted modules is a compile error, which is the desired failure mode.

• [ ] Step 6: Commit

git commit -am "refactor(pairs): delete the stress-allophone special cases

The three sites that had to know ə~ʌ is not a contrast — allophones.py,
allophones.ts, and the test_derived oracle's exception — are unnecessary once ʌ
is not a phoneme. The pairs cannot be generated, so nothing downstream needs a
rule. This is the root fix the suppression was standing in for.

/minimal-pairs keeps a 422 for ʌ, but a generic unknown-phoneme one that points
at /ə/ rather than a bespoke contrast rule — an empty list would be the silent
drop PR #283 added the guard to avoid.

Spec: docs/superpowers/specs/2026-09-13-phonemic-stress-separation-design.md §1"

---

Task 5: Stress-aware pattern predicate

Without this the collapse silently deletes a capability: "find words with a schwa" currently works because ə means unstressed. After Task 2 it means any AH. Unstressed vowel reduction is a clinical target, so the query has to become expressible.

Files: - Modify: packages/web/workers/src/lib/wordFilter.ts - Modify: packages/web/workers/src/config/properties.ts - Test: packages/web/workers/src/__tests__/stressPredicate.test.ts

Interfaces: - Produces: two new WordSearchBody fields — stress_at_phoneme?: { phoneme: string; stress: 0 | 1 | 2 } and primary_stress_syllable?: number — compiled by compileWordFilter into single-LIKE clauses. - Consumes: phonemes_stress_str and syllable_stress_str from Task 3. The positional-pairing problem this task originally had to solve is gone: the fused column makes "unstressed /ə/" one clause, phonemes_stress_str LIKE '%|ə0|%', and stress position one clause against syllable_stress_str. No paired column or query-layer trick is needed.

• [ ] Step 1: Write the failing tests

// packages/web/workers/src/__tests__/stressPredicate.test.ts
/**
 * Stress-aware pattern matching (spec 2026-09-13 §3.4).
 *
 * Before the collapse, ə implied "unstressed" — searching the symbol was a
 * proxy for searching the stress. After it, that has to be said directly, or
 * the capability is silently gone.
 */
import { describe, it, expect } from 'vitest';
import { compileWordFilter } from '../lib/wordFilter';

describe('stress_at_phoneme', () => {
  it('emits a single fused-column clause, not a positional pairing', () => {
    const c = compileWordFilter({ stress_at_phoneme: { phoneme: 'ə', stress: 0 } } as never);
    expect(c.wordsWhere.some((w) => w.includes('phonemes_stress_str'))).toBe(true);
    // One LIKE carries both facts, so they cannot match at different positions.
    expect(c.params).toContain('%|ə0|%');
  });

  it('rejects a stress value outside 0-2', () => {
    expect(() => compileWordFilter(
      { stress_at_phoneme: { phoneme: 'ə', stress: 7 } } as never,
    )).toThrow(/stress/i);
  });

  it('rejects a phoneme outside the inventory', () => {
    expect(() => compileWordFilter(
      { stress_at_phoneme: { phoneme: 'ʌ', stress: 1 } } as never,
    )).toThrow(/ʌ/);
  });
});

describe('primary_stress_syllable', () => {
  it('compiles to a syllable_stress_str clause', () => {
    const c = compileWordFilter({ primary_stress_syllable: 2 } as never);
    expect(c.wordsWhere.some((w) => w.includes('syllable_stress_str'))).toBe(true);
    expect(c.params).toContain('|0|1|%');
  });

  it('rejects a non-positive syllable index', () => {
    expect(() => compileWordFilter({ primary_stress_syllable: 0 } as never))
      .toThrow(/syllable/i);
  });
});

• [ ] Step 2: Run to verify they fail

Run: cd packages/web/workers && npx vitest run src/__tests__/stressPredicate.test.ts Expected: FAIL — compileWordFilter ignores both fields.

• [ ] Step 3: Implement

The positional-pairing dilemma this step was written to confront no longer exists: Task 3's corrected format fuses the stress digit onto the nucleus, so each query shape is a single LIKE.

stress_at_phoneme { phoneme: 'ə', stress: 0 }  ->  phonemes_stress_str LIKE '%|ə0|%'
primary_stress_syllable: 2                     ->  syllable_stress_str LIKE '|0|1|%'

Validate the phoneme against the 39-symbol inventory and the stress value against 0-2, and reject a non-positive syllable index, so a malformed request fails loudly rather than compiling to a clause that matches nothing.

• [ ] Step 4: Run, then expose it

Add both fields to FILTERABLE_PROPERTIES / the property metadata so the frontend renders them from /api/property-metadata rather than hardcoding — the project's single-source rule.

• [ ] Step 5: Commit

git commit -am "feat(search): stress-aware pattern predicates

Before the collapse, searching ə was a proxy for searching unstressedness. After
it, that has to be sayable directly or the capability is silently gone — and
unstressed vowel reduction is a clinical target.

Adds stress_at_phoneme over phonemes_stress_str and primary_stress_syllable over
syllable_stress_str. The latter
has no equivalent today at all: stress position was not a queryable property.

Spec: docs/superpowers/specs/2026-09-13-phonemic-stress-separation-design.md §3.4"

---

Task 6: D3 — measure the merged-phoneme vector, then choose

This task decides nothing until it has measured. phoneme_dots is keyed by IPA; with only ə present, ʌ's learned vector goes unused and every stressed AH is scored with ə's vector. The size of that distortion is unknown.

Files: - Create: research/2026-09-13-similarity-baseline/measure.py - Create: research/2026-09-13-similarity-baseline/RESULTS.md (committed) - Modify: packages/data/src/phonolex_data/pipeline/derived.py (whichever option wins)

Interfaces: - Consumes: baseline.json from Task 1; packages/features/outputs/composites.csv. - Produces: RESULTS.md naming the chosen option and its measured score movement.

• [ ] Step 1: Quantify the dot-product change for each option

# research/2026-09-13-similarity-baseline/measure.py
"""How much does the merged-phoneme vector move similarity? (spec §D3)

Three candidates for the vector representing the collapsed AH phoneme:
  1. use ə's learned vector          — simplest; distorts stressed instances
  2. frequency-weighted merge of ə and ʌ
  3. make similarity stress-aware and look up the allophone's vector

Reports the L2 and cosine movement each option induces across the dot-product
matrix that soft-Levenshtein consumes, so the choice is measured rather than
argued.
"""
from __future__ import annotations

import csv
import itertools
from pathlib import Path

import numpy as np
import polars as pl

FEATURES = Path("packages/features/outputs/composites.csv")
WORDS = Path("data/runtime/words.parquet")


def load_vectors() -> dict[str, np.ndarray]:
    rows = list(csv.DictReader(FEATURES.open(encoding="utf-8")))
    feats = [c for c in rows[0] if c != "segment"]
    return {
        r["segment"]: np.array([float(r[f]) for f in feats], dtype=np.float64)
        for r in rows
    }


def main() -> None:
    V = load_vectors()
    w = pl.read_parquet(WORDS, columns=["phonemes_str"]).drop_nulls()
    # Observed frequency of the two realizations, for option 2's weighting.
    n_schwa = w.filter(pl.col("phonemes_str").str.contains("|ə|", literal=True)).height
    n_wedge = w.filter(pl.col("phonemes_str").str.contains("|ʌ|", literal=True)).height
    total = n_schwa + n_wedge
    print(f"realization counts: ə {n_schwa:,}  ʌ {n_wedge:,}")

    options = {
        "1-schwa-vector": V["ə"],
        "2-freq-weighted": (n_schwa * V["ə"] + n_wedge * V["ʌ"]) / total,
    }
    inventory = [p for p in V if p not in ("ʌ", "ɝ")]

    for name, merged in options.items():
        deltas = []
        for other in inventory:
            if other == "ə":
                continue
            before_schwa = float(np.dot(V["ə"], V[other]))
            before_wedge = float(np.dot(V["ʌ"], V[other]))
            after = float(np.dot(merged, V[other]))
            deltas.append(abs(after - before_schwa))
            deltas.append(abs(after - before_wedge))
        d = np.array(deltas)
        print(f"\n{name}: dot-product shift vs the two realizations")
        print(f"  mean |Δ| {d.mean():.4f}   median {np.median(d):.4f}   max {d.max():.4f}")
        print(f"  as a share of ə's norm²: {d.mean() / float(np.dot(V['ə'], V['ə'])):.2%}")

    print(f"\nə vs ʌ, for scale: cos "
          f"{float(np.dot(V['ə'], V['ʌ']) / (np.linalg.norm(V['ə']) * np.linalg.norm(V['ʌ']))):.4f}")


if __name__ == "__main__":
    main()

Run: uv run python research/2026-09-13-similarity-baseline/measure.py

• [ ] Step 2: Re-run the Task 1 probes and diff the rankings

After Tasks 2–3 and a parquet rebuild, re-run similarityBaseline.test.ts and diff against the committed baseline.json: report, per probe, how many of the top-20 neighbours changed, and the largest rank displacement. Ranking movement is the number that matters — dot-product deltas are the input, neighbour lists are what a user sees.

• [ ] Step 3: Write RESULTS.md and choose

Record both measurements, name the chosen option, and state the movement a user would see. If option 1 or 2 displaces more than a handful of neighbours per probe, option 3 (stress-aware lookup) is the answer even though it is the largest change — a silently worse ranking on most of the lexicon is not an acceptable cost for a smaller diff.

• [ ] Step 4: Implement the chosen option and commit

---

Task 7: Rebuild, verify, reseed

Files: - Create: research/2026-09-13-similarity-baseline/verify_collapse.py - Modify: packages/web/workers/scripts/d1-seed.manifest.json

• [ ] Step 1: Rebuild

uv run python packages/data/scripts/build_runtime_parquet.py

• [ ] Step 2: Verify every §6 criterion

Write verify_collapse.py in the shape of the previous branch's verify.py — a check() helper, non-zero exit on failure — asserting:

phonemes_str contains no ʌ and no ɝ, lexicon-wide
phonemes_stress_str has the same slot count as phonemes_str for every row
syllable_stress_str has one slot per syllable
ipa still reads /kʌp/, /bɝd/, /əˈbaʊt/, /ˈlɛtɚ/
zero ə~ʌ and ɚ~ɝ substitution pairs, with no suppression rule in the tree
the phonemes table holds 39 rows
abduction's phonemes contain ə twice
ablate is /ˌəˈbleɪt/

Plus: run the previous branch's verify.py unchanged — except its ipa minus stress == phoneme sequence assertion, which Task 2 Step 5 supersedes. Everything else in it must still pass, since this branch must not regress PR #283.

• [ ] Step 3: Confirm the audio inventory agrees

packages/audio/src emits ə and never ʌ, so the two halves should now agree. Assert it:

grep -rc "ʌ" packages/audio/src || echo "no ʌ in audio — inventories agree"
uv run python -c "
import polars as pl
w = pl.read_parquet('data/runtime/words.parquet')
assert w.filter(pl.col('phonemes_str').str.contains('|ʌ|', literal=True)).height == 0
print('lexicon emits no ʌ either — audio and lexicon inventories now match')
"

Check PHON-150/151 before this lands, not after — they touch the same label set.

• [ ] Step 4: Full matrix, seed, upload, PR

Same sequence as the previous branch: full three-package matrix; export-to-d1.py; apply locally with sqlite3, not wrangler d1 execute --file (at >300 MB wrangler errors and exits 0 having done nothing); upload-seed-to-r2.py; commit the manifest; PR to develop.

---

Self-Review

Spec coverage: §1 (double encoding)→Task 2; §1 "rule does not compose"→Task 4; §1 mis-assigned blame→Task 2 Step 1's test_ablate_vowel_is_now_correct; §2 (vectors keep both)→Global Constraints + Task 6; §3.1→Task 2; §3.2→Task 3; §3.3→Task 2 Step 4; §3.4→Task 5; §5 A–G→Tasks 2,3,2,4,5,6,7; §6 criteria 1–10→Task 7 Step 2 (1–5, 8), Task 4 (6), Task 5 (7), Task 6 (9), Task 7 Step 4 (10); §7 risks→Task 1 (similarity), Task 7 Step 3 (audio), Global Constraints (syllable_count).

Placeholder scan: Three steps deliberately stop and report rather than guess — Task 1 Step 2 (test-pool D1 wiring), Task 4 Step 3 (the a/uh assertion depends on post-collapse phonemes), Task 5 Step 3 (positional pairing may need a Task 3 amendment). Each says what to check and why guessing is worse, which is instruction rather than omission. Task 6's choice is gated on its own measurement by design.

Type consistency: phonemes_stress_str(syllables, phonemes) -> str | None and syllable_stress_str(syllables) -> str | None are used identically in Task 3's tests, the emit site, and Task 7's verifier. Task 3's originally planned stress_str no longer exists — the format was corrected during execution and the plan rewritten to match, so no task references it. STRESSED_ALLOPHONE (Python, dict[str, str]) and ALLOPHONE_OF (TS, Record<string, string>) are named per language and serve different roles — render substitution vs a user-facing hint — so they are deliberately not mirrors of each other. scoreSimilarityScan's signature is quoted from the existing export and not altered.