LLM Stimulus Fidelity Benchmark (PHON-215) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build the versioned prompt suite + deterministic PhonoLex-backed scorer that measures how often LLMs violate clinician-stated stimulus constraints, ready for the first pinned-model run.

Architecture: A self-contained Python module tree (stimfid/) inside research/2026-08-12-llm-stimulus-fidelity/, staged as: grid → rendered suite (frozen, content-hashed) → checkpointed model runner → deterministic extraction → adjudication-ladder scorer over the runtime parquets → bootstrap analysis + report. The live run itself is operations (RUNBOOK), not a plan task.

Tech Stack: Python 3.12 via uv run from repo root; polars for all tables; pytest; lazy-imported openai/anthropic/google-genai SDKs (tests use fakes, never live calls); numpy + matplotlib for analysis; spec at docs/superpowers/specs/2026-08-12-llm-stimulus-fidelity-benchmark-design.md.

Global Constraints

• Seed 215 everywhere randomness exists (grid sampling, naive-phrasing rotation, audit sample, bootstrap). Same inputs → byte-identical suite.
• IPA canonical: ɡ is U+0261, never ASCII g. Normalize via phonolex_data.phonology.normalize.to_ipa (workspace package, importable under uv run).
• Primary pronunciations only (PHON-154). Variant-satisfied items are flagged variant_only, never counted as pass or fail in the headline metric.
• Ground truth = data/runtime/words.parquet + pairs.parquet. Resolve the directory as $PHONOLEX_RUNTIME_DIR, else <repo-root>/data/runtime. (Worktrees don't have it — point the env var at the main checkout's data/runtime.)
• API keys from repo-root .env: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY. Never committed, never logged.
• No live API calls in tests. Client SDKs are imported lazily inside methods so the test env needs only polars/pytest/numpy.
• Tests run: uv run --with pytest python -m pytest research/2026-08-12-llm-stimulus-fidelity/tests/ -v from repo root.
• CLIs run: cd research/2026-08-12-llm-stimulus-fidelity && uv run python -m stimfid.<stage> (cwd is on sys.path under -m).
• Commit after every task; branch feat/phon-215-llm-stimulus-fidelity.

Directory layout (all under research/2026-08-12-llm-stimulus-fidelity/):

README.md                  (exists)
conftest.py                sys.path bootstrap for pytest
stimfid/
  __init__.py
  lexicon.py               Lexicon + WordRecord + tertiles
  constraints.py           Constraint + check_item ladder
  families.py              build_grid() — 5 families × ~30 cells
  render.py                naive/expert arm rendering
  build_suite.py           CLI: grid → suite_v1.parquet + manifest
  clients.py               ChatClient protocol + 3 vendors + FakeClient
  run_models.py            CLI: checkpointed JSONL runner
  extract.py               deterministic parsers + pair splitter + audit sample
  extract_llm.py           extraction-only LLM fallback (lazy)
  score.py                 CLI: adjudication ladder → scored.parquet
  analyze.py               metric tables + cluster bootstrap
  report.py                CLI: report.md + SBIR figure
  mine_prompts.py          CLI: Reddit union-recall pass (external drive)
  scope_public_datasets.py CLI: WildChat/LMSYS keyword scan (timeboxed)
models.yaml                pinned roster (verify per RUNBOOK before run)
prereg.md                  pre-registration (committed before any live run)
RUNBOOK.md                 run order: verify roster → smoke → full → score
ui_spotcheck.md + ui_spotcheck_sheet.csv
tests/                     test_lexicon.py, test_constraints.py, ...
data_local/                fetched wordlists, run outputs (gitignored)

---

Task 1: Scaffold + Lexicon interface

Files: - Create: research/2026-08-12-llm-stimulus-fidelity/conftest.py, stimfid/__init__.py, stimfid/lexicon.py, .gitignore (data_local/, out/, runs/) - Test: tests/test_lexicon.py

Interfaces: - Produces: Lexicon.from_dir(runtime_dir: Path | None) -> Lexicon; Lexicon.get(word: str) -> WordRecord | None (lowercased lookup, primary pronunciation); WordRecord frozen dataclass with word, phonemes: list[str], syllable_count: int, initial_phoneme: str, final_phoneme: str, cv_shape: str, is_canonical: bool, root: str | None, norms: dict[str, float | None] (keys: aoa, concreteness, log_frequency, neighborhood_density, familiarity); Lexicon.pair_info(w1: str, w2: str) -> dict | None (row from pairs.parquet, order-insensitive); Lexicon.tertiles(prop: str) -> tuple[float, float] (33rd/66th percentile over canonical rows, cached); Lexicon.feature_distance_threshold() -> float (75th percentile of pairs.feature_distance). - Consumes: data/runtime/{words,pairs}.parquet columns listed in Global Constraints.

• [ ] Step 1: Write conftest + fixture lexicon + failing test

conftest.py:

import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent))

tests/test_lexicon.py — build a tiny runtime dir in tmp_path with polars, then exercise the loader. Fixture data (reuse in later test files via a fixtures.py helper in tests/):

import polars as pl
import pytest
from stimfid.lexicon import Lexicon

def make_runtime(tmp_path):
    words = pl.DataFrame({
        "word":            ["cat", "bat", "rabbit", "ring", "missing", "kingdom", "wig"],
        "phonemes_str":    ["|k|æ|t|", "|b|æ|t|", "|ɹ|æ|b|ɪ|t|", "|ɹ|ɪ|ŋ|", "|m|ɪ|s|ɪ|ŋ|", "|k|ɪ|ŋ|d|ə|m|", "|w|ɪ|ɡ|"],
        "syllable_count":  [1, 1, 2, 1, 2, 2, 1],
        "initial_phoneme": ["k", "b", "ɹ", "ɹ", "m", "k", "w"],
        "final_phoneme":   ["t", "t", "t", "ŋ", "ŋ", "m", "ɡ"],
        "cv_shape":        ["CVC", "CVC", "CVCVC", "CVC", "CVCVC", "CVCCVC", "CVC"],
        "is_canonical":    [True, True, True, True, True, True, True],
        "root":            ["cat", "bat", "rabbit", "ring", "miss", "kingdom", "wig"],
        "aoa":             [3.0, 4.0, 4.5, 4.2, 5.5, 7.0, 4.1],
        "concreteness":    [4.9, 4.8, 4.9, 4.6, 2.1, 3.9, 4.7],
        "log_frequency":   [3.2, 2.8, 2.9, 3.0, 3.5, 2.7, 2.5],
        "neighborhood_density": [30.0, 32.0, 4.0, 18.0, 6.0, 1.0, 25.0],
        "familiarity":     [6.8, 6.5, 6.7, 6.6, 6.9, 5.9, 6.4],
    })
    pairs = pl.DataFrame({
        "word1": ["bat"], "word2": ["cat"],
        "phoneme1": ["b"], "phoneme2": ["k"],
        "position": [0], "position_type": ["initial"],
        "feature_distance": [1.8], "sonorant_diff": [0.0],
        "is_canonical": [True],
    })
    words.write_parquet(tmp_path / "words.parquet")
    pairs.write_parquet(tmp_path / "pairs.parquet")
    return tmp_path

def test_get_returns_record(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    rec = lex.get("Cat")          # case-insensitive
    assert rec.phonemes == ["k", "æ", "t"]
    assert rec.syllable_count == 1
    assert rec.norms["aoa"] == 3.0
    assert lex.get("zzzz") is None

def test_pair_info_order_insensitive(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    assert lex.pair_info("cat", "bat")["phoneme1"] == "b"
    assert lex.pair_info("bat", "cat") is not None
    assert lex.pair_info("cat", "ring") is None

def test_tertiles_and_threshold(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    t1, t2 = lex.tertiles("aoa")
    assert t1 < t2
    assert lex.feature_distance_threshold() == 1.8

• [ ] Step 2: Run to verify failure — uv run --with pytest python -m pytest research/2026-08-12-llm-stimulus-fidelity/tests/test_lexicon.py -v → FAIL (ModuleNotFoundError: stimfid then ImportError).

• [ ] Step 3: Implement stimfid/lexicon.py

from __future__ import annotations
import os
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path
import polars as pl

NORM_KEYS = ["aoa", "concreteness", "log_frequency", "neighborhood_density", "familiarity"]

@dataclass(frozen=True)
class WordRecord:
    word: str
    phonemes: list[str]
    syllable_count: int
    initial_phoneme: str
    final_phoneme: str
    cv_shape: str
    is_canonical: bool
    root: str | None
    norms: dict[str, float | None]

class Lexicon:
    def __init__(self, words: pl.DataFrame, pairs: pl.DataFrame):
        self._words = words.with_columns(pl.col("word").str.to_lowercase())
        self._by_word = {r["word"]: r for r in self._words.iter_rows(named=True)}
        self._pairs = pairs

    @classmethod
    def from_dir(cls, runtime_dir: Path | None = None) -> "Lexicon":
        d = Path(runtime_dir or os.environ.get("PHONOLEX_RUNTIME_DIR")
                 or Path(__file__).resolve().parents[3] / "data" / "runtime")
        return cls(pl.read_parquet(d / "words.parquet"),
                   pl.read_parquet(d / "pairs.parquet"))

    def get(self, word: str) -> WordRecord | None:
        row = self._by_word.get(word.strip().lower())
        if row is None:
            return None
        phonemes = [p for p in row["phonemes_str"].split("|") if p]
        return WordRecord(
            word=row["word"], phonemes=phonemes,
            syllable_count=row["syllable_count"],
            initial_phoneme=row["initial_phoneme"], final_phoneme=row["final_phoneme"],
            cv_shape=row["cv_shape"], is_canonical=bool(row["is_canonical"]),
            root=row.get("root"),
            norms={k: row.get(k) for k in NORM_KEYS},
        )

    def pair_info(self, w1: str, w2: str) -> dict | None:
        a, b = sorted([w1.strip().lower(), w2.strip().lower()])
        hit = self._pairs.filter(
            ((pl.col("word1") == a) & (pl.col("word2") == b))
            | ((pl.col("word1") == b) & (pl.col("word2") == a)))
        return hit.row(0, named=True) if hit.height else None

    @lru_cache(maxsize=None)
    def tertiles(self, prop: str) -> tuple[float, float]:
        s = self._words.filter(pl.col("is_canonical"))[prop].drop_nulls()
        return (s.quantile(1 / 3), s.quantile(2 / 3))

    @lru_cache(maxsize=None)
    def feature_distance_threshold(self) -> float:
        return self._pairs["feature_distance"].quantile(0.75)

(lru_cache on methods is fine here — one long-lived Lexicon per process.)

• [ ] Step 4: Run to verify pass — same command → 3 PASS.
• [ ] Step 5: Commit — git add -A research/2026-08-12-llm-stimulus-fidelity && git commit -m "feat(phon-215): stimfid scaffold + Lexicon over runtime parquets"

---

Task 2: Constraint model + item adjudication ladder

Files: - Create: stimfid/constraints.py - Test: tests/test_constraints.py

Interfaces: - Produces: Constraint dataclass (kind: str, params: dict) with to_json/from_json (list-level helpers constraints_to_json(list) -> str, constraints_from_json(str) -> list); check_item(word: str, constraints: list[Constraint], lex: Lexicon) -> ItemVerdict; ItemVerdict dataclass: word: str, status: str in {"pass","fail","oov","not_a_word","multiword"}, failed: list[str] (kinds that failed). Constraint kinds and params (exact): - phoneme_at {phoneme, position: "initial"|"medial"|"final"} - contains_phoneme {phoneme} - cluster_at {phonemes: list[str], position: "initial"|"final"} (contiguous subsequence at word edge) - syllable_count {min: int, max: int} - cv_shape {shape: str} - norm_band {prop: str, min: float | None, max: float | None} (null-tolerant: missing norm → fail with reason norm_missing:<prop>) - Consumes: Task 1 Lexicon, WordRecord. IPA normalization from phonolex_data.phonology.normalize.to_ipa applied to every phoneme param at construction. - Note: status="oov"/"not_a_word" are assigned by the scorer (Task 8) after the secondary wordlist check; check_item returns oov for any word lex.get misses and the scorer refines it.

• [ ] Step 1: Write failing tests (uses make_runtime from tests/fixtures.py — move the helper there now and import it in both test files)

from stimfid.constraints import Constraint, check_item
from stimfid.lexicon import Lexicon
from fixtures import make_runtime

def _lex(tmp_path):
    return Lexicon.from_dir(make_runtime(tmp_path))

def test_phoneme_position(tmp_path):
    lex = _lex(tmp_path)
    c = [Constraint("phoneme_at", {"phoneme": "ɹ", "position": "initial"})]
    assert check_item("rabbit", c, lex).status == "pass"
    v = check_item("cat", c, lex)
    assert v.status == "fail" and v.failed == ["phoneme_at"]

def test_medial_excludes_edges(tmp_path):
    lex = _lex(tmp_path)
    c = [Constraint("phoneme_at", {"phoneme": "s", "position": "medial"})]
    assert check_item("missing", c, lex).status == "pass"

def test_ascii_g_normalized(tmp_path):
    lex = _lex(tmp_path)
    c = [Constraint("contains_phoneme", {"phoneme": "g"})]   # ASCII g in constraint params
    assert check_item("wig", c, lex).status == "pass"        # matches only if normalized to U+0261
    assert check_item("kingdom", c, lex).status == "fail"    # ŋ is not ɡ

def test_norm_band_and_missing(tmp_path):
    lex = _lex(tmp_path)
    ok = [Constraint("norm_band", {"prop": "aoa", "min": None, "max": 5.0})]
    assert check_item("cat", ok, lex).status == "pass"
    assert check_item("kingdom", ok, lex).status == "fail"

def test_multiword_and_oov(tmp_path):
    lex = _lex(tmp_path)
    assert check_item("hot dog", [], lex).status == "multiword"
    assert check_item("florb", [], lex).status == "oov"

def test_roundtrip_json(tmp_path):
    from stimfid.constraints import constraints_to_json, constraints_from_json
    cs = [Constraint("syllable_count", {"min": 1, "max": 2})]
    assert constraints_from_json(constraints_to_json(cs)) == cs

• [ ] Step 2: Run to verify FAIL.
• [ ] Step 3: Implement. Core of check_item:

def check_item(word: str, constraints: list[Constraint], lex: Lexicon) -> ItemVerdict:
    w = word.strip().lower()
    if " " in w or "-" in w:
        return ItemVerdict(word=w, status="multiword", failed=[])
    rec = lex.get(w)
    if rec is None:
        return ItemVerdict(word=w, status="oov", failed=[])
    failed = [c.kind for c in constraints if not _check_one(c, rec)]
    return ItemVerdict(word=w, status="pass" if not failed else "fail", failed=failed)

_check_one dispatch: phoneme_at — initial: rec.phonemes[0] == p; final: rec.phonemes[-1] == p; medial: p in rec.phonemes[1:-1]. cluster_at — rec.phonemes[:len(ps)] == ps (initial) / rec.phonemes[-len(ps):] == ps (final). norm_band — value non-null and within [min, max] (open ends allowed); null → fail recorded as norm_missing:<prop> appended to failed instead of the kind. All phoneme params pass through to_ipa in Constraint.__post_init__.

• [ ] Step 4: Run to verify PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): constraint model + item adjudication"

---

Task 3: Task families + deterministic grid

Files: - Create: stimfid/families.py - Test: tests/test_families.py

Interfaces: - Produces: Cell dataclass: cell_id: str (e.g. "A1-017"), family: str in {"A1","A2","A3","B1","B2"}, tier: int (1–3), slots: dict, constraints: list[Constraint] (empty for A3 — contrast cells carry contrast instead), contrast: dict | None (A3 only: {"kind": "minimal"|"maximal"|"multiple", "phonemes": [p1, p2] | [target, c1, c2, c3]}), list_length: int (10 for word lists, 8 for pair lists). build_grid(lex: Lexicon) -> list[Cell] — deterministic (seed 215), ~30 cells/family, norm-band numeric cutoffs baked in from lex.tertiles at build time. - Consumes: Task 1 Lexicon.tertiles, Task 2 Constraint. - Slot pools (fixed literals in families.py): EARLY = ["m","b","p","d","n","w"], MID = ["k","f","tʃ","ʃ"] (ɡ excluded from MID to avoid the U+0261 trap in prompts; it appears in A3 contrasts), LATE = ["ɹ","s","l","θ","ð"]; positions ["initial","medial","final"]; A2 clusters [["s","p"],["s","t"],["s","k"],["b","ɹ"],["k","ɹ"],["p","l"],["k","l"]]; A3 contrasts minimal [("t","k"),("s","θ"),("ɹ","w"),("k","ɡ"),("f","θ"),("s","ʃ")], maximal [("ɹ","b"),("s","m"),("l","p")], multiple [("s", ["t","ʃ","θ"]), ("ɹ", ["w","l","j"])]. - Tier rules: A1/A2 — tier 1 = EARLY×initial×1–2 syl; tier 2 = MID×any or medial; tier 3 = LATE×(medial|final) or 3-syl or + age qualifier (norm_band aoa ≤ tertile-1). A3 — minimal=1, maximal=2, multiple=3. B1/B2 — tier by band tightness: tier 1 single band, tier 2 two bands, tier 3 three bands (B1) / density×frequency crossing (B2).

• [ ] Step 1: Failing tests

from stimfid.families import build_grid
from stimfid.lexicon import Lexicon
from fixtures import make_runtime

def test_grid_shape_and_determinism(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    grid = build_grid(lex)
    fams = {c.family for c in grid}
    assert fams == {"A1", "A2", "A3", "B1", "B2"}
    for f in fams:
        n = sum(1 for c in grid if c.family == f)
        assert 25 <= n <= 35, f
        assert {c.tier for c in grid if c.family == f} == {1, 2, 3}, f
    assert [c.cell_id for c in grid] == [c.cell_id for c in build_grid(lex)]  # deterministic
    assert len({c.cell_id for c in grid}) == len(grid)

def test_a3_cells_carry_contrast_not_constraints(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    for c in build_grid(lex):
        if c.family == "A3":
            assert c.contrast is not None and not c.constraints
        else:
            assert c.contrast is None and c.constraints

def test_b1_bands_are_numeric(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    b1 = [c for c in build_grid(lex) if c.family == "B1"]
    for cell in b1:
        for con in cell.constraints:
            if con.kind == "norm_band":
                assert con.params["min"] is not None or con.params["max"] is not None

• [ ] Step 2: Run to verify FAIL.
• [ ] Step 3: Implement. Each family gets a private generator (_a1(lex), …) that enumerates the full cartesian product of its slot pools, assigns tiers by the rules above, then random.Random(215).sample down to 30 per family stratified by tier (10/10/10; if a tier has <10 combos, take all and backfill from the largest tier). cell_id = f"{family}-{i:03d}" assigned after sampling, in sorted-slot order. B1 bands: for each of aoa/concreteness/log_frequency, "low" = (None, t1), "high" = (t2, None) from lex.tertiles(prop); B2 same for neighborhood_density × log_frequency.
• [ ] Step 4: Run to verify PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): five task families + deterministic grid"

---

Task 4: Prompt rendering — naive and expert arms

Files: - Create: stimfid/render.py - Test: tests/test_render.py

Interfaces: - Produces: render(cell: Cell, arm: str) -> str (arm in {"naive","expert"}), deterministic (naive phrasing variant chosen by hash of cell_id → stable rotation through the family's phrasing pool, no RNG state). - Consumes: Task 3 Cell. - Naive pools are paraphrases of observed Reddit phrasing (provenance adapted for A1/A2, reconstructed for A3/B1/B2 — recorded by Task 5, not here). Include theme injection in ~1/3 of A1/A2 naive variants (themes pool: ["dinosaurs", "Minecraft", "space", "cooking"], rotated deterministically). Spelled sound names in naive arm ("r", "s", "th" — mapping NAIVE_SOUND = {"ɹ": "r", "θ": "th", "ð": "th", "tʃ": "ch", "ʃ": "sh", "ɡ": "g", ...} for every phoneme in the slot pools); IPA + explicit definitions in expert arm. - Example fixed templates (exact strings live in render.py as module constants): - A1 naive v0: "give me {n} {syll_phrase}words with the {sound} sound at the {position} for a kid working on {sound}" - A1 expert: "List exactly {n} common English words that contain the phoneme /{ipa}/ in {position} position ({position_def}). Each word must have {syll_spec} syllable(s). One word per line, no other text, no proper nouns, no repeats.{age_clause}" where position_def spells out e.g. "the first sound of the word — not merely the first letter", and age_clause renders any norm_band on aoa as " Every word must be acquired by age {max} (age-of-acquisition rating ≤ {max})." - A3 expert (minimal): "List exactly {n} minimal pairs contrasting /{p1}/ and /{p2}/. A minimal pair is two real English words whose pronunciations differ in exactly one phoneme, in the same position: /{p1}/ in one word and /{p2}/ in the other. Format: one pair per line as word1 - word2. No other text." - B1 expert renders each numeric band: "log10 word frequency between {min} and {max}" / "age-of-acquisition rating ≤ {max}" etc.

• [ ] Step 1: Failing tests

from stimfid.render import render
from stimfid.families import build_grid
from stimfid.lexicon import Lexicon
from fixtures import make_runtime

def test_both_arms_render_all_cells(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    for cell in build_grid(lex):
        for arm in ("naive", "expert"):
            text = render(cell, arm)
            assert len(text) > 20 and "{" not in text  # no unfilled slots

def test_expert_arm_uses_ipa_naive_does_not(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    a1 = next(c for c in build_grid(lex) if c.family == "A1"
              and any(x.params.get("phoneme") == "ɹ" for x in c.constraints))
    assert "/ɹ/" in render(a1, "expert")
    assert "ɹ" not in render(a1, "naive")

def test_render_deterministic(tmp_path):
    lex = Lexicon.from_dir(make_runtime(tmp_path))
    cell = build_grid(lex)[0]
    assert render(cell, "naive") == render(cell, "naive")

• [ ] Step 2: Run to verify FAIL.
• [ ] Step 3: Implement — per-family naive pools (3–4 variants each) + one expert template each; variant index = int(hashlib.sha256(cell.cell_id.encode()).hexdigest(), 16) % len(pool).
• [ ] Step 4: Run to verify PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): naive/expert prompt rendering"

---

Task 5: Suite builder + freeze manifest

Files: - Create: stimfid/build_suite.py - Test: tests/test_build_suite.py

Interfaces: - Produces: CLI python -m stimfid.build_suite --out data_local/suite_v1.parquet and function build_suite(lex: Lexicon) -> pl.DataFrame with columns: prompt_id ("{cell_id}:{arm}"), cell_id, family, tier (int), arm, provenance ("adapted" for A1/A2 naive, "reconstructed" for A3/B1/B2 naive, "constructed" for all expert), prompt_text, constraints_json (Task 2 serialization), contrast_json (A3, else null), list_length (int). Alongside the parquet: suite_manifest.json — {"suite_sha256": <hash of canonical row serialization>, "n_prompts": int, "tertiles": {prop: [t1, t2]}, "feature_distance_threshold": float, "lexicon_words_sha256": <sha256 of words.parquet bytes>, "built": "<ISO date>"}. Freeze rule: if the manifest exists and suite_sha256 differs, the CLI exits 1 with a message to bump the version (new filename), never overwrites. - Consumes: Tasks 1–4. - Wild track: --wild wild_track.csv optionally appends hand-annotated rows (columns prompt_text, family, constraints_json, contrast_json, list_length, provenance="observed"; cell_id = "WILD-{i:03d}", tier = 0, arm = "naive"). The CSV is produced by Task 10 mining + hand annotation; the builder just ingests it.

• [ ] Step 1: Failing tests — build from fixture lexicon; assert row count = 2 × grid size; prompt_ids unique; provenance rules hold; rebuilding produces identical suite_sha256; freeze rule: tamper with one prompt, assert SystemExit.

  def test_suite_shape_and_hash(tmp_path):
      lex = Lexicon.from_dir(make_runtime(tmp_path))
      df, manifest = build_suite_with_manifest(lex)
      grid_n = len(build_grid(lex))
      assert df.height == 2 * grid_n
      assert df["prompt_id"].n_unique() == df.height
      naive_a3 = df.filter((pl.col("family") == "A3") & (pl.col("arm") == "naive"))
      assert set(naive_a3["provenance"].unique()) == {"reconstructed"}
      df2, manifest2 = build_suite_with_manifest(lex)
      assert manifest["suite_sha256"] == manifest2["suite_sha256"]
  

• [ ] Step 2: FAIL. — expose build_suite_with_manifest(lex) -> (pl.DataFrame, dict) as the tested unit; the CLI wraps it with file I/O + freeze check.
• [ ] Step 3: Implement. Canonical hash: sha256 over "\x1f".join(row values) joined by "\x1e", rows sorted by prompt_id. Date from datetime.date.today().isoformat() (manifest metadata only — never used in the hash).
• [ ] Step 4: PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): suite builder + freeze manifest"

---

Task 6: Model clients + checkpointed runner

Files: - Create: stimfid/clients.py, stimfid/run_models.py, models.yaml - Test: tests/test_run_models.py

Interfaces: - Produces: Completion dataclass (text: str, model_id: str, usage: dict, error: str | None); ChatClient protocol with complete(prompt: str, temperature: float) -> Completion; OpenAIClient(model_id), AnthropicClient(model_id), GeminiClient(model_id) (SDK imports inside complete), FakeClient(responses: dict[str, str] | callable); load_env(repo_root: Path) -> dict[str, str] (parses .env lines KEY=VALUE, no dotenv dep); run_suite(suite: pl.DataFrame, clients: dict[str, ChatClient], out_dir: Path, replicates: int = 3, temperature: float = 1.0) -> None — appends JSONL per model key at out_dir/{model_key}.jsonl, one line per call: {"prompt_id","replicate","model_key","model_id","ts","temperature","text","usage","error"}; resume: on start, read existing files, skip (prompt_id, replicate) already present with error == null; retry lines with non-null error. Flush after every line (checkpoint discipline); KeyboardInterrupt re-raises after the current line completes. - Consumes: Task 5 suite parquet. - models.yaml initial content (tier→model mapping is a run-time verification item — RUNBOOK step 1 re-checks against vendor docs on run day and edits this file before the full run):

# Verify against vendor docs on run day (RUNBOOK step 1). Record verification date.
verified: null
models:
  chatgpt_paid:  {provider: openai,    model_id: gpt-5.1}
  chatgpt_free:  {provider: openai,    model_id: gpt-5.1-mini}   # free-tier proxy — verify
  claude_free:   {provider: anthropic, model_id: claude-sonnet-5}
  gemini_free:   {provider: google,    model_id: gemini-2.5-flash} # verify current free default

- Executor note: load the claude-api skill before writing AnthropicClient (current SDK call shapes + model ids).

• [ ] Step 1: Failing tests — all against FakeClient, tmp out_dir:

  def test_runner_writes_all_calls_and_resumes(tmp_path):
      suite = tiny_suite()  # 2 prompts, built inline with pl.DataFrame
      fake = FakeClient(lambda prompt: f"1. cat\n2. bat  [{prompt[:5]}]")
      run_suite(suite, {"m1": fake}, tmp_path, replicates=2)
      lines = read_jsonl(tmp_path / "m1.jsonl")
      assert len(lines) == 4
      run_suite(suite, {"m1": fake}, tmp_path, replicates=2)   # resume: no dupes
      assert len(read_jsonl(tmp_path / "m1.jsonl")) == 4
  
  def test_runner_retries_errored_lines(tmp_path):
      # seed a line with error != null, rerun, assert it was replaced by a clean line
  

• [ ] Step 2: FAIL.
• [ ] Step 3: Implement. Vendor complete bodies (lazy import, one retry with 30s backoff on rate-limit errors, error string captured into Completion.error otherwise). Keys via load_env looked up at client construction; a missing key raises at construction, not mid-run.
• [ ] Step 4: PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): model clients + checkpointed runner"

---

Task 7: Extraction — deterministic parsers, pair splitting, audit sample

Files: - Create: stimfid/extract.py, stimfid/extract_llm.py - Test: tests/test_extract.py

Interfaces: - Produces: Extraction dataclass (items: list[str], method: str in {"numbered","bulleted","lines","comma","none"}, refusal: bool); extract_items(raw: str) -> Extraction; extract_pairs(raw: str) -> list[tuple[str, str]] (splits each extracted item on " - ", " – ", " — ", "/", " vs ", " vs. "; items that don't split into exactly 2 are kept as (item, "") so the scorer can fail them explicitly); make_audit_sample(rows: list[dict], frac: float = 0.05, seed: int = 215) -> list[dict]; extract_llm.fallback(raw: str, client) -> Extraction (prompt included below; method="llm"). - Parsing rules (in order, first match wins): numbered lines (^\s*\d+[.)]\s+), bulleted (^\s*[-*•]\s+), one-item-per-line (≥3 non-empty lines, each ≤ 6 words after cleaning), comma-separated single block. Cleaning per item: strip markdown bold/italic markers, trailing parentheticals (...), trailing IPA slashes /.../, surrounding quotes, terminal punctuation; preserve internal spaces/hyphens (multiword detection is the scorer's job, not the extractor's). Refusal detection: zero items AND raw matches (?i)(I can('|’)?t|I cannot|I'm unable|as an AI|consult a) → refusal=True. - extract_llm.fallback prompt (extraction ONLY, never judgment):

The following is a chatbot response to a request for a list of words or word pairs. Output ONLY the candidate items, one per line, exactly as they appear (do not correct, add, or remove words). If there are no list items, output nothing.\n\n---\n{raw} - Consumes: nothing internal — pure functions (LLM fallback takes any Task 6 ChatClient).

• [ ] Step 1: Failing tests with realistic chat outputs as literals: numbered list with bold + IPA annotations ("1. **Rabbit** (/ɹ/ initial)" → "rabbit" wait — cleaning lowercases? No: extraction preserves case; scorer lowercases. Assert "Rabbit"), preamble + bulleted list, prose refusal (refusal=True, items==[]), comma block, pair lines "1. cat - bat" → ("cat","bat"), malformed pair "1. cat" → ("cat",""). Audit sample: 100 rows → 5 rows, deterministic across calls.
• [ ] Step 2: FAIL.
• [ ] Step 3: Implement.
• [ ] Step 4: PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): deterministic extraction + audit sampling"

---

Task 8: Wordlist fetch + scorer (word-list families)

Files: - Create: stimfid/fetch_wordlists.py, stimfid/score.py - Test: tests/test_score.py

Interfaces: - Produces: - fetch_wordlists.py CLI → downloads ENABLE1 (public domain, https://raw.githubusercontent.com/dolph/dictionary/master/enable1.txt) to data_local/enable1.txt, verifies sha256 against the constant pinned in the module docstring on first successful fetch (fetch once, record hash, hard-fail on future mismatch); load_wordlist(path) -> set[str]. - score.py: score_response(prompt_row: dict, raw_text: str, lex: Lexicon, wordlist: set[str]) -> tuple[list[dict], dict] — (item rows, list row). - Item row: prompt_id, model_key, replicate, item, status, failed — status refines Task 2: extractor item → check_item; if oov and lowercased item not in wordlist → not_a_word (fabrication); if oov and in wordlist → stays oov (real word, out of lexicon; structural constraints unscorable → excluded from the satisfaction denominator, counted in oov_rate). - List row: prompt_id, model_key, replicate, n_requested, n_returned, n_pass, n_fail, n_oov, n_not_a_word, n_multiword, n_dupes (after lowercasing), n_morph_pad (items sharing a root with an earlier item), all_pass (bool: n_returned == n_requested and every scored item passed), refusal (bool), extract_method. - CLI python -m stimfid.score --suite ... --runs data_local/runs/ --out data_local/scored.parquet producing two parquets: scored_items.parquet, scored_lists.parquet. - Consumes: Tasks 1, 2, 5, 7. - g2p is out of v1 scope (spec's "where structural constraints remain checkable" is satisfied by reporting oov_rate separately; a g2p pass over oov items is a labeled follow-up in RUNBOOK, not silent).

• [ ] Step 1: Failing tests — fixture lexicon + toy wordlist {"cat","bat","rabbit","ring","missing","kingdom","zephyr"}:

  def test_scoring_ladder(tmp_path):
      lex = Lexicon.from_dir(make_runtime(tmp_path))
      row = a1_prompt_row(phoneme="ɹ", position="initial", n=3)   # helper builds suite-row dict
      raw = "1. rabbit\n2. ring\n3. cat\n4. zephyr\n5. florb\n6. hot dog\n7. rabbit"
      items, lst = score_response(row, raw, lex, WORDLIST)
      by = {i["item"]: i["status"] for i in items[:6]}
      assert by["rabbit"] == "pass" and by["ring"] == "pass"
      assert by["cat"] == "fail"
      assert by["zephyr"] == "oov"          # real word, not in fixture lexicon
      assert by["florb"] == "not_a_word"    # fabrication
      assert by["hot dog"] == "multiword"
      assert lst["n_dupes"] == 1 and lst["n_returned"] == 7
      assert lst["all_pass"] is False
  
  def test_refusal_list_row(tmp_path):
      ...  # refusal text -> n_returned 0, refusal True
  

• [ ] Step 2: FAIL.
• [ ] Step 3: Implement (including the CLI loop over runs/*.jsonl joined to the suite by prompt_id).
• [ ] Step 4: PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): scorer with OOV/fabrication ladder"

---

Task 9: A3 contrast-set scoring

Files: - Modify: stimfid/score.py (add score_pairs_response, dispatch on contrast_json non-null) - Test: tests/test_score_pairs.py

Interfaces: - Produces: pair item rows with status in {"pass","wrong_contrast","not_minimal","variant_or_gt1","not_a_word","oov","malformed"} and pair-specific fields w1, w2. Logic per claimed pair: 1. Either side empty → malformed. Either side fails the word ladder → that ladder status. 2. Both real & in lexicon → align primary phoneme lists: if equal-length and differ in exactly one index → substitution pair; else variant_or_gt1 (covers >1-phoneme diffs and variant-only pairs — primary-only per PHON-154). 3. Substitution pair: contrast phonemes (order-insensitive) == cell's {p1,p2} → pass for kind=minimal; else wrong_contrast. 4. kind=maximal: additionally require lex.pair_info(w1,w2)["feature_distance"] >= manifest threshold (threshold passed in via the suite manifest dict). If the alignment says single-substitution but the pair table has no row (unexpected for canonical pairs — feature distances live only in the pair table), keep the alignment verdict for the minimal-pair component, set feature_distance unmet (wrong_contrast is not implied), and flag pair_table_miss=True on the row so these surface in the prereg'd sensitivity table. 5. kind=multiple: first item = target; each subsequent pair (target, cᵢ) scored as minimal vs the cell's contrast set; plus list-level contrast_phonemes_distinct bool. - List row additions: false_pair_rate numerator fields (n_wrong_contrast, n_variant_or_gt1, n_malformed). - Consumes: Tasks 1, 7 (extract_pairs), 8.

• [ ] Step 1: Failing tests — fixture pairs table has cat/bat (b~k). Cases: "cat - bat" with contrast {b,k} → pass; contrast {t,k} → wrong_contrast; "rabbit - ring" → variant_or_gt1; "cat - florb" → not_a_word; "cat" → malformed.
• [ ] Step 2: FAIL.
• [ ] Step 3: Implement.
• [ ] Step 4: PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): contrast-set pair scoring"

---

Task 10: Analysis — metric tables, cluster bootstrap, report + figure

Files: - Create: stimfid/analyze.py, stimfid/report.py - Test: tests/test_analyze.py

Interfaces: - Produces: summarize(items: pl.DataFrame, lists: pl.DataFrame, suite: pl.DataFrame) -> pl.DataFrame — one row per (family, model_key, arm, tier) with columns constraint_satisfaction (pass / (pass+fail+not_a_word), i.e. oov excluded from denominator, fabrications counted as violations), fabrication_rate, oov_rate, list_pass_rate (share of lists with all_pass), dup_rate, refusal_rate, false_pair_rate (A3 only, else null), n_items, n_lists; bootstrap_ci(df: pl.DataFrame, metric_col: str, cluster_col: str = "cell_id", n_boot: int = 2000, seed: int = 215) -> tuple[float, float] (resample clusters with replacement, percentile 2.5/97.5); report.py CLI → out/report.md (all tables) + out/figures/satisfaction_by_tier.png — the SBIR figure: x = tier (1–3), y = constraint satisfaction, one line per model_key, panel per arm, flat dashed line at 1.0 labeled "PhonoLex (by construction)". matplotlib, no seaborn, ≥300 dpi. - Consumes: Tasks 5, 8, 9 outputs.

• [ ] Step 1: Failing tests — synthetic scored frames with known rates (e.g. 8 pass / 2 fail → 0.8); bootstrap: degenerate single-cluster frame returns (metric, metric); seeded CI reproducible; summarize excludes oov from denominator (construct a frame where including it would give a different number and assert the right one).
• [ ] Step 2: FAIL.
• [ ] Step 3: Implement.
• [ ] Step 4: PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): analysis tables + cluster bootstrap + SBIR figure"

---

Task 11: Reddit prompt mining (union recall) + wild-track sheet

Files: - Create: stimfid/mine_prompts.py - Test: tests/test_mine_prompts.py (regex/tiering logic only — synthetic frames; the external drive is never touched in tests)

Interfaces: - Produces: CLI python -m stimfid.mine_prompts --data-root /Volumes/ExternalData1/speech-community-analysis-data --out data_local/prompt_corpus.parquet → columns post_id, subreddit, created_utc, tier ("A"|"B"|"C"), matched_tool, quoted_prompts (list[str]), text; plus data_local/wild_track_candidates.csv (tier-A rows exploded one quoted prompt per line, columns post_id, quoted_prompt, family_guess, constraints_json, contrast_json, list_length, provenance — the last four left blank for hand annotation; the annotated file is saved as wild_track.csv and fed to Task 5's --wild). - Pure functions (the tested surface): find_tool_mentions(text: str) -> list[str] — union regex: competitors-style aliases (ChatGPT|chat ?gpt|GPT-?[45][a-z0-9.-]*|OpenAI) ∪ broadened genAI (Claude|Gemini|Bard|Copilot|Perplexity|LLM|large language model|chatbot), case-insensitive, word-bounded; excludes the PROMPT-method false friend via negative lookahead (?!PROMPT\s+(certification|technique|trained|therapy)). extract_quoted_prompts(text: str) -> list[str] — double/single-quoted spans ≥ 4 words that either open with an imperative verb from a fixed list (give|write|create|list|make|generate|produce|convert|describe) or contain a wh-question opener, with an LLM cue word within 250 chars of the span. assign_tier(text: str) -> str — A if extract_quoted_prompts non-empty; B if a generation verb + concrete object (word list|words|sentences|story|passage|worksheet|minimal pairs|goals?|report|note) co-occurs within 600 chars of a tool mention; else C. - Consumes: nothing internal; reads the external-drive parquets listed in the spec (interim/threads_*.parquet, schema post_id, subreddit, created_utc, text, ...). - CLI behavior: if --data-root missing/unmounted → exit 2 with a clear "mount ExternalData1" message. Public-release policy note in module docstring: quoted prompts are for internal annotation; anything published gets paraphrased (spec §2).

• [ ] Step 1: Failing tests — literals: text with "I asked chatgpt \"give me 10 words with the r sound\"" → tier A, one quoted prompt; "I use ChatGPT to make word lists" → tier B; "AI is ruining the field" → C (no tool alias → C even with 'AI'? chatbot|LLM list doesn't include bare 'AI' — assert C); "PROMPT certification course" → no mention.
• [ ] Step 2: FAIL.
• [ ] Step 3: Implement (pure functions + thin CLI).
• [ ] Step 4: PASS.
• [ ] Step 5: Commit — git commit -m "feat(phon-215): Reddit union-recall prompt mining + wild-track sheet"

---

Task 12: Public-dataset scoping script (timeboxed)

Files: - Create: stimfid/scope_public_datasets.py - Test: none (manual, network + HF auth; the deliverable is a report artifact)

Interfaces: - Produces: CLI uv run --with datasets python -m stimfid.scope_public_datasets --dataset allenai/WildChat-1M --limit 500000 --out data_local/public_hits_wildchat.parquet — streams the dataset, keeps the first user turn of conversations matching any of: speech therapy|speech-language|SLP|articulation|phoneme|minimal pair|apraxia|phonolog (case-insensitive), writes conversation_id, matched_kw, first_user_turn, model_name + prints a count summary. Same invocation for lmsys/lmsys-chat-1m (gated — requires HF login; if datasets raises an auth error, print the huggingface-cli login instruction and exit 2). - Timebox (RUNBOOK): one day total including skimming hits; outcome recorded in README §4.1 either way (counts, whether any hits join the wild track).

• [ ] Step 1: Write the script (no TDD — no deterministic surface worth pinning; it's a one-shot filter whose output is hand-reviewed).
• [ ] Step 2: Smoke-run on 10k rows of WildChat (--limit 10000), confirm it streams and writes.
• [ ] Step 3: Commit — git commit -m "feat(phon-215): public prompt-dataset scoping script"

---

Task 13: Pre-registration, RUNBOOK, UI spot-check protocol

Files: - Create: prereg.md, RUNBOOK.md, ui_spotcheck.md, ui_spotcheck_sheet.csv (all in the research dir)

Interfaces: documents only; prereg.md MUST be committed before any live-model call (RUNBOOK enforces the order).

• [ ] Step 1: Write prereg.md with exactly these sections (content drawn from the spec, stated as testable predictions):
• Hypotheses. H1: pooled constraint-satisfaction < 0.90 for every free-tier model, naive arm. H2: satisfaction declines monotonically with tier (Page-type trend, per model×arm). H3: A3 false-pair rate > A1 position-error rate (contrast sets are the bleed zone). H4: expert arm improves satisfaction but does not close it (expert-arm satisfaction still < 0.95 pooled). H5: B1 norm-band violation rate > A1 structural violation rate (norms are less prompt-accessible than phonology).
• Operationalizations. The frozen tertile numbers + feature-distance threshold (copied from suite_manifest.json at freeze), the oov-exclusion denominator rule, dup/morph-pad definitions (lowercase string match; shared root column), refusal regex.
• Exclusions. Extraction failures (method="none", non-refusal) → excluded from satisfaction, reported as extraction-failure rate; pair_table_miss items → sensitivity table.
• Analysis. The Task 10 table spec + cluster bootstrap (2000, seed 215); no metric added after data lands except in a labeled "exploratory" section.
• [ ] Step 2: Write RUNBOOK.md — ordered checklist: (1) verify models.yaml roster against vendor docs, set verified: <date>; (2) fetch_wordlists; (3) build_suite (+ optional --wild); (4) commit prereg.md + suite + manifest; (5) smoke run: 1 cell × all models × 1 replicate, eyeball raw JSONL; (6) full run (replicates: 3, temperature 1.0); (7) extract audit sample → hand-audit CSV, record extraction accuracy; (8) score; (9) analyze/report; (10) UI spot-check per ui_spotcheck.md; (11) labeled follow-ups: g2p pass over oov items, LLM-fallback extraction for method="none" responses. Include the exact uv run command for every step and PHONOLEX_RUNTIME_DIR guidance for worktrees.
• [ ] Step 3: Write ui_spotcheck.md + sheet — selection: seeded (215) stratified sample of 25 prompt_ids (5/family, naive arm only); protocol: paste into each free-tier web UI logged out where possible, copy raw response verbatim into ui_spotcheck_sheet.csv (prompt_id, ui_product, date, raw_text); scored by the same score.py (the sheet is just another run source); comparison table = UI vs API satisfaction per family (Task 10 summarize on the merged frame with model_key = ui_*).
• [ ] Step 4: Commit — git commit -m "docs(phon-215): prereg, runbook, UI spot-check protocol"

---

Self-Review (run after drafting — resolved inline)

1. Spec coverage: taxonomy/grid → T3; arms → T4; provenance + freeze → T5; roster/API+checkpointing → T6; extraction + audit → T7; ladder incl. fabrication/OOV/dup/morph-pad/refusal → T8; false pairs → T9; difficulty curves, per-list metric, bootstrap, SBIR figure → T10; Reddit mining + wild track → T11; WildChat/LMSYS → T12; prereg-before-run, UI spot-check, re-run playbook → T13 (+ freeze manifest = the versioning story). Deferred g2p and LLM-extraction fallback are explicit, labeled follow-ups (RUNBOOK 11), not silent drops.
2. Placeholder scan: models.yaml free-tier ids are flagged as run-day verification items by design (model drift is a spec premise), with concrete initial values — not placeholders.
3. Type consistency: check_item statuses referenced by T8 match T2's set plus scorer refinement (not_a_word), stated in both places; extract_pairs tuple contract matches T9 consumption; build_suite_with_manifest is the tested unit wrapped by the T5 CLI.