Skip to content

LLM Stimulus Fidelity Benchmark — design

2026-08-12. Status: designed, approved for planning. Supersedes the scope sections of research/2026-08-12-llm-stimulus-fidelity/README.md where they differ; the README's rationale (§1–2) and failure-mode honesty (§5) carry forward unchanged.

Purpose

Measure how often general-purpose LLMs produce articulation and naming stimuli that violate the phonological or psycholinguistic constraints the clinician asked for, using PhonoLex as ground truth. Two consumers: preliminary data for the NIH SBIR (PA-27-100, receipt 2027-01-05) and a standalone measurement paper.

Decisions (settled 2026-08-12)

Question Decision
Artifact Reusable benchmark + study — versioned prompt suite + deterministic scorer; the SBIR/paper numbers are its first run
Publication Public suite, private scorer — prompts, taxonomy, methodology, results public; scoring runs against PhonoLex data, which never ships
v1 scope All five task families (A1–A3, B1–B2)
Model roster Clinician-named + tiers: ChatGPT free + paid, Gemini free, Claude free (~4–5 pinned versions)
Query mode API with pinned versions + manual UI spot-check (~25 prompts) validating the tier→model mapping
Suite architecture Approach 3: constraint grid core + in-the-wild subset
Home Develop in research/2026-08-12-llm-stimulus-fidelity/; extract the public half to a separate repo at release

1. Task taxonomy — the grid

Five parameterized families, ~30 base cells each (~150 total), every cell rendered in a naive and an expert arm. Each family has an explicit difficulty axis so results are curves, not one number.

  • A1 — structural word lists. Slots: target phoneme (sampled across early-8 / middle-8 / late-8 + vocalic ɹ), word position (initial/medial/final), syllable count, list length, optional age qualifier (adjudicated against AoA). Difficulty: phoneme × position rarity.
  • A2 — clusters & complexity. /s/-, /ɹ/-, /l/-blends by position; CV-shape requests (CCVC, CVCC); 3–4-syllable words containing the target.
  • A3 — contrast sets. Minimal pairs for a stated contrast, maximal opposition, multiple opposition (3+ contrasts to one target). Scored against the post-PHON-154 pair table and feature distances. Predicted primary bleed zone.
  • B1 — norm-banded naming stimuli. Frequency × AoA × concreteness/imageability bands, tight vs loose, adult naming-therapy framing.
  • B2 — neighborhood/phonotactics. High- vs low-density neighborhoods; density × frequency crossings.

Arms. Naive-arm phrasing templates are derived from the ~38 verbatim Reddit prompts (theme injections — "Minecraft words with /s/" — are observed behavior and stay in). Expert arm is fully explicit: IPA, defined numeric bands, position stated unambiguously. For A3/B1/B2, where the corpus has no observed prompts, naive phrasings are adapted from Stratum-A patterns and labeled reconstructed.

Wild track. The ~24 verbatim stimulus prompts (paraphrased in any public release) plus whatever the public-dataset scoping pass yields, hand-annotated with implied constraints, scored by the same scorer, reported separately as ecological validation of the grid.

2. Prompt corpus provenance

Corpus facts (established 2026-08-12 from ~/Repos/speech-community-analysis): the true ChatGPT-mention denominator is 233 unique posts, not 362 (mention rows ≠ units); ~30 posts (~13%) quote verbatim prompt text, yielding ~38 distinct prompt strings (~24 stimulus requests); ~80 more posts describe reconstructible task intents. Coverage is pediatric-heavy: minimal pairs n=1, anomia/naming n=0, dysphagia n=0 — Stratum B cannot be ecologically grounded from Reddit and is not claimed to be.

Three sources, every suite prompt tagged observed | adapted | constructed:

  1. Reddit union-recall pass — competitors.yml regex ∪ AI topic cluster ∪ broadened genAI regex (~330–350 post superset), re-tiered A/B/C.
  2. Public prompt datasets — WildChat-1M and LMSYS-Chat-1M searched (keyword + embedding) for SLP stimulus requests; timeboxed to one day; any hits are observed prompts with real model outputs attached.
  3. Team-authored grid templates.

Public-release policy: Reddit-derived prompts are paraphrased, never quoted verbatim (the corpus project's deferred IRB/mod-notification question stays deferred by not republishing identifiable content).

3. Ground truth & scorer

  • Data. Local runtime parquets (words, pairs, norms) via polars — no D1 dependency. Run manifest pins the lexicon snapshot by seed-manifest sha256. Constraints adjudicate against primary pronunciations only (PHON-154); variant-satisfied items are a separate sensitivity row.
  • Extraction. Deterministic parsers first (numbered/bulleted/comma lists). LLM fallback is extraction-only — it pulls word strings out of prose, never judges — and a hand-audited ~5% sample certifies extraction accuracy as a reported number. All scoring downstream of extraction is table lookup.
  • Adjudication ladder. (1) Normalize; multiword answers to single-word requests are their own violation category. (2) In PhonoLex → score directly. (3) Not in PhonoLex → secondary real-word check (SCOWL/ENABLE-class wordlist) before "fabricated" is assigned; real out-of-lexicon items get g2p phonology where structural constraints remain checkable and are flagged oov, killing the CMU-coverage confound. (4) Refusals, duplicates, morphological-variant padding → degenerate-output categories, never dropped.
  • Qualitative constraint operationalization. Every naive-arm qualitative constraint ("age-appropriate for a 6-year-old") gets a pre-registered numeric operationalization written before any model is queried.

4. Metrics & analysis

The six README metrics stand: constraint satisfaction rate (headline), position error rate, fabrication rate, norm-band violation rate, false minimal-pair rate, degenerate-output rate. Additions:

  • Difficulty curves — every metric against the grid's difficulty axes; the SBIR figure is satisfaction falling as constraints tighten, with PhonoLex flat at ceiling by construction (stated as such — the comparator establishes satisfiability, not superiority).
  • Per-list satisfaction alongside per-item — a 10-word list with 3 bad items is a failed clinical deliverable at 70% item accuracy.

CIs via cluster bootstrap, prompt cell as cluster. Arms difference = prompt-sensitivity estimate; free-vs-paid difference = the equity finding. Analysis pre-registered before the first scored run.

5. Pipeline

research/2026-08-12-llm-stimulus-fidelity/, staged and resumable:

  1. mine_prompts.py — Reddit union-recall + WildChat/LMSYS scoping → prompt_corpus.parquet with provenance.
  2. build_suite.py — grid × arms → frozen suite_v1.parquet; content hash = benchmark version.
  3. run_models.py — API calls with per-call checkpointing and resume (long-job policy); records model id, call date, temperature, raw response. Keys: ANTHROPIC/OPENAI/GEMINI in .env.
  4. extract.py → score.py — extraction + audit sample; deterministic scoring → scored.parquet.
  5. analyze.py / report.py — tables, difficulty curves, SBIR figure, bootstrap CIs.

prereg.md (hypotheses, operationalizations, exclusion rules) is committed before step 3 runs against any real model. UI spot-check: written protocol + hand-entered results for ~25 prompts across free tiers, reported as a mapping-validity table. Everything keyed to lexicon sha + suite hash so drift re-runs are one command against a frozen suite.

6. Deliverables & versioning

  1. Suite v1 — frozen prompts + taxonomy + provenance (the public half).
  2. First-run results — scored dataset; error tables by family × model × arm × difficulty; the Specific Aims figure.
  3. Write-up — short measurement paper; SBIR pulls from it.
  4. Re-run playbook — dated, suite-pinned drift tracking on major model releases.

Failure modes (carried from README §5, still binding)

LLMs may be fine on easy cells (report the bounded result — the difficulty curve is the finding either way); expert arm may close the gap (finding becomes prompt-engineering burden); OOV ≠ fabricated (ladder handles it); results are snapshots (versioning handles it).

Prerequisites before building

  1. Assign the PHON ticket (README's own rule: before branching).
  2. Co-investigator confirms the Stratum B constraint set actually used in practice; design proceeds on frequency/AoA/imageability/density and re-weights B1/B2 cells before suite freeze if the answer differs.
  3. Second-coder κ on the Reddit A/B/C re-tiering if the prompt-corpus mining is reported as a result in its own right (optional for the benchmark itself).

Deferred (unchanged from README)

Clinical-appropriateness rating (needs blinded SLP raters → commercial IRB → Phase I aim, not pilot). Rater-independence constraint stands: co-investigator and the two collaborating SLPs are contaminated for rating purposes; their role is design feedback and letters of support.