Telemetry Improvements — Design¶

Date: 2026-07-21 Status: Draft for review Context: PHON-168 follow-on. The capture pipeline (frontend track() → POST /api/events → Workers Analytics Engine behind a strict allowlist) is live and verified in both prod and staging. This spec closes the three gaps found in the 2026-07-21 telemetry investigation: instrumentation holes (including the newly shipped Materials Editor), the WAE 3-month retention ceiling, and the reporting experience.

Goals¶

Every user-facing surface emits enough allowlisted events to answer "which components get used, how often, and do they fail" — with zero content capture (no words, no text, no identifiers beyond the existing anonymous device UUID).
Aggregate usage history survives past WAE's 3-month retention window, without weakening the privacy policy's "data points expire after approximately 90 days" statement.
Weekly numbers are readable in one command.

Non-goals¶

The PHON-177 therapy-pack funnel events (pack_started, pack_item_completed, pack_completed, pack_abandoned, pack_exported) stay reserved and unimplemented — separately ticketed.
No dashboards (Grafana, hosted HTML). Revisit if traffic outgrows the CLI report.
No changes to capture semantics: the allowlist mechanism, the /api/events validation rules, and the data-point layout (BLOB_LAYOUT / DOUBLE_LAYOUT) are unchanged except for the four new count-only event names below.
No new user-facing surface of any kind.

Workstream A — Instrumentation gaps (frontend + allowlist)¶

A.1 Contrast Sets `search_executed`¶

ContrastiveInterventionTool.tsx currently fires nothing. Add search_executed on the primary retrieve action with the same prop shape as the other tools:

track('search_executed', {
  tool: 'contrastive',        // matches the tool id used by tool_opened
  constraint_count, result_count, elapsed_ms,
});

No allowlist change — the event and all props already exist.

A.1b Shared export path `results_exported` (found in the sweep)¶

results_exported currently fires only from the therapy-pack ExportDialog. The CSV export and copy-to-clipboard affordances that Word Lists, Sentences, and Contrast Sets all render via the shared components (WordListTable / SelectionToolbar and the export button they wrap) fire nothing. Root fix, once, in the shared components: instrument the export button's download handler and the copy handler (WordListTable.tsx clipboard write), with the tool id threaded in as a new optional analyticsTool prop from the three tool call sites (format: 'csv', item_count = exported/copied row count). No per-tool duplication.

A.2 `error_shown` wiring¶

The event exists end-to-end (allowlist, blob6 category) but has zero call sites. Wire it with a coarse category enum — never free text, never error messages:

category	fired from	tool prop
`crash`	`ErrorBoundary.tsx` componentDidCatch	active tool id if known, else `''`
`api_error`	fetch/HTTP failure paths in the six tools; word-resolution failures in the editor's manual add fields	the tool id; `'editor'` for in-editor resolution failures
`export_failed`	`ExportDialog.tsx` download catch + copy-CSV catch (both currently swallow errors silently)	`'therapyPack'`
`audio`	client-side audio failures in `AudioAnalysisTool` (mic denied, upload rejected)	`'audio'`

The category set is closed: these four strings only. Anything new requires a spec/allowlist revision, same as event names.

A.3 Materials Editor pack lifecycle (found in the 2026-07-21 sweep)¶

pack_seeded covers only the guided landing-page path. Blank creation, reopening saved materials, duplication, and deletion fire nothing — so the core adoption signal for the commercialization wedge (do users return to saved materials?) is invisible.

Four new count-only events (props {}, no new blob/double slots, no content — same privacy class as pack_edited):

event	fired from
`pack_created`	`newPack()` call sites (PackEditor empty state, PackHeader "New set", MyPacksDialog) — blank creation only; the seeded path keeps `pack_seeded`
`pack_opened`	`loadPack()` via MyPacksDialog `handleOpen`
`pack_duplicated`	`duplicatePack()` (PackHeader + MyPacksDialog)
`pack_deleted`	`deletePack()` (MyPacksDialog)

These names deliberately avoid the reserved PHON-177 set. They are added to ANALYTICS_EVENTS in workers/src/config/analyticsEvents.ts (+ tests) and to the report's event listing. Because clients can't send unknown names, the frontend and worker changes ship in the same PR.

Consistency fixes (existing event, new call sites): renamePack (via MyPacksDialog) and startOver fire pack_edited — both are content/title edits and MyPacksDialog rename currently bypasses the edit() wrapper that PackHeader title changes use.

A.4 Confirmed covered (no change)¶

Sweep results for the record: content edits/reorders/removals and title changes (pack_edited via the edit() wrapper), manual adds (pack_edited), tool-sourced adds (add_to_editor), all four export paths (results_exported: csv download, csv copy, cards PDF, worksheet PDF), in-editor tool opens (route through App.tsx handleToolSelect → tool_opened), landing funnel (landing_view/target_resolved/ pack_seeded), Lookup (word_viewed), Word Lists/Text Analysis/Sentences (search_executed), audio usage (server-emitted audio_request).

Workstream B — Retention rollup¶

Storage: new D1 database `phonolex-analytics`¶

One database shared by prod and staging rows (an env column mirrors the existing WAE dataset split). Deliberately separate from the lexicon DBs: the seed pipeline can never touch analytics history, and a lexicon DB recreate costs nothing here.

Three narrow tables, grain = one row per ISO week (+ dims):

CREATE TABLE weekly_event_counts (
  week_start TEXT NOT NULL,   -- ISO date of the Monday, e.g. '2026-07-13'
  env        TEXT NOT NULL,   -- 'prod' | 'staging'
  event      TEXT NOT NULL,
  tool       TEXT NOT NULL DEFAULT '',
  format     TEXT NOT NULL DEFAULT '',
  category   TEXT NOT NULL DEFAULT '',
  outcome    TEXT NOT NULL DEFAULT '',
  n          INTEGER NOT NULL,
  PRIMARY KEY (week_start, env, event, tool, format, category, outcome)
);
CREATE TABLE weekly_devices (
  week_start TEXT NOT NULL,
  env        TEXT NOT NULL,
  devices    INTEGER NOT NULL,  -- COUNT(DISTINCT session), collapsed to a number
  PRIMARY KEY (week_start, env)
);
CREATE TABLE weekly_search_stats (
  week_start        TEXT NOT NULL,
  env               TEXT NOT NULL,
  tool              TEXT NOT NULL,
  searches          INTEGER NOT NULL,
  median_elapsed_ms REAL,
  p90_elapsed_ms    REAL,
  PRIMARY KEY (week_start, env, tool)
);

Policy invariant (load-bearing, restated in the schema file): rollup rows are aggregate-only. No session/device identifier, no per-event rows, nothing keyed finer than (week × env × enum dims) is ever written. This is what keeps the privacy policy's ~90-day expiry statement true while aggregate counts persist.

Compute: new Worker `phonolex-analytics-rollup`¶

A separate, route-less Worker with a weekly cron trigger (Mondays 06:00 UTC). Separate from phonolex-api because the WAE SQL API requires an account-scoped Analytics-Read API token as a secret (ANALYTICS_READ_TOKEN), which should not be attached to the public API Worker or its dev config.

Each run:

Computes the prior ISO week's [Monday, Monday) window.
Runs the aggregate queries against both WAE datasets (phonolex_events, phonolex_events_staging) via the SQL API, using SUM(_sample_interval) and quantileWeighted per the documented layout in analyticsEvents.ts.
Upserts idempotently: DELETE the (week_start, env) slice, then INSERT — re-running a week is always safe.
On any failure, logs and exits nonzero-equivalent (observability enabled); no retries beyond the next weekly run. A missed week is recoverable manually for ~3 months (see backfill).

Backfill: the Worker exposes a fetch handler on its workers.dev route that requires Authorization: Bearer <ANALYTICS_READ_TOKEN> (the same secret; constant-time compare) and rolls up every complete ISO week still inside WAE's 3-month window. Any other request gets 404. Run once at deploy via curl. Events began early July 2026, so shipping before early October loses nothing.

Location: packages/web/workers/rollup/ (own wrangler.toml, package.json sharing the workspace toolchain). Deployed manually (wrangler deploy) — it changes rarely and does not belong in the paths-filtered app deploy workflows.

Workstream C — Reporting¶

Replace analytics-report.sh with packages/web/workers/scripts/analytics-report.py (repo scripts are already Python; run via uv run):

Live section — queries the WAE SQL API directly for the last 7 and 28 days: events by name, tool opens, searches + median/p90 latency by tool, error categories by tool, unique devices.
History section — reads phonolex-analytics via npx wrangler d1 execute phonolex-analytics --remote --json: devices and headline events per week with deltas vs the prior week.
Env selectable (--env prod|staging, default prod). Credentials: CF_ACCOUNT_ID + CF_API_TOKEN env vars (same as today's script) plus wrangler auth for the D1 read.
Either source failing degrades to a partial report with a one-line warning — never a stack trace, never a hard exit while the other source has data.
The bash script is deleted in the same PR (its layout comment moves to the Python script).

Testing¶

Frontend: extend existing component tests (Builder.test.tsx pattern) to assert the new track() calls: Contrast Sets search, the shared export/copy path (results_exported with the threaded tool id), each error_shown category at one representative site, the four lifecycle events, and the rename/start-over pack_edited fixes.
Worker allowlist: extend analyticsEvents / events-route tests for the four new names (accepted, count-only, unknown props stripped).
Rollup Worker: vitest with mocked SQL-API responses asserting the aggregation SQL matches the documented blob/double layout, idempotent re-run (delete-then-insert), and — explicitly — that no written row contains a session value.
Report script: golden-output test against canned WAE + D1 JSON fixtures (pytest, alongside the existing scripts tests).

Sequencing¶

A — one PR (frontend + allowlist + tests). No deploy dependencies.
B — new DB + rollup Worker + backfill run. Hard deadline: before early October 2026 (first WAE data ages out ~3 months after early-July ingest).
C — report script; reads B's tables, so lands after B (the live section works regardless).

Policy compliance summary¶

New events are count-only or enum-dimensioned; the prohibition on free text in analyticsEvents.ts is unchanged and continues to gate everything server-side.
Rollups persist aggregates only — no device identifiers outlive WAE's ~90-day window, so PrivacyPolicy.tsx requires no amendment.
No third-party services are introduced; the rollup Worker and D1 are first-party Cloudflare, same as the existing pipeline.