Testing Data Model

What the agent-testing stack stores today, what it should store, and the two calls still open. Current-state facts were verified against the simv2-chatbased-poc worktree on 2026-08-15.

Status: proposal

Nothing in the "proposed spine" section is built. This is the design under discussion for the testing MVP; the "what we have today" section is the only part that describes reality. Two decisions — where suites live, and how wide the MVP is — are open, and everything downstream of them follows.

What we have today

Twelve tables, none designed for this. The testing stack grew by borrowing Hamming's shape, and most of its rough edges trace back to that borrowing.

Single-turn replay fixtures appear nowhere on that diagram: they are JSON files in a gitignored directory, deliberately, because unit-integration-tests-poc forbade itself any migration.

Where the rough edges come from

Symptom	Cause in the schema
Run name doesn't display	team_test_plans has no name column. It is stashed in a metrics jsonb on a synthetic row.
Voice and chat runs block each other	assistant_test_config_id is both the phone-hold lock and the display config, and there is no test_mode column to tell lanes apart.
Prompt edits can't be correlated to results	promptReference lives inside a jsonb blob, so it cannot be queried or grouped. The VAPI lane writes an empty map anyway.
No change-over-time view	A clause is identified by its own text. Rewording an expectation silently breaks its history.
Call-derived regression cases have no mocks	tool_call_mocks keys on Hamming's case id, so our mocking depends on a vendor's identifier.
Tool arguments were missing from the results UI	Tool calls are not rows. They are parsed back out of a transcript jsonb.
Results for a Hamming-free run	…are written to hamming_test_case_results under a synthetic chat:-prefixed id.

The sharpest one is not a bug yet

test_case_id is already polymorphic. On a suite run it holds a test_suite_template_cases.id; on an ad-hoc or call-derived run it holds a Hamming case id. Two namespaces in one column, with no discriminator — and both mocks and results key on it. Nothing has collided only because the id spaces happen not to overlap.

The shape underneath all three

Suite simulation, single-turn replay, and call-derived regression look like three products. They are one object seen from three angles: pinned inputs, plus assertions, run against a config, producing per-assertion verdicts. What differs is only what got pinned.

Why this matters for migrations, not architecture

If the schema models scenario directly as columns, single-turn replay forces a rewrite later. If it models case of a kind, replay lands as a new kind plus one artifact table — which is the seat unit-integration-tests-poc is already holding when its brief says the gitignored fixtures directory "stands in for the table a productized version would need."

Proposed spine

Four groups. Definition is what a test is; execution is what happened; judgment is the verdict; mocking is the world it ran in.

What each group buys:

• Stable assertion ids turn the per-clause verdicts already being written into a real time series. Rewording becomes "clause retired, clause added" instead of a phantom regression.
• A prompt_digest on the run — a digest over sorted sub-agent hashes, not one arbitrary member — makes "did the prompt change between these two runs" a single equality check, and makes prompt-diff-beside-clause-flips buildable.
• Tool calls as rows give arguments, output, and how each call resolved natively rather than re-parsed from a blob. It also makes "did any live read happen in this run" answerable, which today it is not.
• A separate phone-hold table retires assistant_test_config_id's double duty, so only the voice lane can ever block anything.
• test_case_fixtures is written now and used later: an S3 key plus a content hash, mirroring how compiled prompts are already stored. That is the entire accommodation single-turn replay needs.

Where suites live

The constraint from the internal build ledger is do not build a competing suite store. Category-1 quality work owns test_suite_templates, which stores suites as DB templates that generate cases from Knowledge Base data. The question is whether we sit inside that, beside it, or on top of it.

Option A — they own recipes, we own runs

test_suite_templates stays what it already is: a generator. Its output materialises into our test_cases, and our test_suites keeps a nullable source_template_id pointing back at the recipe it came from.

No competing store, because a template is a recipe and not a suite. Ad-hoc, call-derived, and single-turn cases get a home templates were never going to give them. The cost is two concepts to explain, and a materialisation step that can drift from its recipe.

Option B — ours outright

We own suites and cases fully; templates become one importer among several, no different from the call-derived path. Simplest model and fastest to build. It is also the thing the ledger warned against: if two first-class suite stores exist, someone eventually authors in the wrong one.

Option C — extend the existing tables in place

Add kinds and columns to test_suite_templates and its case table; no new store at all. Zero duplication, but we would not control the schema or its roadmap, single-turn fixtures fit badly into a KB-generation recipe, and every change we need becomes a negotiation.

How wide the MVP is

All four widths assume the spine above, so a narrower choice costs no migration later — it changes only what ships first.

W1 — suite simulation, production-ready. The new schema, the API on apps/api, the UI, and prompt-to-result correlation. Any scenario suite runs on VAPI or ElevenLabs against any config, with per-clause history. What stays broken: read tools still hit the live CRM, so a red case may be a real regression or may be yesterday's availability having changed, and the result cannot tell you which.

W2 — plus dynamic mocking. Per-case mocks owned by our ids, real-call tool-trace replay, and the comingFrom primitive. A test constructs the world it needs, so a red result means the agent changed. This is also what turns call-derived cases into actual regression tests rather than replays against drifting data.

W3 — plus deterministic verification. Factual outcomes stop being LLM-judged: a transfer-routing or booking assertion reads the database record instead of asking a model to infer it from a transcript. Highest-leverage unbuilt item in the ledger, with research already complete — but it brings a second subsystem onto the critical path, and it is worth strictly more once W2 exists.

W4 — plus post-call and CRM. Assert the booking actually landed in the CRM, correctly. Blocked on paid test instances of the real CRMs, which is an organizational commitment rather than engineering. Against a mock CRM it cannot catch the bug class that motivated it, because that bug was the divergence between the real CRM and our picture of it.

Already settled

Decision	Consequence for the migration
Hamming stays for voice; we own everything else	test_cases.id is ours and is the primary key. external_ref holds the Hamming id, nullable, used by the voice lane. Mocks and assertions key on our id — which is what gives call-derived cases mockability.
Port everything to apps/api	The lanes reach into web-only modules (agent factory, tool dispatch, Supabase helpers), so the port is the largest single work item and wants sequencing early: porting after the schema lands means touching every call site twice.

Related

• Simulation Testing v2 — canonical context for simv2; read it before proposing anything in this space
• Tool Call Mocks — the mock resolution tiers as they exist today
• Deterministic verification — the W3 path
• Code path: live vs mock — why a red result can be data drift
• Planning: testing-infrastructure (build ledger), simv2-chatbased-poc (the POC this model came out of), unit-integration-tests-poc and single-turn-blueprint-regression (the single-turn half the spine reserves room for)