Skip to content

Testing Data Model ​

What the agent-testing stack stores today, what it should store, and the two calls still open. Current-state facts were verified against the simv2-chatbased-poc worktree on 2026-08-15.

Status: proposal

Nothing in the "proposed spine" section is built. This is the design under discussion for the testing MVP; the "what we have today" section is the only part that describes reality. Two decisions — where suites live, and how wide the MVP is — are open, and everything downstream of them follows.

What we have today ​

Twelve tables, none designed for this. The testing stack grew by borrowing Hamming's shape, and most of its rough edges trace back to that borrowing.

Single-turn replay fixtures appear nowhere on that diagram: they are JSON files in a gitignored directory, deliberately, because unit-integration-tests-poc forbade itself any migration.

Where the rough edges come from ​

SymptomCause in the schema
Run name doesn't displayteam_test_plans has no name column. It is stashed in a metrics jsonb on a synthetic row.
Voice and chat runs block each otherassistant_test_config_id is both the phone-hold lock and the display config, and there is no test_mode column to tell lanes apart.
Prompt edits can't be correlated to resultspromptReference lives inside a jsonb blob, so it cannot be queried or grouped. The VAPI lane writes an empty map anyway.
No change-over-time viewA clause is identified by its own text. Rewording an expectation silently breaks its history.
Call-derived regression cases have no mockstool_call_mocks keys on Hamming's case id, so our mocking depends on a vendor's identifier.
Tool arguments were missing from the results UITool calls are not rows. They are parsed back out of a transcript jsonb.
Results for a Hamming-free run…are written to hamming_test_case_results under a synthetic chat:-prefixed id.

The sharpest one is not a bug yet

test_case_id is already polymorphic. On a suite run it holds a test_suite_template_cases.id; on an ad-hoc or call-derived run it holds a Hamming case id. Two namespaces in one column, with no discriminator — and both mocks and results key on it. Nothing has collided only because the id spaces happen not to overlap.

The shape underneath all three ​

Suite simulation, single-turn replay, and call-derived regression look like three products. They are one object seen from three angles: pinned inputs, plus assertions, run against a config, producing per-assertion verdicts. What differs is only what got pinned.

Why this matters for migrations, not architecture

If the schema models scenario directly as columns, single-turn replay forces a rewrite later. If it models case of a kind, replay lands as a new kind plus one artifact table — which is the seat unit-integration-tests-poc is already holding when its brief says the gitignored fixtures directory "stands in for the table a productized version would need."

Proposed spine ​

Four groups. Definition is what a test is; execution is what happened; judgment is the verdict; mocking is the world it ran in.

What each group buys:

  • Stable assertion ids turn the per-clause verdicts already being written into a real time series. Rewording becomes "clause retired, clause added" instead of a phantom regression.
  • A prompt_digest on the run — a digest over sorted sub-agent hashes, not one arbitrary member — makes "did the prompt change between these two runs" a single equality check, and makes prompt-diff-beside-clause-flips buildable.
  • Tool calls as rows give arguments, output, and how each call resolved natively rather than re-parsed from a blob. It also makes "did any live read happen in this run" answerable, which today it is not.
  • A separate phone-hold table retires assistant_test_config_id's double duty, so only the voice lane can ever block anything.
  • test_case_fixtures is written now and used later: an S3 key plus a content hash, mirroring how compiled prompts are already stored. That is the entire accommodation single-turn replay needs.

Where suites live ​

The constraint from the internal build ledger is do not build a competing suite store. Category-1 quality work owns test_suite_templates, which stores suites as DB templates that generate cases from Knowledge Base data. The question is whether we sit inside that, beside it, or on top of it.

Option A — they own recipes, we own runs ​

test_suite_templates stays what it already is: a generator. Its output materialises into our test_cases, and our test_suites keeps a nullable source_template_id pointing back at the recipe it came from.

No competing store, because a template is a recipe and not a suite. Ad-hoc, call-derived, and single-turn cases get a home templates were never going to give them. The cost is two concepts to explain, and a materialisation step that can drift from its recipe.

Option B — ours outright ​

We own suites and cases fully; templates become one importer among several, no different from the call-derived path. Simplest model and fastest to build. It is also the thing the ledger warned against: if two first-class suite stores exist, someone eventually authors in the wrong one.

Option C — extend the existing tables in place ​

Add kinds and columns to test_suite_templates and its case table; no new store at all. Zero duplication, but we would not control the schema or its roadmap, single-turn fixtures fit badly into a KB-generation recipe, and every change we need becomes a negotiation.

How wide the MVP is ​

All four widths assume the spine above, so a narrower choice costs no migration later — it changes only what ships first.

W1 — suite simulation, production-ready. The new schema, the API on apps/api, the UI, and prompt-to-result correlation. Any scenario suite runs on VAPI or ElevenLabs against any config, with per-clause history. What stays broken: read tools still hit the live CRM, so a red case may be a real regression or may be yesterday's availability having changed, and the result cannot tell you which.

W2 — plus dynamic mocking. Per-case mocks owned by our ids, real-call tool-trace replay, and the comingFrom primitive. A test constructs the world it needs, so a red result means the agent changed. This is also what turns call-derived cases into actual regression tests rather than replays against drifting data.

W3 — plus deterministic verification. Factual outcomes stop being LLM-judged: a transfer-routing or booking assertion reads the database record instead of asking a model to infer it from a transcript. Highest-leverage unbuilt item in the ledger, with research already complete — but it brings a second subsystem onto the critical path, and it is worth strictly more once W2 exists.

W4 — plus post-call and CRM. Assert the booking actually landed in the CRM, correctly. Blocked on paid test instances of the real CRMs, which is an organizational commitment rather than engineering. Against a mock CRM it cannot catch the bug class that motivated it, because that bug was the divergence between the real CRM and our picture of it.

Already settled ​

DecisionConsequence for the migration
Hamming stays for voice; we own everything elsetest_cases.id is ours and is the primary key. external_ref holds the Hamming id, nullable, used by the voice lane. Mocks and assertions key on our id — which is what gives call-derived cases mockability.
Port everything to apps/apiThe lanes reach into web-only modules (agent factory, tool dispatch, Supabase helpers), so the port is the largest single work item and wants sequencing early: porting after the schema lands means touching every call site twice.
  • Simulation Testing v2 — canonical context for simv2; read it before proposing anything in this space
  • Tool Call Mocks — the mock resolution tiers as they exist today
  • Deterministic verification — the W3 path
  • Code path: live vs mock — why a red result can be data drift
  • Planning: testing-infrastructure (build ledger), simv2-chatbased-poc (the POC this model came out of), unit-integration-tests-poc and single-turn-blueprint-regression (the single-turn half the spine reserves room for)