Assistant Test Model Design — Alt 1: Single Pass Policy

Date: 2026-08-16 Status: Superseded by the current model

Kept for comparison. This is the shape before success criteria became their own table: strictness is a single pass_policy column on the suite, which cannot express two bars over one case, and cannot give two consumers different answers from one run.

Goal

Consolidate agent testing onto one data model, so that a new vertical or a new kind of test extends it rather than becoming another system.

Solution

One namespaced foundation, assistant_test_*, that today's testing surfaces move onto and tomorrow's extend. What forced a new system each previous time — a case that had to come from a call, a run that had to target a configuration, a verdict locked in a blob — becomes data rather than structure. Execution stays outside the model, so a new vendor is an adapter rather than a migration, and every existing producer keeps working.

The problem

There is no shared model. There are four — and each new need became a new system because the existing tables structurally refuse new kinds of test.

Three constraints do the damage:

• prompt_test_cases.call_id is NOT NULL. Every case must originate from a real call, so a generated or hand-written case has nowhere to live.
• prompt_test_runs.assistant_config_id is NOT NULL. A run can only target a config, so testing against a set of recorded calls cannot be expressed.
• A verdict is a JSON blob, and assertions exist only at judge time. Nothing is queryable per assertion, so no behaviour can be trended and no assertion can be shared between cases.

Four lineages, and no foreign key between any two of them. A solid line is a real foreign key; a dashed one is a join on a vendor's identifier or a string prefix, which is what holds the Simulation v2 lineage together.

The proof that this is structural rather than neglect is what three capable engineers each did instead of extending it, in the last four weeks.

What three engineers did instead

Mitchell needed run state — outcome, lease, scores, error code — and had no columns, so fifteen fields went into config_snapshot, a column that means something else entirely.

Caleb needed assertions and had no table, so the content lives in Hamming and we keep only vendor IDs.

The chat lane needed a run and had no home, so it writes into the Hamming voice tables, separated by a chat- string prefix.

Nobody made a mistake. The model had no seat for any of them.

Why now

Avoca configures a growing variety of customers on one system. At that variety reliability stops being something care can deliver, and starts requiring what software gets: build, test, deploy.

Configuration changes arrive from several surfaces — manual Blueprint edits, Avo, the knowledge base, call replication — and each is growing its own checks. Without one place recording what was tested and what it concluded, no gate can span them. "Is this configuration safe to ship" gets a different answer per surface, none of which compose, and reliability stays a function of who happened to make the change.

Decisions

One data model, adapters at the edges. Not a central engine that runs tests. Three of the four runners are vendors we cannot make implement our interface, so the model defines what a result is and each runner gets a translation. A new vendor becomes an adapter rather than a migration.

The atom is one result per assertion. Not one verdict per case. This is the difference between a behaviour that can be trended across runs, configurations and teams, and a blob that can only be read. Every other table exists to produce this row or to make it interpretable.

Evidence is separate from judgement. A case run holds what came back; an outcome holds what a judge concluded. The same experiment can be judged again — by a new assertion, or a better model — without rewriting what the run said.

Assertions are versioned, and a result references the version. Not the text. A behaviour's history survives its own rewording, which is the only way a trend means anything over months.

New tables; the legacy ones are frozen. Not extended. Each was built for one lineage, and its constraints encode that purpose. Generalising means dropping the very constraints that define the table — which leaves a new model wearing an old name and carrying old rows.

The model

Nine tables. Four describe what is to be tested, five record what happened when it ran and what judging it concluded.

Audit columns are assumed and left out of the diagrams for legibility: created_at everywhere, updated_at only where a row mutates after insert. Its absence on assertions and assertion_results is deliberate — those rows are designed never to change, which is what the versioning scheme rests on.

Every line is a foreign key, labelled with the column that holds it; the child is always on the crow's-foot end.

New tests are written as cases. Suites group them for a target. Assertions live on either a case or a suite, never both — a case-level assertion travels with its case, a suite-level one applies to every case in the run.

Assertions are versioned. assertion_key names the behaviour, version names the wording. A result points at the version it was judged against, so rewording starts a new version and past runs keep their meaning.

Cases are not shared between assistants. A case carries one team's facts, so what travels is the template it came from (source_recipe_id).

Running a suite creates a run, and each case in it becomes a case run. The case run stores what came back and nothing we concluded: transcript, tool calls, timing, and the provider's own payload — including its verdict, where it has one. A vendor saying PASSED is evidence, kept as evidence. Its status says whether the case executed, not whether it was any good.

Judging is a separate act against that stored evidence. Each pass is an outcome, and there can be more than one — a new assertion, or a better judge. A pass writes one result per assertion. The evidence is never rewritten.

A run's outcome is stamped once, when it finalises, against the pass_policy it snapshotted at start. Later passes add outcomes. They never change what the run said.

How tests and suites are defined

How test execution is recorded

Runners

The model does not run tests. Runners do, and most are not ours.

Runner	Executes	Evaluates
Hamming	remote, voice	remote — its own guardrails
Platform chat (VAPI / ElevenLabs)	remote, text	local — our judge
VAPI-native scorers	remote	remote — its own scorers
Ours	nothing, for a structural check	local

The run table already assumes this: lease_expires_at, deadline_at, timed_out, and request_id are meaningless for an in-process function. They exist because a runner can die, may never return, and may be retried by something we do not control.

The contract:

1. Claim the run's lease.
2. Record what came back as a case run.
3. Open an outcome.
4. Write one result per assertion.
5. Renew the lease while working.
6. Release at the end.

Two columns exist for this boundary. assertion_results.evaluator records which runner produced a row, because the same rubric judged by our judge and by a vendor's guardrail are two different measurements. assertions.role tells the outbound translation which of our assertions to put in a vendor's designated slot — it is not our pass policy, which stays required plus the run's pass_policy.

What it unlocks

• A publish gate. "Is this configuration safe to ship?" needs one run record spanning structural and behavioural checks. No table can hold one today.
• A verifier Avo can trust. Many Blueprint edits are structural — did the clause survive compilation, is the tool still attached — and those are deterministic checks that return in under a second. Today the same question costs a thirty-minute replay that runs no tools and returns a score.
• Fleet-wide answers. A behaviour becomes a queryable row, so a failure common to forty teams stops being invisible.
• Changing the definition of passing costs nothing. Because a result is a row rather than a blob, a new pass policy can be applied to every historical run without re-running anything. Today that is impossible — verdict is JSON, so re-scoring means re-executing. Three tiers, and only the first is new: re-score (same evidence, new policy — a query), re-judge (same experiment, new judge — model calls), re-run (new experiment — the platform, real calls).

How we get there

The chat lane moves first. It is unreleased, so it has no readers to disrupt, and it currently writes into the Hamming voice tables behind a string prefix.

Each surface has one authoritative store at any moment. There is no dual-write: a surface's port deletes its old write path in the same change, and legacy stays authoritative for a legacy surface until that surface moves.

Legacy cases and runs import once, one-way, as history. Legacy results import as single opaque rows rather than fabricated per-assertion ones — the verdicts they hold were never stable identities, and inventing them would poison the atom the model exists to make trustworthy.

The first extension

Sharing lives outside the nine tables: assistant_test_templates for a shareable recipe, assistant_test_template_groups for the packs a new team starts from.

Simulation v2 already proves the shape — test_suite_templates is global and records which recipe produced which case. Adding both changes nothing in cases, runs, or results, which is the goal's claim made concrete.

Unknowns and unsures

Does anything not fit? This was designed against Blueprint configuration first, in a shape meant to generalise. Message agents fit — the chat lane is the first producer. Speed-to-Lead routing, campaign orchestration, and SMS delivery do not: they have no assistant configuration and no conversation, and a run target is an assistant config or a call set. Whether they should ever live here, or belong in their own model, is unresolved.

Is "assistant" the right word? agent_test_* is equally free. assistant_* matches the family being tested (assistant_configs, assistant_variables) and avoids colliding with "sub-agent", which our own sub_agent_name column uses. Best available, not obviously right.

Reusable cases, or templates that produce identical ones? The model currently assumes the second.

	Shared case	Template-instantiated
For	One row, one history, edited once	Carries team facts — ZIPs, job types, hours
For	Directly comparable across teams	Matches what Simulation v2 already does
Against	A case holding one team's ZIPs cannot be another's	N copies to keep in step
Against	—	Editing a template does not reach cases already made

Unresolved: a structural check is genuinely team-agnostic, so it may want sharing directly rather than copying — which would mean both mechanisms, not one.

Are the legacy tables actually frozen? Blueprint validation shipped onto prompt_test_* on 2026-08-14 and has an active design behind it. Freezing those tables is a cross-team agreement we have not had yet, not a decision this document can make alone.

Cases are not versioned; assertions are. Editing a case changes what an old run appears to have tested. Assertions solved this by versioning and having results reference the version. Cases did not, and it is not obvious they should — the alternative is that a case run snapshots the case content it executed.

Oversight already has this atom. tag_values and eval_results are per-call, per-definition judgment rows — the same shape as an assertion result, on production calls rather than test runs. Whether these converge or stay separate is unexamined.

Does pass policy need more than a column? Today it is all or threshold over the assertions marked required. A criteria table — per-assertion weights, different criteria per target, expressions over assertions — would be more expressive, and nothing yet demands it. Deferred rather than closed.

Hamming's assertion identity is a mutable name. Their join key is assertionName; ours is assertion_key plus version, precisely so a rewording does not break history. At that boundary the guarantee is defeated — rename a guardrail on their side and the correspondence breaks silently. Unresolved whether the mapping is stored and repaired, or whether we accept voice results being identified more weakly than the rest.