Assistant Test Model Design

Date: 2026-08-18 Status: Draft — schema in review, avoca-next #17479

The eleven tables are open as a draft PR:avoca-next #17479 — [Testing] Add assistant_test_* schema. Migration only; no application code reads them yet. What is described below is what that migration creates, verified against a live database — the full insert chain, the (outcome_id, assertion_id) atom rejecting duplicates, and both CHECK constraints rejecting violations.

Where the migration adds columns this document does not show — team_id on the team-scoped tables, name, the evidence pointers on case runs — those are implementation detail deliberately kept out of the diagrams, and recorded in full in the plan's Phase 1 notes.

Success criteria are their own table, keyed by suite and consumer, so one suite run once yields a verdict per interested party over a single set of evidence.

The earlier shape — a single pass_policy column on the suite — is kept for comparison at Alt 1.

Goal

Consolidate agent testing onto one data model, so that a new vertical or a new kind of test extends it rather than becoming another system.

Solution

One namespaced foundation, assistant_test_*, that today's testing surfaces move onto and tomorrow's extend. What forced a new system each previous time — a case that had to come from a call, a run that had to target a configuration, a verdict locked in a blob — becomes data rather than structure. Execution stays outside the model, so a new vendor is an adapter rather than a migration, and every existing producer keeps working.

The problem

There is no shared model. There are four — and each new need became a new system because the existing tables structurally refuse new kinds of test.

Three constraints do the damage:

• prompt_test_cases.call_id is NOT NULL. Every case must originate from a real call, so a generated or hand-written case has nowhere to live.
• prompt_test_runs.assistant_config_id is NOT NULL. A run can only target a config, so testing against a set of recorded calls cannot be expressed.
• A verdict is a JSON blob, and assertions exist only at judge time. Nothing is queryable per assertion, so no behaviour can be trended and no assertion can be shared between cases.

Four lineages, and no foreign key between any two of them. A solid line is a real foreign key; a dashed one is a join on a vendor's identifier or a string prefix, which is what holds the Simulation v2 lineage together.

The proof that this is structural rather than neglect is what three capable engineers each did instead of extending it, in the last four weeks.

What three engineers did instead

Mitchell needed run state — outcome, lease, scores, error code — and had no columns, so fifteen fields went into config_snapshot, a column that means something else entirely.

Caleb needed assertions and had no table, so the content lives in Hamming and we keep only vendor IDs.

The chat lane needed a run and had no home, so it writes into the Hamming voice tables, separated by a chat- string prefix.

Nobody made a mistake. The model had no seat for any of them.

Why now

Avoca configures a growing variety of customers on one system. At that variety reliability stops being something care can deliver, and starts requiring what software gets: build, test, deploy.

Configuration changes arrive from several surfaces — manual Blueprint edits, Avo, the knowledge base, call replication — and each is growing its own checks. Without one place recording what was tested and what it concluded, no gate can span them. "Is this configuration safe to ship" gets a different answer per surface, none of which compose, and reliability stays a function of who happened to make the change.

Decisions

One data model, adapters at the edges. Not a central engine that runs tests. Three of the four runners are vendors we cannot make implement our interface, so the model defines what a result is and each runner gets a translation. A new vendor becomes an adapter rather than a migration.

The atom is one result per assertion. Not one verdict per case. This is the difference between a behaviour that can be trended across runs, configurations and teams, and a blob that can only be read. Every other table exists to produce this row or to make it interpretable.

Evidence is separate from judgement. A case run holds what came back; an outcome holds what a judge concluded. The same experiment can be judged again — by a new assertion, or a better model — without rewriting what the run said.

Assertions are versioned, and a result references the version. Not the text. A behaviour's history survives its own rewording, which is the only way a trend means anything over months.

New tables; the legacy ones are frozen. Not extended. Each was built for one lineage, and its constraints encode that purpose. Generalising means dropping the very constraints that define the table — which leaves a new model wearing an old name and carrying old rows.

The model

Eleven tables. Four describe what is to be tested, two describe who is asking and what would satisfy them, and five record what happened when it ran and what judging it concluded.

Audit columns are assumed and left out of the diagrams for legibility: created_at everywhere, updated_at only where a row mutates after insert. Its absence on assertions and assertion_results is deliberate — those rows are designed never to change, which is what the versioning scheme rests on.

Every line is a foreign key, labelled with the column that holds it; the child is always on the crow's-foot end.

New tests are written as cases. Suites group them for a target. Assertions live on either a case or a suite, never both — a case-level assertion travels with its case, a suite-level one applies to every case in the run.

Assertions are versioned. assertion_key names the behaviour, version names the wording. A result points at the version it was judged against, so rewording starts a new version and past runs keep their meaning.

Cases are not shared between assistants. A case carries one team's facts, so what travels is the template it came from (source_recipe_id).

Running a suite creates a run, and each case in it becomes a case run. The case run stores what came back and nothing we concluded: transcript, tool calls, timing, and the provider's own payload — including its verdict, where it has one. A vendor saying PASSED is evidence, kept as evidence. Its status says whether the case executed, not whether it was any good.

Judging is a separate act against that stored evidence. Each pass is an outcome, and there can be more than one — a new assertion, or a better judge. A pass writes one result per assertion. The evidence is never rewritten.

A run's outcome is stamped once, when it finalises, against the success criteria it referenced at start. Later passes add outcomes. They never change what the run said.

A run is a suite crossed with a criteria: the suite decides which situations are exercised, the criteria decides what has to be true of the results. The same suite judged under two criteria gives two verdicts over one set of evidence, and neither re-runs anything.

How tests and suites are defined

How test execution is recorded

Runners

The model does not run tests. Runners do, and most are not ours.

Runner	Executes	Evaluates
Hamming	remote, voice	remote — its own guardrails
Platform chat (VAPI / ElevenLabs)	remote, text	local — our judge
VAPI-native scorers	remote	remote — its own scorers
Ours	nothing, for a structural check	local

The run table already assumes this: lease_expires_at, deadline_at, timed_out, and request_id are meaningless for an in-process function. They exist because a runner can die, may never return, and may be retried by something we do not control.

The contract:

1. Claim the run's lease.
2. Record what came back as a case run.
3. Open an outcome.
4. Write one result per assertion.
5. Renew the lease while working.
6. Release at the end.

Two columns exist for this boundary. assertion_results.evaluator records which runner produced a row, because the same rubric judged by our judge and by a vendor's guardrail are two different measurements. assertions.role tells the outbound translation which of our assertions to put in a vendor's designated slot — it is not our pass policy, which lives entirely in the run's success criteria.

What it unlocks

• A publish gate. "Is this configuration safe to ship?" needs one run record spanning structural and behavioural checks. No table can hold one today.
• A verifier Avo can trust. Many Blueprint edits are structural — did the clause survive compilation, is the tool still attached — and those are deterministic checks that return in under a second. Today the same question costs a thirty-minute replay that runs no tools and returns a score.
• Fleet-wide answers. A behaviour becomes a queryable row, so a failure common to forty teams stops being invisible.
• Changing the definition of passing costs nothing. Because a result is a row rather than a blob, a new criteria can be applied to every historical run without re-running anything. Today that is impossible — verdict is JSON, so re-scoring means re-executing. Three tiers, and only the first is new: re-score (same evidence, new policy — a query), re-judge (same experiment, new judge — model calls), re-run (new experiment — the platform, real calls).

How we get there

The chat lane moves first. It is unreleased, so it has no readers to disrupt, and it currently writes into the Hamming voice tables behind a string prefix.

Each surface has one authoritative store at any moment. There is no dual-write: a surface's port deletes its old write path in the same change, and legacy stays authoritative for a legacy surface until that surface moves.

Legacy cases and runs import once, one-way, as history. Legacy results import as single opaque rows rather than fabricated per-assertion ones — the verdicts they hold were never stable identities, and inventing them would poison the atom the model exists to make trustworthy.

The first extension

Sharing lives outside these tables: assistant_test_templates for a shareable recipe, assistant_test_template_groups for the packs a new team starts from.

Simulation v2 already proves the shape — test_suite_templates is global and records which recipe produced which case. Adding both changes nothing in cases, runs, or results, which is the goal's claim made concrete.

Unknowns and unsures

Does anything not fit? This was designed against Blueprint configuration first, in a shape meant to generalise. Message agents fit — the chat lane is the first producer. Speed-to-Lead routing, campaign orchestration, and SMS delivery do not: they have no assistant configuration and no conversation, and a run target is an assistant config or a call set. Whether they should ever live here, or belong in their own model, is unresolved.

Is "assistant" the right word? agent_test_* is equally free. assistant_* matches the family being tested (assistant_configs, assistant_variables) and avoids colliding with "sub-agent", which our own sub_agent_name column uses. Best available, not obviously right.

Reusable cases, or templates that produce identical ones? The model currently assumes the second.

	Shared case	Template-instantiated
For	One row, one history, edited once	Carries team facts — ZIPs, job types, hours
For	Directly comparable across teams	Matches what Simulation v2 already does
Against	A case holding one team's ZIPs cannot be another's	N copies to keep in step
Against	—	Editing a template does not reach cases already made

Unresolved: a structural check is genuinely team-agnostic, so it may want sharing directly rather than copying — which would mean both mechanisms, not one.

Are the legacy tables actually frozen? Blueprint validation shipped onto prompt_test_* on 2026-08-14 and has an active design behind it. Freezing those tables is a cross-team agreement we have not had yet, not a decision this document can make alone.

Cases are not versioned; assertions are. Editing a case changes what an old run appears to have tested. Assertions solved this by versioning and having results reference the version. Cases did not, and it is not obvious they should — the alternative is that a case run snapshots the case content it executed.

Oversight already has this atom. tag_values and eval_results are per-call, per-definition judgment rows — the same shape as an assertion result, on production calls rather than test runs. Whether these converge or stay separate is unexamined.

Success criteria are a third axis, not a column. This is the change this alternative makes.

Three things vary, and the base model collapses two of them:

Axis	Bound to	Answers
cases	the team	what situations exist
suites	an assistant	which situations get exercised
criteria	a suite and a consumer	what has to be true of the results
consumers	the application	who is asking, and what they are called

A suite is an assistant's selection of situations, and nothing more. A criteria is one consumer's opinion about that suite — voice, chat, an Avo pre-flight — so a suite carries as many criteria as it has interested parties. Cases stay team-owned and are shared into as many suites as want them, which is what the many-to-many exists for.

That (suite, consumer) key is the point. One suite run once produces one set of evidence, and every consumer reads a verdict out of it under their own criteria — no duplicated suites, no re-execution, and no consumer's bar imposed on anyone else.

Criteria is a separate table rather than more columns on the suite for one reason: it has to be immutable and versioned, and the suite does not. A run stamps the criteria version it was judged under, so a verdict stays readable after someone raises the bar. A suite meanwhile gains and loses cases freely, and nothing about that should mint a new version of the standard.

Consumers are application-level, not team-level

assistant_test_consumers carries no team_id. A consumer is a thing in the product that asks a question — blueprint CI/CD, the publish gate, an Avo pre-flight, a nightly sweep — and those are the same four things for every team. A team does not invent consumers; it inherits them and decides what its suites have to satisfy for each.

This is the row that connects a suite to blueprint CI/CD. The pipeline knows one stable key, resolves the criteria for (suite, consumer), and judges the run under it. Without the table that key is a string spelled independently in two codebases, and a typo produces no criteria, no verdict, and no error — the run simply comes back with nothing to say.

It is a lookup table rather than a text column or an enum on purpose. The repo's standing rule is that enumerated value sets do not belong in the database as enum types or CHECK constraints, because they cannot be evolved without a migration. A table sidesteps both problems: a new consumer is an insert, and the foreign key still catches a criteria pointing at a consumer that does not exist.

What a criteria holds

A spec naming what matters, by assertion_key — never by assertion row id, because a key is the stable identity across versions and across suites that each carry their own copy of the same assertion.

{
  "required":  ["no_raw_liquid", "every_variable_resolves"],
  "threshold": [{ "keys": ["answers_question", "no_invented_commitment",
                           "stays_in_role", "offers_next_step"], "min": 0.75 }],
  "ignore":    ["mentions_business_hours"]
}

Required keys must all be met. Each threshold group must clear its ratio. Anything ignored is still evaluated and recorded — it simply does not affect the verdict, which is what makes an assertion safe to add before anyone has decided whether it should block.

An assertion therefore carries no required flag of its own. Whether a behaviour has to hold is a consumer's opinion, not a property of the behaviour: blueprint CI/CD may demand a clause that a nightly sweep only observes. An assertion states what to check; a criteria states who has to care.

JSON is defensible here in a way it would not be for a result. The model's objection to blobs is about measurements — those must be rows so they can be trended. A criteria spec is configuration: it is read to interpret rows, never counted as one.

The verdict is a projection, not a gate

A consumer's criteria is a relatively unbinding statement about what they care about, and runs.outcome is a projection over stored results — derived, never authoritative. Nothing in the model requires anyone to act on it.

This matters more than it sounds, because it decides how much the criteria design has to be right:

• The durable thing is the result rows. One per assertion, each with its met, value and evaluator. Those are measurements and they stand on their own. A verdict is arithmetic over them.
• A wrong criteria costs a re-read, not a re-run. Edit the spec, stamp a new version, recompute over evidence already stored. No call is placed.
• If the column proves unused, it can be dropped and nothing is lost. A publish gate that ends up reading three specific assertion keys directly, rather than a rolled-up pass, is a legitimate outcome — the rows were always the point.

So the criteria table is worth having because it makes a consumer's intent explicit and comparable, not because a boolean has to come out of the far end. The shapes that were considered and set aside fail this test in a way worth noticing: putting strictness on the case, or under the assertions, welds it to the thing being measured — so being wrong about it means editing tests rather than editing an opinion.

What it drops

assistant_test_suites.pass_policy — superseded, since strictness now lives in the criteria.

team_id stays on suites. It is derivable through voice_assistant_id, but this codebase denormalizes team_id deliberately — the same note appears on prompt_test_cases, tag_values and eval_results — and every team-scoped query would otherwise join through the assistant to get there.

Two risks worth naming

A key that no longer exists fails silently. A criteria naming no_raw_liquid after that assertion is renamed simply matches nothing, and the run passes for the wrong reason. There is no foreign key to catch it. The mitigation is to resolve the spec at run start and fail the run loudly when a required key matches no assertion in the suite — an unmatched required key is an error, not an empty set.

Criteria are edited. criteria_version is stamped on the run for the same reason pass_policy was: a verdict read under a policy that has since changed is uninterpretable. Rows are immutable; an edit writes a new version.

Hamming's assertion identity is a mutable name. Their join key is assertionName; ours is assertion_key plus version, precisely so a rewording does not break history. At that boundary the guarantee is defeated — rename a guardrail on their side and the correspondence breaks silently. Unresolved whether the mapping is stored and repaired, or whether we accept voice results being identified more weakly than the rest.