Avoca — Internal

Appearance

Chat-Based Sim Testing Design ​

Date: 2026-08-18 Status: Draft

Goal ​

Run a full conversation against a candidate config, through the real production code path, without dialling anything.

Solution ​

Both platforms expose a text surface that drives our real tool dispatch — VAPI's Chat API against a scratch squad, ElevenLabs' simulation-type test with mocking off. A test is authored as a situation, translated outbound by a per-platform adapter, and comes back as one outcome that our assertions are judged against.

This is the integration tier: it exercises the whole path — routing, prompt compilation, tool dispatch, multi-turn behaviour — and the only thing it removes is audio.

The problem ​

Voice is the only end-to-end surface we have, and it costs minutes. A Hamming voice suite is the right instrument for a full journey, but it is far too slow and too expensive to sit in front of a deploy. So nothing does, and configs ship having been checked by a person on a phone.

The existing chat lane tested a stripped-down world. Simulation v2's chat path mocked tools by default, so a green run said the model said the right thing — not that the booking would have happened. The interesting failures live in dispatch: a customer lookup that returns nothing, a tool that errors, an argument that arrives malformed.

Adding a platform meant adding a lane. ElevenLabs support arrived beside the VAPI path rather than underneath it, sharing an authoring surface and nothing else.

Why now ​

A spike confirmed the mechanism works — see VAPI + ElevenLabs chat as a real-dispatch test surface. A handful of runs on each platform were driven against real dispatch and checked in ClickHouse (default.public_tool_calls) rather than trusted from vendor docs or the platform's own reported status. Those runs produced real SUCCESS/EXECUTION_ERROR rows at latencies in the 100ms–2s range — the signature of code actually executing, rather than the instant return a mocked placeholder gives.

That is enough to establish the surface is real. It is not yet coverage: the spike drove single messages, and the load-bearing checklist in its smoke-test runbook is mostly unfilled.

The one blocker found — VAPI had no team-resolution path for an ad-hoc scratch squad, so every tool call died before reaching real code — is fixed and merged (#16217). ElevenLabs needed no change at all: webhook-tool-config.ts already binds team_id to the x-team-id header and dispatch already reads it.

Decisions ​

Real dispatch, not mocks, is the default. A chat run exists to exercise the code a call would. Mocking is a deliberate per-case choice for a specific scenario, never the resting state — the value of this tier is precisely that it catches what a mocked run cannot.

One definition, one adapter per platform. The test names a situation — persona, what they want, what must be true — and carries no platform. The adapter builds whatever that vendor accepts and translates the response back into an outcome. This is the adapter contract, not a second mechanism.

Run on whichever platform already runs the agent. A config deployed on ElevenLabs is tested through ElevenLabs. Testing it through VAPI would exercise a path production never takes, and prompt compilation differs between them.

Scratch objects are built and torn down per run. A run provisions its own squad or agent, uses it, and removes it. Nothing accumulates in a vendor account and no run inherits another's state.

The transcript is evidence, not the verdict. Chat runs produce an outcome — transcript, tool calls attempted and executed, any platform-side judgment. Assertions are evaluated against that afterwards, so the same conversation can be re-judged by a better rubric without re-running it.

What a chat run produces ​

The outcome carries more than the words, because the words are the least interesting part:

Why it matters
transcriptwhat was said, turn by turn
tool calls attemptedwhat the model tried to do
tool calls executedwhat actually ran — the gap between these two is a whole class of bug
platform verdictthe vendor's own opinion, kept alongside ours, never instead of it
prompt referencewhich compiled prompt this run actually used

Attempted-versus-executed is the pair worth designing around. A model that calls bookAppointment with a malformed argument looks identical in a transcript to one that booked successfully — the difference only shows in dispatch.

What it does not cover ​

Audio. Interruption, latency, barge-in, transcription error, and everything about how a caller actually experiences a call. Voice remains the instrument for a full journey, and this tier reduces dependence on it rather than replacing it.

Anything post-call. The workflow that writes to a CRM after the call ends is a separate gap, and a chat run stops at the conversation.

How we get there ​

  1. Land the platform-neutral seam — IntegrationTestDefinition and one result shape, with the VAPI translation isolated in its own lane. Built in apps/web/lib/simv2-integration/.
  2. The ElevenLabs adapter against the same definition, proving the seam holds for a second vendor.
  3. Assertions judged locally over the returned outcome, so a vendor's own opinion is an input rather than the answer.
  4. Wire it as a consumer — a suite plus a criteria for blueprint CI, so a config change is checked before it ships.

Unknowns and unsures ​

  • Multi-agent handoff in chat mode specifically. Confirmed in voice, not yet confirmed in chat. A blueprint whose behaviour depends on a sub-agent transfer is untested until it is.
  • Whether a chat run compiles its prompt the way a real call does. This is the trap the VAPI test phone already sprang once — a test surface that dials a pre-built squad never traverses assistant-request, so its prompt is not the one production sends. Must be verified per platform, not assumed.
  • Cost and rate limits at suite scale. A handful of chat runs is cheap. Forty teams' suites on every publish is a different question, and no measurement exists yet.