Appearance
Simulation Testing v2
Avoca's second-generation voice-agent test system. This page is the canonical context for simv2 — read it before proposing changes to voice-agent testing. Everything here was verified against origin/main on 2026-08-12; the Merge state section at the bottom says what is not on main, and how to re-check.
Ownership
Simulation v2 is Avoca-owned (dashboard: settings/responder/simulation-testing-v2, gated avocaOnly while in active development; PR authors Zakariyya, Mohamed). This page is our working model of it and the record of what we contribute — not Avoca's canonical docs.
What it is, in one paragraph
A semi-customer-facing surface, in responder settings, for running scenario tests against a team's real agent before it goes live. The tests are generated from that team's Knowledge Base, so they check the agent against what this customer actually does rather than a generic script. It is, in effect, blueprint testing — but it lives entirely outside the blueprint building environment. The audience is internal: CS, TAMs, integration people, FDEs.
The core idea: Avoca owns the tests, Hamming executes them
Simulation v2 applies the Blueprint pattern to tests. Just as a Blueprint stores the prompt definition in Avoca's database and compiles-and-ships it to the voice platform per call, simulation v2 stores the test definition in Avoca's database and compiles-and-ships it to Hamming per run.
Hamming generates nothing of its own and decides nothing. It is the run engine and the result store.
Two kinds of test, with different lifecycles
| Suite templates | Ad-hoc tests | |
|---|---|---|
| Authored | Avoca-owned, global (no team scope) | By a person, in our dashboard UI |
| Stored as | Recipes in our DB (test_suite_templates, test_suite_template_cases) | A finished case, pushed to and stored in Hamming |
| Compiled | Just in time, at run | Once, at creation |
| Reusable across teams | Yes — same recipe, different team's facts | No |
Suite templates carry generation_instructions, focus_areas, and a tier. Core Business Suite v2 is 28 recipes (10 common, 18 advanced; 20 of them exercise tool calls). At run time the generate job (hamming-generate-team-test-plan → lib/integrations/hamming-generation.ts) compiles each recipe into a concrete case with one gpt-5.2 call, grounded in seven blocks of the team's own data: Knowledge Base facts, services & FAQ, programs & guidance, agent config, fees, transfer reasons, and service-area ZIPs. team_test_plans.hamming_case_ids / hamming_plan_id is the receipt.
Ad-hoc tests are authored in CreateTestCaseDialog — manual scenario-fact rows plus an optional LLM-assist (/api/hamming/test-cases/generate) — then POSTed to Hamming, where they persist. ImportFromCallDialog (/api/hamming/calls/convert) pins a real production call as a test case.
Consequence: the compiled case is not ours
For both kinds, the finished case — the caller's role-play script and the expected-outcome rubric — lives in Hamming. Locally we mirror only hamming_test_case_facts: scenario facts, a title, and a suite_template_case_id provenance link.
Anything that removes Hamming must therefore either recompile the cases or first give them a home on our side.
Anatomy of a test
| Part | What it does | Where it lives |
|---|---|---|
| Persona + instructions | Who the caller is and how they behave | Hamming |
| Expected agent output | - bullet rubric the transcript is graded against | Hamming |
| Scenario facts | The customer details the caller states — defaults customerName, address, serviceType; free-form key/value | Hamming, mirrored to hamming_test_case_facts |
| Tool-call mocks | Per-tool canned responses | Avoca (tool_call_mocks, keyed by test_case_id + tool_name) |
| Context mocks | The calling-from number and the effective date/time | Avoca — see Merge state |
Scenario facts control what the caller says. Context mocks control what the system believes. They are not the same lever, and before context mocking landed there was no way to pull the second one.
The ownership line runs through the middle of a single test
We own the authoring and the mocking. Hamming owns the case record. And tool_call_mocks is keyed by a Hamming test_case_id — so dropping Hamming orphans every existing mock row's key. That is a migration, not a swap.
Execution
Hamming places a real voice call to the team's test phone. The agent answers through the normal inbound path, so the reasoning path is real: real prompt, real compiled agent, real dispatcher. Only leaf tool calls are substituted.
Config targeting. A run goes against the voice assistant's default config, or a per-run override: team_test_plans.assistant_test_config_id routes it (the test phone is re-synced to that config for the duration), while tested_config_id is a display-only snapshot for results attribution. Default runs leave assistant_test_config_id NULL by design.
Chat mode is a vestigial limb
useStartTestRun takes mode: 'voice' | 'chat' and startHammingChatTestRun posts to Hamming's /test-runs/test-vapi-chat-agent. It is VAPI-only, it is not in use, and making it work would be real effort. Treat simv2 as voice-only. The direction of travel is off Hamming entirely, not onto its chat mode — see chat-call-replication-spike.
The mocking policy — read this before assuming "it's all mocked"
Mocking is entirely ours. Hamming's call reaches our dispatch endpoint and resolveToolCallMock (lib/tools/test-call-mock.ts) intercepts. Resolution is three tiers:
- Explicit per-case mock (
tool_call_mocks) → returned. - Team/global default (
mock_defaults) → returned, except live-read tools, which skip this tier so a stray defaults row can never blanket-mock them. - No mock → policy by category: write tools are auto-blocked; read tools execute for real.
The category default is 'read' — TOOL_CALL_CATEGORY[toolName] ?? 'read' — so a tool missing from the inventory runs live.
A simv2 test call does real lookups against the customer's real CRM. That is deliberate: customer-independent reads (fees, promos, ZIPs, availability) validate real config and wiring, and findCustomer correctly returns "not found" for a synthetic caller. The explicit live-read list (TEST_LIVE_READ_TOOL_NAMES) is:
findCustomer, getAppointments, getUpcomingAppointments, getPastAppointments, getScheduledMaintenance, getBookingInfo, checkZipCode, lookupAustralianPostcode, getFeeInfo, getPromoInfo, getJobTypeQuestions
Test-call detection fails open: a call it cannot positively identify as a test is treated as production traffic and nothing is mocked.
The write-block breaks transfer testing
Write tools get a canned refusal, and for transfer that refusal is the literal string Transfer is not available during test calls. So the agent takes a message instead, and any "the agent should transfer here" assertion is structurally unpassable — the highest-stakes call outcome is untestable.
The fix (resolve the real destination, record it, narrate it, skip the dial) is designed and built, and is not on main. See simulation-v2-transfer-mocking and Merge state.
What simv2 structurally cannot check
Hamming only sees the transcript. It can tell you what the agent said; it cannot tell you what the system did. Whether a job was actually created, whether is_booked was set, whether the call reason was classified correctly, whether the transfer log recorded the right destination — all of that is deterministic, visible from our own data and from what the platform hands us at end of call, and entirely outside Hamming's reach.
That is a separate axis, not a gap in simv2's implementation. See Deterministic Verification and the postcall-testing plan.
Known weaknesses
- Slow. Every case is a real phone call.
- Whole-call granularity. No unit-level test of a single turn or decision.
- Vendor-coupled. Hamming's API and UI stability gate all of it.
- Needs a provisioned test phone, so a team without one cannot be tested.
- Write-block makes transfer assertions unpassable (above).
Merge state
Snapshot: 2026-08-12. Most testing work is on branches, not main — this is the single biggest source of confusion when reasoning about this system. Verify before relying on any of it.
| Capability | On main? | Where it is |
|---|---|---|
| Suite templates, ad-hoc tests, tool-call mocks, voice runs | Yes | main |
Transfer simulation fix (perform-simulated-transfer.ts) | No | 3 branches — fix/hamming-transfer-simulation-response and transfer-functionality-testing (identical), postcall-testing (diverged) |
Context mocking — calling-from number (comingFrom) | No | call-context-override-phone-number. Expected to merge |
Context mocking — effective date (asOf) | Partly | Primitive merged; consumers on branches |
Single-turn replay harness (lib/blueprint-regression/) | No | unit-integration-tests-poc only |
VAPI simulation runner (lib/simulation/) | Yes | main — but hardcoded cases, no squads, no tools |
Re-check with:
bash
git -C <avoca-next> ls-tree origin/main --name-only apps/web/lib/<path>/
git -C <avoca-next> log --all --oneline -1 --diff-filter=A -- <file>Where this sits relative to the other testing docs
Three bodies of writing, different jobs — check which one you need before adding a fourth:
| Job | |
|---|---|
state-of-testing/ | The argument. A deck of one-paragraph pages: what exists, what it structurally can't do, where we're going (two verdicts per case, deterministic Oversight assertions, regression gates). Read The Gap and Where We're Going before proposing direction. |
| This page | The reference. How simv2 actually works today, in enough operational detail to change it safely. |
The rest of testing/ | The depth. One mechanism per page — mocks, code paths, seams, how-tos. |
Related pages
- Tool Call Mocks — state ownership, tier resolution, terminology
- Code Path: Live vs Mock — the full traced path
- Deterministic Verification — what the transcript can't tell you, and the two-verdict proposal
- The
comingFromPrimitive — calling-from override - Team Model & Test Seams — where a suite can hook in
- How-To: Test the Agent's Transfer Decision
- How-To: Register a Hamming Agent
Stale sibling
index.md still opens with "Hamming is the canonical testing platform; we do not fragment into Vapi Test Suites or other tools," which is the opposite of the current direction, and its Status section tracks archived plans. Treat this page as authoritative until that one is rewritten.