Avoca — Internal

Appearance

WireMock Justification ​

When is the WireMock-based mock CRM the right tool, when is Avoca's existing Tool Call Mocks layer the right tool, and when do you combine them. Frames the architectural decision, not the implementation.

TL;DR decision ​

You're trying to...Use
Test agent reasoning ("did the agent invoke the right tool with the right args?")Tool Call Mocks
Stand up a CRM stub for a brand new client / CRM you don't have API access to yetWireMock
Build and iterate on a new tool against a CRM you do have access toWireMock
Diagnose a production booking that didn't land — figure out which side of the wire is brokenTool Call Mocks for in-call + WireMock for post-call verification
Validate the agent's full reasoning AND the integration's full write pathTool Call Mocks for some tools + WireMock for post-call writes
Test stateful multi-call flows (book → next availability excludes the slot)WireMock with scenarios
Validate the post-call integration code (LLM extraction, timezone conversion, payload assembly) — not just the agent's reasoningWireMock with executed post-call

The four justifications below explain why WireMock earns its existence next to Tool Call Mocks.

Justification 1: Rapid setup for new client engagements ​

When Avoca is onboarding a new client whose CRM we've never integrated against, we don't have an API key, don't have a sandbox, sometimes don't even have the OpenAPI spec yet. Standing up a WireMock instance against the CRM's published documentation gets us a working substrate in hours, not weeks.

Concretely:

  • Write a small set of canned stubs matching the documented endpoints
  • Wire it as a destination from the avoca-next CRM client
  • Demo the full agent flow end-to-end to the prospect / client
  • They see their agent booking, looking up customers, etc. against shapes that match their real system

Value: unblocks demo cycles + integration development. We don't need to wait for the client to ship us creds before we can show working software.

Cost: initial stubs are best-guess from documentation; first real-prod call may surface drift. The Recording mode discovery loop closes that gap.

This is where the SAME WireMock instance pulls double duty: today it's a demo substrate, tomorrow (after recording from real prod) it's a test substrate.

Justification 2: Iterating on a new tool at scale ​

When we ship a new in-call tool (say a new autoOpsCheckLoyaltyTier), we want to validate it against hundreds of scenarios — different customer types, different shop states, different edge cases. Doing that against the real CRM is hard:

  • Rate limits get hit
  • State pollution piles up (customer records, jobs, etc.)
  • Test data setup is laborious — each scenario needs prep against real systems
  • Cleanup is incomplete and slow
  • Real CRMs sometimes throttle or rate-cap us during heavy testing

WireMock lets you iterate on a new tool's prompt + payload shape 500-600 times without:

  • Touching the real CRM
  • Worrying about cleanup
  • Hitting rate limits
  • Polluting prod data

Value: tool development becomes a fast feedback loop. Author the stub once, then iterate prompt + tool args against deterministic responses until the agent's behavior is exactly right. Then validate against real CRM once at the end.

Important nuance: Tool Call Mocks can also do this for individual tool invocations. WireMock wins when the tool's behavior needs realistic CRM-shape responses (e.g., a tool that fetches a customer record and the agent needs to react to the full payload). For tools whose stub is "here's the canned response," Tool Call Mocks are simpler.

Justification 3: Diagnosing production booking failures (and the decoupling insight) ​

The most common operational scenario: someone notices bookings aren't landing for a team, or a specific call's booking is wrong. We need to isolate the fault:

  • Possibility A: the real CRM is up but its behavior changed (new field, new validation rule, rate limit) — our system is now mis-talking to it
  • Possibility B: our post-call workflow is misbehaving — extracting the wrong scheduledAt, wrong service ID, malformed customer record
  • Possibility C: the agent in-call is confirming something that the post-call workflow can't translate

WireMock isolates Possibility B from A. Run the same prompt against WireMock-mocked CRM; if it works there but fails in prod, it's a CRM-side change (Possibility A). If it fails against WireMock too, the bug is in our post-call workflow (Possibility B).

This is WireMock's bread and butter for production debugging: deterministic, cheap, fast verification of "are WE producing the right outputs?" without needing real-CRM access for every test.

The decoupling insight ​

For diagnostic use, you do NOT need to mock the in-call tool responses with WireMock. The in-call agent can use Tool Call Mocks (existing Avoca infrastructure, fast to author per scenario in the Simulation Testing UI). WireMock's job is to receive the post-call write and let you verify the payload.

Concretely:

  • Set up Tool Call Mocks for the in-call tools (availability, customer-lookup, services) — these are quick to author, give you deterministic agent behavior
  • Let the post-call workflow run normally — its book POST hits WireMock
  • Verify WireMock's journal: was the booking payload shaped correctly?

The two layers don't have to be coupled. Use Tool Call Mocks for what they're good at (per-tool, per-case, in-call) and WireMock for what it's good at (verifying the HTTP write).

This decoupling matters because:

  1. Tool Call Mock authoring is faster per case. Adding a customer-lookup variant per test case is one Supabase row; doing the same in WireMock requires authoring a stub mapping with the matcher logic.
  2. WireMock doesn't need to model 100% of CRM behavior to be useful. It just needs to receive and record the post-call write.
  3. In-call tool changes don't require WireMock stub updates. If avoca-next changes how it calls customer-lookup, you update the Tool Call Mock, not a WireMock stub.

The Mock CRM via WireMock is most valuable as a post-call verification surface, not necessarily as the full in-call response substrate.

Justification 4: Catching bugs in the post-call code path that Tool Call Mocks structurally can't surface ​

Tool Call Mocks intercept at the Vapi tool dispatcher — they short-circuit a tool invocation and return a canned response. Anything that runs after the dispatcher hands the call off to the post-call workflow is invisible to that layer.

The post-call workflow does substantial work between "agent committed to a slot" and "HTTP call to the CRM":

  • LLM extraction of customer, vehicle, service, scheduledAt, and address from the call transcript
  • Timezone normalization via convertScheduledAtToUtcIso, converting the LLM's local-time output to UTC using the team's IANA timezone
  • Service catalog matching against the response from getServices to map "oil change" to a specific srvc_* ID
  • Payload assembly for the actual book POST: customer (existing ID vs new-customer object), vehicle (existing vs new), address, idempotency-key generation
  • Pre-flight verification that the chosen slot is still available before committing
  • Cross-shop resolution when the booking shop differs from the calling shop

Each of these is a class of bug that Tool Call Mocks cannot surface, because in the Tool Call Mocks model the post-call path never runs — the dispatcher short-circuited the tool call and the system never reaches the booking workflow.

Running the post-call workflow against a deterministic backend is the only way to test this layer. That's WireMock's distinctive value vs Tool Call Mocks: same agent reasoning, same tool args, but you actually execute the full code path between "agent decided" and "HTTP request sent."

Concrete example: the timezone bug ​

A demo session surfaced a real bug exactly this way. The agent reasoned correctly (committed to "9 AM Friday"), every in-call tool returned what the prompt expected, the transcript looked clean. Tool Call Mocks would have passed every assertion.

But the post-call workflow extracted 2026-06-05T09:00:00 from the transcript (local time), called convertScheduledAtToUtcIso(..., 'America/Denver') which produced 2026-06-05T15:00:00.000Z, and then verifyScheduledTimeStillAvailable did a fresh availability GET and looked for a slot at 15:00 UTC. Our mock returned slots at 09:00:00.000Z (UTC 9 AM, not "9 AM Denver converted to UTC"), so the verifier rejected the slot and threw "AutoOps booking: scheduled time '2026-06-05T09:00:00' is no longer available for Waiter Oil Change". The book POST never fired.

This bug is doubly diagnostic: it surfaced the mock-fidelity gap (our stub returned UTC-of-literal-hour instead of UTC-of-local-instant, which is what real AutoOps does), AND it confirmed that the post-call code path runs significant logic that needs to be exercised, not bypassed. Tool Call Mocks would have shown a passing test. The agent in-call was fine. The integration was wrong.

Why this matters for the testing strategy ​

The Tool Call Mocks layer is good for what it does: per-tool, per-test-case agent reasoning validation. It's not designed to test the integration code path. WireMock + executed post-call workflow tests precisely the layer Tool Call Mocks can't reach.

For any test suite that needs confidence the whole booking actually works, not just that the agent said the right thing, the post-call must run against a deterministic backend. That backend is WireMock.

When NOT to use WireMock ​

  • Single-call, single-tool agent reasoning tests — Tool Call Mocks are simpler. Don't over-engineer.
  • Tests that need stateful behavior on the read side that's hard to model — WireMock can do scenarios but it's not free. If the test is simple, simple mock layer wins.
  • Production debugging where the CRM is the suspect — WireMock can't replicate "the real CRM's behavior changed." For that, you need a snapshot-from-prod comparison (recording mode + diff), not just stubs.
  • Tests that don't care about HTTP shape — if you're just validating agent decisions, you're paying the HTTP transport tax for no value.

How this maps to our 50-scenario suite ​

For a 50-test-case batch testing a new prompt version, the practical split:

  • 35-40 scenarios test agent decisions (what tool was called, what was said, did transfer fire). → Tool Call Mocks. Faster authoring. Smaller blast radius.
  • 10-15 scenarios test the FULL CHAIN including post-call workflow (was the booking correctly extracted, did the right shopClientId get written). → WireMock for the post-call verification. The in-call portion can still use Tool Call Mocks if you want.

Don't try to make WireMock the universal test surface. The architecture is cleaner — and the pitch to Avoca more credible — when each layer does what it's best at.