Brief: The Blueprint Reliability & Testing Initiative

For: Lazer leadership · From: Sandy · Status: draft, 2026-08-02

Scope note: this brief covers the reliability initiative specifically — why it exists, and the justification for work already shipped (Sandy's and others'). A separate, deeper test-plan doc follows this one, covering testing beyond reliability, including performance.

TL;DR

• Blueprint agents run on one giant prompt for speed. That speed costs reliability — the bigger the prompt, the more it breaks in subtle, hard-to-notice ways.
• Two fixes underway, both deliberate: more tool calls (accept latency, get certainty — Peter's push, several of us applauding), and faster, triage-friendly regression testing via better mocking (building on Matt's regression suite).
• Already shipped: test phone number can point at a different blueprint config; call time can be mocked in Hamming tests.
• Still needed: mocking on real, human-placed phone calls — Sandy proposing and demoing this week.
• A deeper test-plan doc (performance, R&D, and more) is coming separately.

The ask

Understand why Avoca is investing in agent reliability testing right now, what shape that investment is taking, and what's already landed versus what's still open. This isn't a request for a decision — it's the context so leadership can follow what the Blueprint/testing work is actually for.

The architecture, in one paragraph

Blueprint — and Avoca generally — builds voice agents around one large prompt: a lot of information, a lot of instructions, the tool calls available during a call, and pre-call context fed in ahead of time. Vapi and ElevenLabs run that prompt against real customers in real time. The upside of one large prompt with minimal tool calls is speed — everything relevant is already loaded into the prompt, up front, so the interaction is a simple query → prompt → response loop, rather than logic broken up across multiple steps. That's the approach in production today, and it's a deliberate choice, not an oversight.

The problem

That architecture is fast but hard to control, and the risk compounds with scale. The longer a prompt gets, the more places information can contradict itself without anyone noticing. And even a perfectly clean, non-conflicting prompt is still sensitive to ordering and wording — the same prompt can produce different results run twice with nothing changed. None of this is unique to Avoca; it's inherent to using AI at all. But the bigger and more monolithic the prompt, the more exposed the system is to it. See the appendix for concrete cases.

Two axes Avoca is pursuing — deliberately, both at once

Axis 1 — Increased tool calls

Shift the tradeoff: instead of avoiding tool calls because they're slow (and accepting the monolith's uncertainty as the cost of speed), accept the latency in exchange for certainty, then treat reducing that latency as the engineering problem. That latency has to be dealt with somehow, and there are two separate ways to do it:

1. Reduce the latency itself. E.g., ElevenLabs supports asynchronous tool calls; Vapi's are synchronous today.
2. Make latency matter less. Let agents break the fourth wall — telling the caller they need a moment — rather than leaving dead air. That reads as more natural and human, not as a compromise.

Where this is coming from: this isn't a new idea on our side — several Lazer FDEs, Sandy included, have argued for a while that it'd be better to accept latency in exchange for accuracy than the reverse. It hadn't gained traction, in part because it's a hard case to make without data behind it. What changed: Peter, an Avoca FDE who works across multiple Avoca clients (not just EAS), recently proposed increasing tool calls, and it was accepted internally at Avoca (Christian and others) as a good idea — which is what actually opened the door. Sandy has since talked to Peter directly — the tenor of that conversation is "let's experiment and see how aggressive we can be," rather than a fixed plan. (As one illustration of how far that could go: removing the post-call step entirely and booking directly in-call — a separate, much bigger topic on its own, raised here only to show the scale of the new appetite for experimentation, not as something proposed.) Credit for driving this belongs to Peter; it's an initiative several of us are glad to be applauding rather than one we're claiming independently. It's meant to be tested for real, not just assumed to help.

Axis 2 — Safer prompt edits, via regression testing

Make prompt changes with confidence they aren't whack-a-mole — fixing one failure while quietly creating another. The mechanism is testing, and the important part of it is regression testing: does this change break something that used to work?

Matt has been building this regression suite for a long time — that's his work, and he's made real strides on it. More recently, a test-initiative team — Sandy, Caleb, Matt, Eric, and Sonia — has formed around this area more broadly. One of the team's main focuses is exactly this regression-testing foundation Matt built; where it goes next is still to be worked out together.

The catch: because the system is non-deterministic and the number of ways it can fail is large, a thorough regression suite gets big and slow — which is exactly wrong for problems that need a fast fix. Full-suite-every-time isn't the right model. What's needed is a triage-friendly approach.

What makes it triage-friendly: mocking

The immediate, fast, triage-friendly testing piece is a new initiative Sandy is proposing and demoing this week — separate from, and complementary to, Matt's longer-running regression-suite work above.

Better mocking means fewer tests need to run, and each one carries more signal. Three surfaces need it: call context, tool calls, and — later, not the current focus — the post-call workflow.

• Tool-call mocking is already solved, and solved well. It's handled entirely on our side; Hamming (the test runner) doesn't need to know it's happening.
• The gap: that mocking only exists inside Hamming's pipeline — not at the point of triage. Triage is: someone reports a problem, and I need to find out fast if it's real. Today that means placing an actual phone call — 2-3 minutes, versus Hamming's ~15 (maybe 5, with significant engineering investment). The phone call wins on speed but has no mocking, so it isn't controlled or reliably reproducible.

What's still needed:

1. Tool-call and call-context mocking available on real phone calls, not only Hamming-orchestrated ones.
2. A way to test prompt changes without touching production — a test phone number pointable at a different assistant config.

What's already shipped: (2) is done — the test-number-config-override work (shipped 2026-07-30) wires a standing override so the test phone number can point at a different assistant config independent of what's live in production. What's still open is (1): mocking on real, human-placed calls.

What comes next

This brief covers reliability specifically. A separate, ongoing test-plan brief (link to be added once that doc lands) covers the rest. Categories that matter but aren't addressed here:

• Onboarding / migration testing — general quality (not just outcome accuracy) for clients not yet live or migrating off a pre-Blueprint setup; handled via Simulation Testing v2.
• Config auditing — grading an assistant's setup against its stated expectations.
• Post-call correctness — did the booking/reschedule/cancel actually land in the CRM (currently a known gap).
• Stateful mocking — mocks today are static; there's no way to verify a mutation actually took effect (currently a known gap).
• Performance testing — proving whether more tool calls, shorter prompts, etc. are actually faster or slower, and tracking agent performance over time. Not currently being worked.
• Blue-sky / R&D testing — bigger, more exploratory questions, e.g. model-agnosticism (does it matter if it's Claude, Gemini, or GPT underneath). Not currently being worked.

Lazer's impact

Caleb, Matt, and Sandy each independently noticed this problem and had been working on pieces of a solution. Getting receptiveness within Avoca was hard, because the FDE role itself pushes the opposite way: move fast, focus on your one client, fix their problems, move on. That's good for velocity per account — and bad for a cross-cutting problem like this one, which stayed stuck in "if you see it, fix it" mode, solved in silos rather than together.

When Sandy moved to the product team, testing was the first thing he raised — he thought it was the most important gap. What existed already: one testing initiative, a history of testing efforts that had failed or never got wide adoption, and the newest attempt — Simulation V2 — being built by a summer intern.

Since then, Sandy's role has been uniting that scattered work into one real initiative inside Avoca, rather than leaving it as separate threads different Lazer people were each pulling on alone. That effort now has a dedicated Linear project underway and an active Slack channel. As it grows, the expectation is more attention, budget, team, and mindshare within Avoca — earned by the work, not asked for in the abstract.

Appendix: examples

Two real calls that illustrate the problem concretely, rather than abstractly.

Incident 1 — booking confirmed before diagnostic questions were asked. These are dispatch businesses where different techs have different expertise, so the agent needs enough information about the job to route it to the right person. In this call: the agent greets the caller, the customer asks for an appointment on a specific date and time, the agent confirms the slot, the customer agrees, and the agent moves straight to confirming name and address. It never asked what the job actually was — the diagnostic questions that feed the CRM and determine which tech gets dispatched.

The prompt has plenty of logic instructing it to ask what the job is for, with conditional branches by job type. It also has a directive not to ask too many questions once the customer has said yes, so as not to blow the close (a customer getting peppered with questions after agreeing can get irritated and hang up). Both directives are reasonable on their own. They collide here, and because the prompt already carries a lot of other logic, there wasn't a clean way to scope "don't over-ask" so it doesn't swallow "get enough information to route correctly." The fix is possible — but it means growing the prompt further, which is exactly the dynamic this brief is about.

Incident 2 — two jobs, booked as one appointment. A customer calls about a water heater that isn't working, and wants it repaired rather than replaced; the agent confirms and books a tech. The customer then mentions a separate issue with an AC panel. The agent books one appointment covering both.

Two directives are in tension: (a) two different jobs should always get two different appointments, never one booking, and (b) if the issue spans an AC unit and a heating unit, one tech can handle both — because heat pumps do both heating and cooling, so a single combined system shouldn't get two techs sent to it. The agent pattern-matched "heater + AC mentioned in the same call" onto the heat-pump exception and booked one tech, but these were two different physical units in two different locations, needing two different specialists — a boiler tech isn't an AC-electrical tech. The general rule and its exception both need to exist; scoping the exception correctly gets harder the more other logic is already competing for space in the same prompt.

A related pattern: compiling logic into the prompt to avoid a tool call

The two incidents above are single-call failures. This is a standing design choice that trades reliability for speed on purpose, and is worth naming as its own category.

Transfer-window rules, compiled via Liquid. The Blueprint prompt uses Liquid templating to take database-driven rules — transfer this number at this time, a different number at another time, never on holidays — and bake them directly into the instruction set the agent reasons over. The alternative is a tool call: send the relevant details in a request, get back a yes/no and a reason. That's clean and reliable, but costs a round-trip. Compiling the rules into the prompt is fast if the agent never has to stop and ask — but it grows the prompt, and it sometimes gets the rule wrong, because now an LLM is interpreting the rule instead of a deterministic function evaluating it.

Availability: return everything vs. return just enough. The more accurate approach is to look up availability, return a few concrete options, and look for more only if asked. What often happens instead is a single search that returns a full 30-day window of raw availability into context — no round-trip needed no matter what the customer asks next, but now the agent has to make real judgment calls about what to do with all of that raw data, rather than being handed a minimal, exact answer it can just relay.

Both examples are the same shape as the two incidents: a deliberate choice to trade a tool-call round-trip for prompt-embedded logic, in exchange for speed, at some cost to reliability.

Appendix: Recent PRs

• Call context overrides — e.g. the date and time of the phone call (asOf). Already works within a Hamming test today.
• Test phone blueprint selection — the test phone number can point at a different blueprint than the live one, so it can be edited and called without touching production.