LazerAvoca engagement

Blueprint CI/CD

What I built at Avoca, the problem it answers, and why it’s bigger than it sounds.

Sandy Corsillo

What I built

Testing, done right, can transform Avoca. We figured out the right way.

Pin one real turn from a real call. Judge it against a rubric. Run that judgment against every team’s config in the fleet — in under a second, for pennies. A working proof of concept today.

What Avoca is01

An AI assistant that answers the phone for home-service businesses.

Books the job into the CRM, handles reschedules and questions, knows when to escalate — for companies that live on inbound, where a missed call is a lost job. Now expanding into new verticals.

   ServiceTitan · HouseCall Pro · AutoOps · FieldEdge · Tekmetric …
                              ↑
                    Avoca  —  the orchestration layer
                              ↓
                       VAPI  ·  ElevenLabs

Avoca owns neither end of the call. The application is an IDE — where an agent gets built, configured, and shipped. At call time it decides what the agent is told and answers every question the agent asks.

What Avoca is02

The agent is a prompt.

Not a model, not a codebase — a document. Hundreds of handwritten sentences: rules of speech, booking logic, domain judgment, the client’s own knowledge base.

Don’t say the word “cool.”
If the caller asks to book immediately, skip the customer lookup —
  unless they’re an existing member.
Boiler and air conditioning can be one booking — but only if it’s one unit.
A confirmed safety hazard outranks whatever else is happening.

Representative, not verbatim — but this is the material.

At call time, all of it compiles into one string handed to the model:

44,197

characters — one agent, one call

What Avoca is03

Blueprint: one owned prompt, inherited by every agent.

Avoca built Blueprint to take full ownership of the prompt. Modules, knowledge base, global variables, overrides — assembled at call time. (Variables resolve per call, or silently don’t: a huge failure source.) Each layer inherits everything below it and can override any of it — broad control over every agent in the fleet.

The customer layer changes most often — and that’s correct; they’re the only ones who know their business. But the trade is real: every agent now stands on the same foundation — and the foundation changes.

The problem04

One prompt, a couple hundred accounts, no instruments.

A 44,000-character prompt is doing everything at once. Three problems compound as Avoca grows:

That’s a regression problem — model-fitting across a couple hundred live accounts — run today on intuition, spot-checks, and minutes-per-answer voice tests. Churn is already real. It has to be systematized.

The problem05

The fix was obvious: test every base change on everyone who inherits it.

And it wasn’t really thinkable.

Put numbers on it: a real regression sweep — call it 200 cases across 200 accounts, 40,000 calls — at 50–80 seconds each, ten at a time, is roughly three days of wall-clock. Per base change.

Sampling is what you settle for when the real thing is unaffordable.

The insight06

A call is a set of prompts to a stateless model.

Every turn is the same operation: one document goes to the model, one reply comes back. Nothing persists in between.

So any issue surfaces as a response that doesn’t match what you expected — at a specific prompt. Which raises the question: do we have everything needed to recreate that exact point, and test it alone, against changes to the blueprint?

Turns out we do. That’s what I built.

A rough estimate, worth confirming with real numbers: a whole call averages on the order of one tool call. Nearly everything the agent does is decided in the prompt.

The system07

Pin the turn. Replay it. Judge it.

1. 01Pin a turn from a real call — the exact document the model saw at that moment.
2. 02Replay it against any config version, at temperature zero.
3. 03Judge the reply with an LLM-scored rubric — which names the failing clause.
4. 04~1 second, cents. Cheap enough to run on every publish, for every team.

The system08

Per base change: three days, or minutes.

The same 40,000-case sweep, run both ways. This isn’t a testing optimisation — it’s 40,000 conversations becoming 40,000 model calls at the moments that matter.

Illustrative arithmetic — 40,000 = ~200 cases × ~200 accounts; chat at 65 s average, 10 concurrent. The shape holds at any scale you pick.

The system09

A failure becomes a permanent test.

Pinning a turn doesn’t reproduce it once — it creates a fixture: the document, the conversation so far, the recorded tool results, frozen at that moment. Everything else builds on it:

1. 01A/B the fix. Replay the same moment against the current config and the edited one — the diff is the answer.
2. 02The red names its clause. Not “something’s off” — this sentence failed.
3. 03Vary the caller’s words. Same moment, many phrasings — brittleness shows up as a row of reds.
4. 04After the fix, it stays. A regression baseline that only grows.

The system10

It reproduced every turn’s behaviour, most character-exactly.

A small spike, deliberately small: three real calls, both platforms, eleven turns between them, replayed at temperature zero.

Eleven turns is an encouraging signal, not a fidelity rate. Enough to build on; not enough to quote.

What it really is11

At level one, it’s a test.

Every config edit — a module, a KB field, a variable — gets checked against the turns that matter, before it ships.

1. mainThe published blueprint is locked. Not something you pick off a list.
2. branchConfigs are where you work. Edit freely, break nothing.
3. mergePublishing requires passing a suite. The gate is the point.

The loop goes from edit-deploy-dial-listen to seconds.

What it really is12

At level two, it’s leverage.

A defect found by one team becomes a fleet-wide answer — because you can afford to ask every team. No sampling.

1. 01A team hits a problem. Pin that turn as a test. Red.
2. 02They override their own config. Green for them — unblocked, but it's a patch.
3. 03Promote the test to the base config and run it across every inheriting team.
4. 04That run is the blast radius — a list, before anyone argues about priority.
5. 05Fix it once, in the base. Everyone inherits.
6. 06Override comes off. The test stays as a baseline.

What it really is13

At level three, it’s the missing half of the model.

ERP-class software — Workday, Epic, Guidewire — made implementation scale with preview environments and config regression testing. This is that, for voice agents. And it cashes the promise the product is sold on — an AI that learns from its calls and improves. Now that is literally true: any call that exposes an issue is the seed of an application-wide fix.

It turns fitting one base to a couple hundred accounts — the churn problem — from intuition into engineering. Not a testing optimisation: the difference between a system that can evolve and one that’s frozen.

The agent is a prompt.
Now the prompt has tests.

Where it stands14

Where it stands