Appearance
Just-in-time service context
Question: should a service's rules be compiled into every prompt, or fetched when the caller turns out to need that service?
We built the second option, ran both head-to-head on live calls, and found no improvement in call time or accuracy. What the exercise did produce is a clearer picture of where the real problem sits — and it may not be prompt size.
The current setup
Every service a business offers contributes its rules to one compiled prompt, and that prompt is delivered on every call regardless of what the call is about. A plumbing question carries the electrical rules; a furnace call carries the drain-cleaning rules.
The knowledge is split awkwardly:
Three consequences fall out of this shape:
Service data is thin; service behavior is prose. The structured record holds little more than a name and a description. What the agent actually does — which follow-up questions to ask, what special handling applies — lives as sentences inside prompt modules, interleaved with flow control and unrelated policy.
Depth is charged to every call. Because all of it is always present, making one service's guidance richer costs every conversation, including the ones about other services. That caps how good any single service's handling can get, and the cap tightens as a business adds services.
Rules about pairs of services have nowhere to live. Everything above stores facts about one service at a time. So a rule like "these two can share a single visit" can only be written as a sentence in a module shared by every call — which means it cannot be scoped to the two services it concerns.
That third one has already caused a production incident: a business needed one specific pair of maintenance services to be combinable, the rule went into the shared module, and the agent began offering to combine unrelated services.
What we built
Three additions, and one deliberate non-change.
- A real per-service record with room for the things that were previously prose — conditional follow-up questions, branching guidance, and customer-facing facts like duration, cost range, and how to prepare.
- Group tables so "these services can share one visit" is a row scoped to the services it concerns, instead of a sentence everyone reads.
- A tool the agent calls once the caller states a need. It classifies the need against that business's services, returns only the matching record, and — when several needs are stated — returns a verdict on whether they can share an appointment.
- The existing knowledge base is untouched. The new records are seeded from it, so nothing about the current editing surface or compile path changes.
Grouping stands on its own
The group tables need none of the rest. Two small tables and a check at booking time would fix the scoping problem behind the production incident — no tool, no rich records, no prompt changes.
One caveat worth checking first: when we modelled a real business's combination rule properly, its group table came out empty. Their clause was compensating for a classification gap, not describing a real combination — heating and cooling on one physical system is the same service, not two. If most teams' clauses are that shape, the fix is better classification, not a new relation.
What changed in the prompts
The compiled prompt stops describing services and starts pointing at the tool.
| Removed | Replaced with | |
|---|---|---|
| Service catalog | every service's description and handling | names only, plus "call the tool for anything richer" |
| Booking overrides | per-service handling instructions | one line: the tool's result tells you when a service is message-only |
| Intake steps | hardcoded per-service questions | "ask the follow-up questions from the tool's result" |
| Combination rule | the team-specific exception sentence | nothing — the tool returns a verdict |
Everything else — conversation flow, personality, fees, membership, FAQ, transfer handling — is untouched.
How we tested it
Two arms on the same voice assistant, switchable with a single toggle, identical in every respect except delivery:
- All-in-prompt: every service's full record compiled into the prompt. The tool is not registered on this arm at all, because tools attach by being mentioned in the prompt and this arm's prompt never mentions it.
- Just-in-time: no service content in the prompt; the tool fetches it mid-call.
Two controls mattered:
Same content on both sides. The all-in-prompt block was generated by the tool's own formatter from the same records, so the two arms carry equivalent content and differ only in when it arrives.
One source of questions per arm. Each arm's intake step points at exactly one place — the tool's result, or the in-prompt catalog. Getting this wrong the first time produced the most interesting failure of the exercise (below).
Calls were placed on both voice platforms, each arm running the same scenario.
Findings
No improvement in time or accuracy
| Pair | All-in-prompt | Just-in-time |
|---|---|---|
| Platform A | 161s | 164s |
| Platform B | 201s | 218s |
Every completed call in both arms booked correctly. Both arms performed identical expert triage — asked the equipment's brand, and on hearing a brand whose units have no digital display, correctly asked for the diagnostic LED blink count instead of a fault code.
Accuracy: no difference. Call time: no improvement, slightly worse.
The quality came from the content, not the delivery
The conversation got dramatically better during this exercise — but that was because we wrote a much richer service record, not because of how it was delivered. The all-in-prompt arm, holding the same record, ran the same expert conversation.
Delivery mechanism and content depth are orthogonal. The improvement was available without building anything.
What just-in-time does measurably buy
Prompt size stops scaling with the catalog:
| Services offered | All-in-prompt | Just-in-time |
|---|---|---|
| 4 | 51,159 | 47,755 |
| 15 | 62,173 | 47,755 |
| 30 | 77,005 | 47,755 |
| 60 | 107,560 | 47,755 |
| 79 | 127,032 | 47,755 |
The just-in-time figure moved by one byte across all five. The static side grows about a thousand characters per service. Across the fleet, a typical business offers around a dozen services, the busiest tenth offer sixty or more, and the largest offers several hundred — where the static approach has no viable version.
This is a scaling property, not a quality property.
Caveats — the sample is small and the setup was imperfect
- One call per arm on the final configuration. More calls were made, but the setup changed underneath them as problems were fixed.
- We tested where the design predicts no difference. The advantage is a function of catalog size, and the final comparison ran on a small catalog where the two prompts differ by roughly a tenth. A wash there is the expected result, not evidence against.
- Voice adds noise. Speech recognition mangled a brand name; caller phrasing varied between runs.
- No planted checks. Nothing was instrumented to detect whether instructions were being followed — success was judged by reading transcripts.
The prompt-structure problem this exposed
Independent of whether just-in-time is worth building, two things surfaced that are worth fixing either way.
Service logic is smeared across unrelated modules. For one business it appeared in the overrides block, the classification module, the not-provided list, the multiple-request policy, and two numbered intake steps. Finding it required compiling the prompt and marking spans in the rendered output, then tracing each back to its source — you cannot locate it reliably by reading the source modules.
Adding a tool does not remove the static instruction that answers the same question. Our sharpest failure: the agent announced "let me see what questions I need to ask" and then never called the tool. The cause wasn't the platform — the prompt still contained the old hardcoded intake questions, so the agent already knew what to ask and the tool was redundant. It then improvised plausible questions and booked the appointment successfully.
The failure was invisible. The call completed, the customer was satisfied, and nothing looked wrong unless you compared the questions asked against the record that should have been used.
A proper per-service data model removes the first problem by construction, and makes the second one detectable.
Where this actually points
Neither arm could tell when it had gone wrong. That, not prompt size, is what the calls kept demonstrating:
- The tool didn't fire, so the agent invented questions — and booked successfully.
- The classifier matched a second service that the caller never mentioned — and the agent booked two appointments.
- In production, an over-general combining rule led to wrongly combined services — and the calls completed.
In every case the conversation stayed locally coherent while being globally wrong, and nothing in the system noticed. Just-in-time doesn't help with this. It adds one more way to be silently wrong — the tool can simply be absent — which is a genuine mark against it. The static approach cannot fail that way.
So the more promising direction may be mid-call evaluation rather than context management. A goal-oriented evaluator running alongside the conversation, continuously assessing where the call stands relative to what it is supposed to achieve, and feeding that assessment back to the agent each turn — so it develops a "this is going sideways" instinct instead of confidently proceeding.
Three things make that attractive compared to what we tested:
- It introduces no new failure mode. If the evaluator is unavailable, behavior degrades to what happens today. If a just-in-time tool is unavailable, the agent improvises and conceals it.
- It targets the failure we actually observed — repeatedly, in both arms, and in production.
- It is orthogonal to everything above. It would improve outcomes whether service context is injected or compiled.
That is a different system and a different experiment: define the goal state, evaluate against it each turn, and measure whether agents recover from drift they currently sail straight through. It deserves designing as its own test rather than as a variation on this one.
Implementation details, schema, and the method used to extract service content from compiled prompts: implementation notes.