Appearance
Gotchas
Surprises FDEs hit during Avoca work that aren't intuitive from the architecture diagrams. Each section names one thing the system does that runs counter to a reasonable assumption, why it surprises, the right mental model, and what to do about it.
How to use this
Read top-to-bottom on first onboard. Skim by category once familiar. When you hit something genuinely surprising during a session that survives "wait, this is documented somewhere I missed," add an entry in the right category.
Categories
| Category | What goes here |
|---|---|
| Footguns | Easy ways to inadvertently cause harm — patterns that look reasonable but break in non-obvious ways |
| Naming Confusions | Overloaded terminology where the same word means different things in different layers |
| Environmental Quirks | Surprises in the local-vs-prod-vs-shared-state relationship, or in tooling / CI behavior |
| Architectural Debt | Places where two patterns coexist (legacy-vs-current) and you have to know which one applies |
| Process & Workflow | Day-to-day workflow traps: tooling rituals, client-communication norms, doc/code drift |
Footguns
Patterns where the obvious-looking action is the wrong one. The system doesn't stop you from making the mistake; it lets you ship and the consequences surface later.
F1. The "workflow" endpoint is a multiplexer, not a workflow
TL;DR
/api/responder/common/workflow receives many different event types mid-call. Only ONE of them (end-of-call-report) actually kicks off the post-call workflow pipeline. The others are high-frequency cheap-handling observability. Don't put expensive logic in the handler keyed on the high-frequency event types — one call generates dozens of them.
What surprises
The endpoint name is /workflow. A reasonable read: "this is where workflows fire — when Vapi sends an event here, a workflow runs."
Look at a real call's dev-server logs and you see something like:
POST /api/responder/common/workflow?teamId=2678 speech-update 200 in 13ms
POST /api/responder/common/workflow?teamId=2678 conversation-update 200 in 17ms
POST /api/responder/common/workflow?teamId=2678 speech-update 200 in 14ms
POST /api/responder/common/workflow?teamId=2678 conversation-update 200 in 22ms
... (continues for 30+ events during a 5-minute call)
POST /api/responder/common/workflow?teamId=2678 end-of-call-report 200 in 89msReasonable conclusion from the name: each of these triggers a workflow run. Wrong. The endpoint is a multiplexer. Most of those events return in ~15ms with result: undefined because the handler does almost nothing for that messageType. Only end-of-call-report schedules the Inngest function (EndOfCallReportInngestFunction) that runs the actual post-call workflow.
The actual routing
Reading the diagram:
- Three of the four paths go to the same URL (
/api/responder/common/workflow) and dispatch internally bymessageType. That's the multiplexer. - The fourth path (mid-call tool dispatch) goes to a completely different URL — each function tool carries its own
server.urlat priority 1 in Vapi's resolution stack (see vapi-squads.md). For Avoca, that's${NGROK_BASE_URL ?? 'app.avoca.ai'}/api/vapi/tools/dispatch, baked at agent-build time byFunctionToolviabuildUrl(). Tool dispatches do not pass through the multiplexer endpoint, and they do not pass through sub-agent or phone-level URLs either.
That's the trap: looking at workflow-endpoint logs alone, you see lots of activity and assume "I'm wired up." But tool dispatches are routing through a separate URL whose configuration may not match. Two practical consequences:
- For FDE local dev: redirecting only
RESPONDER_COMMON_WORKFLOW_URL(which controls priority 2) sends workflow events to your tunnel but leaves tool dispatches hitting prod, because per-tool URLs are priority 1. You also needNGROK_BASE_URLset sobuildUrl()emits per-tool URLs pointing at your tunnel. See Local Dev Setup. - For platform-level reasoning about traffic shape: if you're estimating cost or capacity for the responder endpoint, do NOT include tool-dispatch traffic — that's a separate URL with its own scaling profile.
The endpoint is two different concerns under one URL:
| Branch | When it fires | What it does | Cost profile |
|---|---|---|---|
| Telemetry (most messageTypes) | Continuously during the call (one per turn or speech segment) | Logs / persists state / returns 200 fast | Cheap, sync, fire-and-forget |
Workflow trigger (end-of-call-report only) | Once at end of call | Schedules an Inngest function, which runs the post-call dispatch chain | Heavy, async via Inngest |
Why this is a footgun
The endpoint's high-frequency telemetry path is a deceptively easy place to add logic. "I want the agent to track if a caller mentioned X — I'll add a check on conversation-update." Naive failure modes:
- Cost-per-call explosions. A 5-minute call generates ~30+
conversation-updateevents. If your handler does an LLM call or external API hit, you've multiplied per-call cost by 30. - Call stalls. Every event waits for a 200 response. If the handler grows synchronous expensive work (DB query that takes 2s, LLM that takes 5s), Vapi waits, events back up, the call stalls or events get dropped.
- State-corruption races. Multiple in-flight events for the same call. Naively-written handlers that read/modify state without locking can let one event's writes overwrite another's.
- Misplaced architecture. "I want the agent to do X when caller mentions Y" — the right place is a tool call (declarative, the LLM decides when to invoke) or a sub-agent prompt change (declarative, the squad's hand-off rules). Adding it as a workflow-endpoint side-effect is the architectural equivalent of polling.
Right mental model
- Telemetry events are fire-and-forget. Read-only or trivial-write at most. If you need durable state, write it asynchronously (queue, don't block the response).
- Decisions about what the agent should do live in the agent's design — sub-agent prompt content, tool definitions, tool dispatch handlers. NOT in the multiplexer endpoint.
- The workflow endpoint is misnamed. Mentally rename it to
eventsfor the telemetry branch andpost-call-triggerfor the end-of-call branch. The single URL is a transitional artifact.
What to do
| Need | Right surface |
|---|---|
| Track real-time state during a call (e.g., "caller mentioned X") | A tool that the agent invokes when relevant; OR a sub-agent prompt that handles the case explicitly. NOT a conversation-update handler. |
| Persist transcript or status updates | Lightweight async write in the telemetry handler (existing pattern). Return 200 fast. |
| Run business logic after the call ends | The post-call Inngest workflow (existing pattern). Triggered once per call by end-of-call-report. |
| Add new mid-call event handling | Pause and ask: is this telemetry (cheap, fire-and-forget) or business logic (should be a tool call instead)? |
Cross-cutting recommendation (not just for AutoOps)
Avoca's voice-assistant platform should probably split this endpoint at the responder layer — not specific to any one client/CRM. Two distinct URLs with distinct cost contracts:
/api/responder/common/events— mid-call observability/api/responder/common/end-of-call— post-call workflow trigger
Phased migration would be: add the new URLs as aliases first, recommend new code use them, eventually migrate assistant configs to the new URLs and deprecate the old.
This is an Avoca-platform-wide architectural concern, tracked in unknowns.md. For now, document the trap so FDEs don't fall into it.
F2. transferCall silently disappears from the assistant config
TL;DR
The agent factory removes transferCall from a sub-agent's toolIds at call-start time if no transfer destination passes the time-window / holiday filter. The system prompt still contains "use transferCall" language. Result: the LLM says "let me transfer you" and then dead air, because the tool isn't actually attached. The prompt and the tool-availability check are two independent decisions that can disagree.
What surprises
The Blueprint UI shows a transferCall checkbox on a sub-agent, and you check it. In the compiled JSON preview (right panel), only the AutoOps tools appear in the output. transferCall is gone with no error or warning on the surface.
The same omission happens at runtime per-call: when Vapi POSTs assistant-request, the factory rebuilds the sub-agent toolIds list and drops transferCall if the destination filter returns an empty list for the current time.
The mechanism
Two pieces of code, in order:
Step 1: per-call availability check in agent-factory.ts (lines 456-477):
ts
const needsColdTransfer = config.multiAgentConfig.subAgents.some((agent) =>
agent.toolIds.includes(ToolName.TRANSFER_CALL)
);
let coldTransferUnavailable = false;
if (needsColdTransfer && teamId) {
const destinations = await getFilteredTransferDestinations({
teamId,
voiceAssistantId: metadata?.voiceAssistantId ?? null,
});
coldTransferUnavailable = !destinations || destinations.length === 0;
// ... logs a warn and continues
}Step 2: tool omission (lines 482-509):
ts
const tools = agentConfig.toolIds.filter((toolId) => {
if (toolId === ToolName.TRANSFER_CALL && coldTransferUnavailable) {
AgentFactory.logger.warn(
`Cold transfer tool unavailable — omitting from agent "${agentConfig.name}"`,
...
);
return false;
}
return true;
});Step 3: what getFilteredTransferDestinations actually filters in transfer-destinations.ts (lines 372-418):
ts
// 1. team_id + active=true
// 2. filterByVoiceAssistantAccess (assistant assignment)
// 3. filterDestinationsByTimeWindow (per-destination time window)
// 4. filterDestinationsByHolidayThe time-window filter is the most common reason destinations vanish. A destination with "Time-based routing" toggled ON in the admin UI only counts as "available" inside its configured time window. Outside that window (or holiday), it disappears from the result and transferCall gets dropped.
Why this is a footgun
The destination filter and the system prompt's business-hours logic are two independent decisions about the same question: "Can the agent transfer right now?"
- The prompt has
{{ "now" | date: ..., "America/Denver" }}Liquid that injects the current time, plus a hardcoded business-hours list, plus rules like "If outside business hours, do not use transferCall." - The destination filter has its own per-destination time window, possibly in a different timezone, evaluated against current time on the server.
These two layers can disagree. Concrete failure cases:
- Destination time window narrower than prompt's stated hours. Prompt says "9 AM to 6 PM, you may transfer." Destination is configured for 10 AM to 5 PM. At 9:30 AM, prompt says "transferring" but tool was filtered out. Dead air.
- Timezone mismatch. Prompt's Liquid uses
"America/Denver". Destination's time window evaluated in a different team timezone. Edges of the day misalign. - Blueprint preview is misleading. The preview compiles using the current time of the viewer. Open it at 8pm, see no
transferCall. Open it at 10am tomorrow, see it. The preview is one-shot, doesn't reflect what real calls at any hour would see. - Holiday filter. Same pattern with holidays.
When prompt and tool disagree, the LLM follows the prompt (says "let me transfer you") and the tool isn't there to actually do it. The call hangs with no audio, the customer says "are you still there?", call abandons.
Right mental model
There should be one source of truth for "can the agent transfer right now," and it should resolve at the moment the agent actually tries. Today there are two sources resolving at different points:
| Layer | When it resolves | Authority |
|---|---|---|
| Prompt's Liquid + business hours | When prompt is rendered at call start | What the LLM thinks |
| Destination filter | When agent factory builds tools at call start | What the tool actually does |
Both resolve at call start, which is good. The bug is that they're two separate decisions that need to be kept in sync manually.
What to do
For testing right now: if transferCall is missing from the preview but you have a Front Office destination configured, the most likely cause is time-based routing. Toggle "Time-based routing" OFF on the destination, save, and re-check the Blueprint preview. The destination will then be always-eligible.
For production: keep time-based routing ON, but ensure each destination's time window is at least as wide as (or matches) the team's stated business hours from the system prompt. Otherwise edge-of-day calls will hit the dead-air bug.
For the platform team:
- Move the availability decision into the
transferCalltool handler itself. Always includetransferCallintoolIdsat compile time. Let the handler return a structured "unavailable, please take a message" result at call time. The agent reads the tool result and gracefully pivots. One source of truth, no possible disagreement. - Until that lands, add a prompt-level fallback: "If you say you will transfer and the tool returns unavailable or is not invokable, immediately apologize and take a message instead." Doesn't fix the root cause but stops the dead air.
- Surface the omission in the admin UI / Blueprint preview. A check mark next to
transferCallshould not silently produce a config withtransferCallremoved. At minimum show a warning.
Related
- Meeting with EAS team 2026-05-14 surfaced this symptom: Kareem at 06:20 ("the agent tried to transfer... it cut out") and Katie at 09:30 (Hannah's call had "are you still there"). Both calls were during the first round of testing before transfer destinations were configured. The current state (destination IS configured but time-window filter still drops it for off-hour previews) is the residual form of the same bug.
- Bug surfaced via direct code inspection:
agent-factory.ts:456-509andtransfer-destinations.ts:372-418.
Naming Confusions
Same term, different meanings depending on which layer you're in. Disambiguate by always saying "X at layer Y" rather than just "X."
(no entries yet — candidate fillers below)
Candidates to capture:
- "Workflow" is overloaded across at least three layers. The
/api/responder/common/workflowendpoint name; theresponder_workflow_runsSupabase table; the Inngest "workflow function" abstraction; themulti_agent_configfield that some folks colloquially call "the agent's workflow." Each refers to something different. Disambiguate explicitly. NGROK_BASE_URLis misnamed. Reads like "set this to ngrok specifically." It actually accepts any tunnel hostname (Cloudflare, ngrok, Tailscale Funnel, a deployed staging URL). The historical name predates Cloudflare-tunnel adoption. Mental rename:FDE_TUNNEL_BASE_URL.server.urlat three priority levels. Vapi hasserver.urlat the per-tool, sub-agent, and phone-number levels. Same field name, three different scopes. When someone says "the server URL," ask "at which priority level?" — see vapi-squads.md.
F3. Voice-assistant admin page does NOT auto-save
TL;DR
The voice-assistant admin page lets you configure inbound number, test number, provider-agnostic routing, and other settings in sequence — but it does NOT auto-save. Navigate away or refresh before clicking Save and every change is lost.
What surprises
The page presents multiple substantive admin actions (provision phone, provision test phone, set up provider-agnostic routing, etc.) as if each one is its own committed action. Most of them ARE committed individually (they hit server actions that write to the DB). But the page-level form state (e.g., toggles, dropdowns, name fields) is local state until you click Save. If you do a bunch of in-form edits, then navigate away to do something else, the in-form edits are gone.
The mechanism
Mixed action pattern: some buttons run server actions immediately (Provision Phone, Provision Test Phone, Set Up Provider-Agnostic Routing — these write to the DB the moment you click), while form-level state (toggles, selects on the page itself) only persists when the Save button is clicked.
The user-visible problem: you can't tell from the UI which actions are immediate-commit vs which are pending-save. Treat the page defensively.
How to avoid it
After every meaningful change in the admin UI, hit Save before moving on. Don't batch.
Discovered: 2026-05-20 during EAS replica setup.
Surprises in the local-vs-prod-vs-shared-state relationship, or in tooling / CI behavior. The thing that broke isn't your code; it's the environment.
E1. Supabase migrations don't auto-run
TL;DR
Merging a *.sql file under apps/web/supabase/migrations/ does NOT apply it to prod. Migrations are applied manually by the PR author (or someone with prod DB access). If your work depends on a migration being live, confirm it was applied before assuming. PR #10222 is in flight to add a GitHub Actions migration runner.
No CI workflow in .github/workflows/ references Supabase or migration commands. The six workflows (hamming-checks, lint-biome, lint, stale-prs, test, type-check-tsgo-web) cover lint, test, type-check, and Hamming voice testing only. No npm script applies migrations either; the closest one is gen-supabase-types, which regenerates TypeScript types from the prod schema and is unrelated to applying migrations.
Jackson's housekeeping PR #10216 ("Remove Stale Migration Files") acknowledges the gap directly in its description: "stale, (in theory already-run or never actually run) migration files." That parenthetical is engineering admitting nobody currently tracks which migrations have hit prod. The 3-PR migrations series (PR #10216 / #10222 / a third pending) is the fix in flight.
Real example: the AutoOps mirror-sync RPC migration (20260426205400_add_autoops_job_services_replace_rpc.sql) was merged in PR #9450 on 2026-04-26 but had to be applied manually by Kareem when downstream work depended on the RPC being live.
Candidates to capture:
- Localhost admin UI mutates prod-shared services. Clicking "Sync test phone" in localhost writes to prod Vapi cloud + prod Supabase. Surprising for FDEs assuming localhost ≈ sandbox.
- Vapi sub-agent serverUrl priority shadows phone-level URL. Already covered in vapi-squads.md. Worth a punchline copy here for surface area, since this is the single most common reason an FDE thinks they're testing locally when they're not.
- Stale Vapi config in admin UI vs Vapi UI. Admin UI writes flow Avoca → Vapi, but Vapi UI edits don't propagate back. Easy to assume both surfaces are equal.
- Force-push to main silently skips lint CI. Was a real bug in
lint.ymlandlint-biome.yml; fixed in PR #10119. Worth a short note here so the lesson survives.
Architectural Debt
Places where two patterns coexist (legacy-vs-current) and which one applies depends on the specific client / surface you're touching. Reading the codebase without this context leads to wrong mental models.
(no entries yet — candidate fillers below)
Candidates to capture:
- Test calls don't run post-call by default.
postCall=falsebaked into test-phone URLs — surprising for FDEs trying to validate end-to-end. (Now env-toggleable viaTEST_PHONE_POST_CALLafter Plan P.) - Squad sub-agent hand-offs aren't editable in Vapi UI for blueprint-managed assistants. They live in
assistant_configs.multi_agent_configJSON, edited via Avoca admin / blueprint editor. Vapi UI is read-only relative to Avoca's data for these clients. Already covered structurally in agent-architecture-legacy-vs-current.md; worth a footgun-style summary here. - Some clients are still on legacy Vapi-resident assistants. Don't assume EAS-pattern (blueprint + multi-agent + dynamic resolution) generalizes — check
assistant_modeon theassistant_configsrow first.
Process & Workflow
Day-to-day workflow traps. Not in the architecture, but they cost you time on day one if no one tells you. See also Lazer FDE Onboarding for the broader ramp context.
P1. SSH clone requires a passphrased key
TL;DR
A blank-passphrase SSH key won't clone AvocaAI/avoca-next. The clone fails or hangs. Set a passphrase on the key you've registered with GitHub for Avoca work.
If you already have an SSH key with a passphrase registered with your Avoca-email GitHub account, you're fine. If your existing key has no passphrase, generate a new one (ssh-keygen -t ed25519 -C "you@avoca.ai"), set a passphrase, register the public key in GitHub under the Avoca-email account, and add the private key to ssh-agent.
P2. is_internal flag toggles admin-only UI
Local dev: turn the flag on to see all admin buttons (test-call seeding, internal-only debugger surfaces, hidden-from-clients features). Turn it off, or revert the change, before committing. Several PRs have shipped with is_internal accidentally set, exposing internal UI in client-facing builds.
P3. Supabase enum changes require type regeneration
Adding or renaming a value in a Supabase enum doesn't propagate to TypeScript types automatically. CI's type check fails until you regenerate. Use the package.json script (search for supabase in scripts), commit the regenerated types alongside the migration. Forgetting this means a red CI on every PR until you push the type changes.
P4. Resolve all open PR comments before merging
PR comments block merges by policy. A reviewer's nit-pick that you "meant to address later" will block you. Resolve or reply to each one before clicking merge.
P5. Clients don't have visibility into internal tooling
Clients see only the Avoca dashboard (app.avoca.ai / dashboard.avoca.ai). They do not have access to Asana, Vapi, Datadog, Vercel, Notion, or Supabase. Don't reference these tools in client conversations. Avoca treats them as internal implementation. When debugging with a client, describe behavior in terms of what they can see (the dashboard, the call transcript, the booking outcome) and don't expose the underlying stack.
Adjacent: never mention Vapi to clients by name. Most clients aren't aware the voice agent runs on a third-party platform, and Avoca keeps that as internal knowledge.
P6. Architecture docs can drift from code
Avoca's internal docs, the captured docs on this site, and Notion pages are all maintained by humans; the code is the only enforced source of truth. Before quoting a doc to a customer ("yes, our system always does X"), verify the behavior in code. Field-level promises like call_reason and forwarded_phone_number have historically been documented but not reliably populated. Same goes for non-ServiceTitan CRM integration modes.
When you find drift, fix the doc (or the code, if the doc was right) and note the date.
P7. No automated regression testing for prompt behavior
There is no CI check that says "this prompt change won't break the booking flow." Hamming is being built to address this; in the meantime, manual test runs through Hamming or via real test calls are the regression path. Don't ship prompt changes assuming the type checker covers you, it doesn't reach the voice agent's behavior at all.
P8. "Frontend" and "Backend" mean different things in Avoca's vocabulary
| Term in Avoca | What it usually means |
|---|---|
| Frontend | The voice agent's prompt and configuration (hosted on Vapi). Not the web UI. |
| Backend | The post-call code that books appointments and writes to CRMs. |
| Dashboard | The Avoca app at app.avoca.ai and the newer dashboard.avoca.ai. |
When someone says "the frontend is broken" they almost always mean the agent's behavior, not the React app. Clarify if ambiguous.
Adding new entries
When you hit a surprise during a session, ask:
- Footgun? Does the obvious-looking action lead to harm or wrong behavior?
- Naming? Is the same word being used for different concepts at different layers?
- Environmental? Is the surprise in the dev/prod/CI/tooling boundary, not in code semantics?
- Architectural? Is the system in a state of two-patterns-coexist, and you got the wrong one?
- Process? Is it a workflow ritual, client-communication norm, or doc/code-drift trap?
Pick the best-fit category and add an entry. If a surprise spans multiple categories, prefer the one that best describes how the FDE encounters it (not the deepest root cause).
Related pages
- Architecture: Legacy vs Current — explains why the responder endpoint exists in its current shape.
- Vapi Squads — the priority-shadowing gotcha is documented there in detail.
- In-Call Sequence — the runtime loop showing where each event class flows.
- Unknowns — open architectural concerns that haven't yet become gotcha entries.