Appearance
Common Webhook — Architecture
How the unified webhook system works end-to-end. This is the system that CRM migrations are migrating to. The legacy system they migrate from is covered in legacy-comparison.md; the step-by-step recipe for executing a migration is in migration-playbook.md.
How to read this
The page is structured as four layers, top-down: (1) the HTTP entry point, (2) the durable queue, (3) the triager lifecycle, (4) the per-CRM dispatch. Each layer has a "what it does" + "where it lives in code" + diagram. Skim the diagrams first if you want a quick mental model; the prose drills into mechanics.
Unsolicited Opinion
There are four layers, and the failure modes look different at each. Most "the call didn't get booked" reports trace to layer 4 (CRM dispatch) — a wrong outcome type, an unsupported branch, a swallowed leaf error. Most "we never even tried" reports trace to layer 2 (the queue) — a row stuck in SCHEDULED, a cron miss. Knowing which layer the symptom belongs to is half the debug.
Why this exists: N channels × M CRMs
The migration isn't primarily about cleaning up the inbound-responder dispatch. The deeper motivation is combinatorial: Avoca has multiple channels (inbound responder, outbound, text, speed-to-lead / S2L) and multiple CRMs (ServiceTitan, HouseCall Pro, Field Routes, Job Nimbus, etc.). Without a shared dispatch surface, every channel needs its own per-CRM integration — N × M permutations of routing code, transcript extraction, customer resolution, post-call orchestration.
The common webhook collapses that into a single dispatch surface: each channel converts its post-call event into the same canonical PostCallReport shape, then funnels through the same triager → CRM-package pipeline. When a new channel ships (S2L, a new outbound campaign type), it integrates with one surface and inherits every CRM. When a new CRM gets migrated onto the common webhook, it works for every channel.
Unsolicited Opinion
The clean framing for "running up the flagpole" is: agents speak a common language, CRMs speak adapter language. The triager is the translator. A new CRM connects to the common language, not to each agent. A new channel converts its event into the common language, not into each CRM. The migration delivers that translation layer; once it exists, the marginal cost of (new channel, new CRM) drops from O(N×M) to O(N+M).
Speed-to-lead (S2L) is named but not yet integrated
S2L is the explicit second motivation cited during the HCP migration discussion. As of 2026-05-12, the S2L team is "going to make some configuration and have speed-to-lead outbound other campaigns as well ready for the CRM" once the common webhook is in place. The integration point exists; the consumer hasn't fully landed.
The four layers at a glance
| Layer | What it owns | Code location |
|---|---|---|
| 1. HTTP entry | Route handler, team verification, test-call skip, message-type dispatch | pages/api/responder/common/workflow.ts |
| 2. Durable queue | responder_workflow_runs row lifecycle, Inngest event scheduling, cron sweep | lib/workflow/post-call/end-of-call-report.ts + EndOfCallReportInngestFunction.ts |
| 3. Triager lifecycle | prepare / runOutcome / finalize phases, retry contract, outcome row state | lib/workflow/stages/run/WorkflowRunTriager.ts |
| 4. CRM dispatch | Per-vertical executeOutcome, CRM branches, booking-<crm>/ packages | lib/workflow/stages/run/run-triage-*.ts + lib/workflow/stages/booking-*/ |
Layer 1: The unified HTTP entry
What it does
pages/api/responder/common/workflow.ts is the single inbound route for all common-webhook teams. Vapi POSTs every event for those teams here. The handler:
- Resolves teamId from the URL query string AND cross-checks it against the team that owns the Vapi assistant on the message (
verifyVapiTeam). Guards against a stale or hand-editedserver.urlrouting data into the wrong tenant. - Short-circuits test calls by checking
isTestPhoneNumber(callerPhone). Returns 200 immediately. - Dispatches by
message.type:
| Message type | Handler |
|---|---|
assistant-request | handleAssistantRequest(...) — returns the dynamic assistant config to Vapi |
end-of-call-report | fromVapiEndOfCallReport(...) (normalize) → handleEndOfCallReport(...) (enqueue) |
transfer-destination-request | handleTransferCall(...) |
status-update | handleStatusUpdate(...) |
hang | handleHang(...) |
The route is intentionally shallow — it does routing and queueing only. The actual work happens in Inngest workers downstream.
Name collision: common/webhook.ts is NOT this route
A sibling file pages/api/responder/common/webhook.ts exists in the same directory. It is a legacy ServiceTitan single-shot route — synchronous LLM extraction, inline ST job creation, no queue. Migrations target workflow.ts, never webhook.ts. Tracked as an obs in future-brief.md.
Where to look if it broke
- Vapi dashboard → webhook delivery status for this teamId (200/4xx/5xx)
- Datadog →
service:avoca-next-prod operation:responder-workflow @teamId:<N>for cross-tenant verification flips, test-phone skips, message-type stats verifyVapiTeamlog line if teamId in URL doesn't match the assistant's owning team
Layer 2: The durable queue
What it does
For end-of-call-report only, this layer turns Vapi's synchronous POST into a durably-queued background workflow. Full detail lives in post-call-dispatch-chain.md — that page maps the L0-L7 chain step-by-step. The summary for this section:
scheduleEndOfCallReportinserts a row in Supabaseresponder_workflow_runswithstatus: 'SCHEDULED'. This is the durable queue.- In
NODE_ENV === 'development': immediately flips the row toQUEUEDand callsinngest.send({ name: END_OF_CALL_REPORT_INNGEST_NAME, ... }). - In production/staging: does NOT call
inngest.senddirectly. The separate cron Inngest functionEndOfCallReportCronInngestFunctionpolls SCHEDULED rows on a schedule, marks them QUEUED, and dispatches the Inngest event.
When the event fires, EndOfCallReportInngestFunction consumes it. That function is structured as a sequence of Inngest step.run blocks so each phase is independently retryable:
| Step | What it does |
|---|---|
validate-workflow-run | Loads the responder_workflow_runs row, atomically claims it (idempotency) |
prepare-workflow | Resolves teamId if missing, calls prepareCallData, sets call's processing_status: 'processing', loads TypedWebhookConfig, instantiates the triager via WorkflowRunTriagerFactory.create, runs triager.prepare() to determine + persist outcomes |
execute-outcome-{i}-{type} per outcome row | Rebuilds the triager from serialized prepared data, calls triager.runOutcome(row) |
finalize-workflow | Rebuilds the triager, calls triager.finalize(...) — corrects is_bookable, marks rows completed |
This is the part that makes the common webhook fundamentally different from the legacy per-CRM webhooks: per-stage retry safety. A failure in one outcome row's execution doesn't roll back prepare; a failure in finalize doesn't roll back outcomes that already executed.
Where to look if it broke
- Rows piling up at
status: SCHEDULED: prod cron is paused or errored. Queryresponder_workflow_runs WHERE status = 'SCHEDULED'. - Row dispatched but never executes: check Inngest function logs for the specific event. Look for the
validate-workflow-runstep's "stale invocation" skip (race / duplicate dispatch). teamIdresolution error inprepare-workflow: payload missing botheventTeamIdand a Vapi assistant id throws here. Surfaces as a workflow-run-failure with telemetry attribution to L1's teamId.
Layer 3: The triager lifecycle
What it does
WorkflowRunTriager is the abstract base class every vertical-specific triager extends. It defines three phases, each invoked as its own Inngest step:
Phase 1: prepare()
Idempotent. Returns existing outcome_results rows on retry, or runs determineWorkflowOutcomes() (subclass-implemented) and persists new rows as pending on first run.
typescript
const existingRows = await getOutcomeResults({ callId });
if (existingRows.length > 0) return existingRows; // idempotent re-run
const outcomes = await this.determineWorkflowOutcomes(); // LLM classifier
validateOutcomes(outcomes);
const rows = await insertOutcomeResults({ callId, outcomes }); // status: pending
return rows;determineWorkflowOutcomes is where the LLM classifies the call into outcome types. For HOME_SERVICES, the seven possible types are: booking, message, rescheduling, cancellation, eta, job-notes, confirmed-appointment, update-customer-info. Other verticals have different vocabularies (AUTO_SERVICE has its own classifier).
Phase 2: runOutcome(row)
Called per row. Skips if completed/executed (returns cached workflow_result). Otherwise calls subclass executeOutcome(row). The return shape is a retry control signal:
| Return shape | Row outcome |
|---|---|
{results: [], errors: [...]} | Row stays pending. Inngest retry or manual retrigger re-attempts. |
{results: [{bookingResponse: {error: <truthy>}}], ...} | Row stays pending. Same as above. |
| Otherwise | Row marked executed, workflow_result payload persisted. |
This contract is implicit and easy to miss
The executeOutcome return-shape contract is enforced by WorkflowRunTriager.runOutcome via findBookingError and the result.results.length === 0 && result.errors.length > 0 check. Any per-CRM dispatch block must return a shape that fits this contract — return {results: [], errors} on exception, push the result into results on success.
Phase 3: finalize(rows, result)
After all rows have been runOutcome'd. Two things happen:
- Corrects
is_bookable: if no row's type is outsideNON_BOOKING_OUTCOME_TYPES(rescheduling,eta,cancellation,job-notes,update-customer-info,confirmed-appointment), the call is markedis_bookable: false. Otherwise it's a booking-or-message call. - Marks executed rows as completed.
State transitions on a single outcome row
Where to look if it broke
- Outcome row stuck
pending: leaf returned a failure shape. Inspect Datadog for the row'soutcomeIdin the executeOutcome step's logs. If genuinely transient, manual retrigger via the responder admin tool re-fires runOutcome. - Row marked
executedbut downstream consumers can't read it: payload serialization issue.extractSerializableWorkflowDataruns onresult.results— if a leaf returns a non-JSON-safe value (class instance, Map, etc.), the row gets executed butresult.workflow_resultis corrupted. is_bookabledoesn't match expectation: walk the rows for the call. If the only row isrescheduling/cancellation/eta/job-notes/update-customer-info/confirmed-appointment, finalize forcesis_bookable: falseregardless of what the booking-row-default-handler did.
Layer 4: The CRM dispatch
The factory selects a vertical triager. Each vertical triager's executeOutcome(row) is where per-CRM routing happens. There are four coexisting shapes for this routing inside HomeServicesWorkflowRunTriager alone, plus the simpler single-CRM verticals.
The vertical / CRM matrix
| Vertical | Triager class | Supported CRMs | Shape |
|---|---|---|---|
AUTO_SERVICE | AutoServiceWorkflowRunTriager | AUTO_OPS only | Thin: one branch, all work in booking-autoops/ |
WINDOW | WindowWorkflowRunTriager | DYNAMICS_365 only | Thin: one branch, all work in booking-dynamics365/ |
FLOORING | FlooringWorkflowRunTriager | OASIS only | Thin: one branch, all work in booking-oasis/ |
JUNK_REMOVAL | JunkRemovalWorkflowRunTriager | SALESFORCE only | Thin: one branch |
ROOFING | RoofingWorkflowRunTriager | SALESFORCE only | Thin: one branch (uses booking-salesforce-omnia/) |
| (default) | HomeServicesWorkflowRunTriager | SERVICE_TITAN (implicit default), ALBIWARE, FOUR_SEASONS, HOUSECALL_PRO (incoming PR #10249) | Fat: four coexisting patterns |
Unsolicited Opinion
The matrix is sparse — only HOME_SERVICES is multi-CRM. The single-CRM verticals are effectively redundant abstraction: a CRM that owns its own vertical could be implemented as another single-outcome branch inside HOME_SERVICES. The reason verticals exist is action vocabulary — AUTO_SERVICE's "schedule a service" has a different shape than HOME_SERVICES' "book / reschedule / cancel". Worth questioning whether the vocabulary actually differs enough to justify the split, or whether it's accidental complexity. Tracked in future-brief.md.
The four coexisting dispatch shapes in HOME_SERVICES
When migrating a CRM into HOME_SERVICES, you choose one of these shapes. All four are in production:
| Pattern | Example | When to use |
|---|---|---|
| Outer early-return + single outcome | ALBIWARE | CRM only supports one outcome type (no cancel, no reschedule). determineWorkflowOutcomes overridden to always return [{type:'booking'}]; executeOutcome early-returns. |
| Outer early-return + internal type switch | HOUSECALL_PRO (PR #10249) | CRM supports multiple outcomes but diverges from generic stage semantics. Outer if (crm === X) with own row.type switch and own no-op branches for unsupported types. This is the canonical pattern for migrating CRMs from lib/non-st-workflow/. |
| Per-case inline branch | FOUR_SEASONS (only for rescheduling) | CRM uses generic flow for most outcomes but diverges in one specific case. Inline if (crm === X) inside one case. |
| Implicit default → generic stage | SERVICE_TITAN (no explicit branch) | CRM uses generic action stages (booking/run-booking.ts et al.) which then have their own internal CRM-specific branches (e.g., booking/booking-triage.ts has ACCULYNX and FOUR_SEASONS branches). |
Predicting what runs for a given (CRM, outcome) requires knowing all four shapes
For a CRM you've never touched, the answer to "what code runs for this outcome?" requires checking all four layers: outer early-return, internal type switch, per-case inline branch, default → generic stage's own branches. There is no single grep that finds them all reliably. Tracked in future-brief.md.
The booking-<crm>/ package shapes
Each CRM-specific package under lib/workflow/stages/booking-<crm>/ has its own file inventory. The shapes vary substantially:
| Package | Files | Pattern | Leaves owned by |
|---|---|---|---|
booking-oasis/ | 6 (context, notifier, post-call-process, 3 actions) | Multi-outcome | lib/oasis/ (greenfield) |
booking-hcp/ (PR #10249) | 4 (context, 3 actions) | Multi-outcome | lib/non-st-workflow/hcp/utils/ (legacy, wrapped) |
booking-albiware/ | 3 (post-call, prompt, single-action) | Single-outcome | Inline + lib/albiware/ |
booking-autoops/ | 4 (tests, post-call, workflow, helper) | Single-workflow | lib/autoops/ |
booking-salesforce/ | 4 (info, prompt, workflow, email) | Single-workflow | lib/salesforce/ |
booking-dynamics365/ | 2 (workflow, post-call) | Thinnest single-workflow | lib/dynamics365/ |
booking-salesforce-omnia/ | (similar to salesforce) | Single-workflow | lib/salesforce-omnia/ |
Unsolicited Opinion
The structural distinction that matters for migrations: leaf-owning vs leaf-wrapping. Greenfield CRMs (built fresh under the common webhook, like oasis) own their leaves under lib/{crm}/. Migrating CRMs (built on legacy, moving to common, like HCP) wrap the existing legacy leaves under lib/non-st-workflow/{crm}/utils/. PR #10249's description claims oasis as the precedent, but every CRM on Sandy's backlog is the wrapping shape, not the owning shape. The migration playbook mirrors HCP, not oasis.
The seven outcome types
For HOME_SERVICES vertical, determineWorkflowOutcomes produces rows of these types:
| Type | Treated as | What runs in default (ServiceTitan) path |
|---|---|---|
booking | Booking | runBookingWorkflow |
message | Booking (info-only call) | runBookingWorkflow (handles non-service path internally) |
rescheduling | Non-booking | runReschedulingWorkflow (or FOUR_SEASONS inline path) |
cancellation | Non-booking | runCancellationWorkflow |
eta | Non-booking | runETAWorkflow |
job-notes | Non-booking | runJobNotesWorkflow |
confirmed-appointment | Non-booking | runConfirmedAppointmentWorkflow |
update-customer-info | Non-booking | runUpdateCustomerInfoWorkflow |
"Non-booking" means finalize will force is_bookable: false if no booking-typed row exists. The base class's NON_BOOKING_OUTCOME_TYPES set governs this — anything not in the set (including booking, message, or an unknown type) is treated as bookable.
The default case in executeOutcome is permissive
Any row.type value not matched by the explicit cases falls through to runBookingWorkflow. A typo or a new outcome type added upstream without updating downstream branches will silently book. This is true for both the generic stage AND for per-CRM internal type switches (HCP's PR has this in its default case). Tracked in future-brief.md.
Configuration surface
A team's behavior on the common webhook is driven by two columns in responder_webhook_configs:
| Column | Effect | Examples |
|---|---|---|
vertical | Selects which triager class the factory instantiates | HOME_SERVICES, AUTO_SERVICE, WINDOW, FLOORING, JUNK_REMOVAL, ROOFING |
crm | Selects which CRM branch within the triager | SERVICE_TITAN, HOUSECALL_PRO, ALBIWARE, FOUR_SEASONS, AUTO_OPS, OASIS, DYNAMICS_365, SALESFORCE, ACCULYNX |
Plus the assortment of feature flags that gate per-outcome behavior: rescheduling_enabled, cancellations_enabled, update_eta_calls_job_summary, job_notes_enabled, update_customer_info_enabled, confirm_appointment_enabled, create_customer_if_not_found, alwaysBookJob, alwaysBookEstimate, oncall_enabled, oncall_calendar_v2_enabled.
For the rollout side of this (per-team migration flips), see migration-playbook.md.
Canonical persistence
The triager's outcome rows aren't the only thing the pipeline writes. Two tables hold the canonical post-call state:
| Table | Purpose | Who writes it |
|---|---|---|
calls | The canonical call record per Vapi call. Includes is_bookable, processing_status, transcript, recording_url, duration, data_store (JSON binary, CRM-specific extras), call_reason, etc. | prepareCallData in L2 step 2 + finalize in L3 (corrects is_bookable) + each leaf workflow's post-call hook |
call_conversations | Per-Vapi-call-id conversation record. Holds processed call data, is_booked boolean, references the calls row. | addVapiCallConversation in L2 + update_call_conversation post-extraction |
Architectural mandate: the type of data persisted to these tables must be CRM-agnostic. Downstream features (dashboards, analytics, automated tasks, follow-up campaigns) read calls and call_conversations directly without knowing which CRM produced the row. A new feature should never need a CRM-specific code path to read the canonical state.
The escape hatch is the data_store JSONB column on calls. Anything CRM-specific that doesn't fit the canonical type goes there. This is where adapter-pattern decisions about "what fields belong everywhere vs. what belongs in data_store" get made.
Unsolicited Opinion
The "common language" mandate is enforced at this layer. If a new CRM's adapter wants to add a new top-level column to calls because the CRM has a concept that ST doesn't, the right answer is almost always "no, put it in data_store." The canonical type expanding to accommodate every CRM's idiosyncrasies is how the common-language idea decays. Tracked as a structural concern in future-brief.md (canonical result-shape contract).
What you get for free
Migrating a CRM onto the common webhook delivers these capabilities by construction, without any per-CRM code:
| Capability | Mechanism |
|---|---|
| Cross-tenant teamId verification | verifyVapiTeam in layer 1 |
| Test-phone short-circuit | isTestPhoneNumber in layer 1 |
| Provider-normalized post-call report | fromVapiEndOfCallReport → PostCallReport (decouples from Vapi-specific shape) |
| Durable retries per phase | responder_workflow_runs row + Inngest steps + cron sweep |
| Idempotent prepare | Existing rows returned on re-run |
| Per-row retry control | The runOutcome return-shape contract |
| Structured logging | Logger + formatLogData produce queryable Datadog logs |
| Status-update + hang handling | handleStatusUpdate + handleHang |
| Automated task generation | AutomatedTaskFactory on RESPONDER_POST_CALL trigger (separate from outcome execution) |
Legacy per-CRM webhooks bypass most of this — they run inline, no Inngest retries, no idempotency claim, sequential post-call calls without per-stage isolation, ad-hoc logging. The migration is largely "stop reimplementing all of the above per CRM."
Cross-references
- The full L0-L7 post-call dispatch chain trace:
post-call-dispatch-chain.md - The legacy per-CRM webhook system (what we migrate from):
legacy-comparison.md - The step-by-step migration recipe:
migration-playbook.md - PR #10249 walked through:
crms/housecall-pro.md - The agent architecture (separate from the webhook architecture):
agent-architecture-legacy-vs-current.md - Future shapes / improvements running up the flagpole:
future-brief.md