Hamming Features (investigation log)

A live working doc capturing Hamming dashboard features as we encounter them. Hamming's docs are gated and don't index well in search, so this is our ground-truth record of what each surface does and how it relates to our testing infrastructure.

Working surface

Sandy is clicking through Hamming UI; this page captures findings as we go. Entries marked Captured are confirmed via screenshot or UI inspection. Entries marked Inferred are best guesses pending verification.

Guardrails (Assertions)

The grading layer in Hamming. Each guardrail describes a single pass/fail criterion that gets evaluated against a test run.

Creation dialog (captured 2026-05-20)

Fields:

• Name (required) — display name
• Importance (Low / Medium / High dropdown) — affects how failures are surfaced
• Description (optional)
• Category (e.g., General)
• Trigger (required, dropdown including "Custom...") — when the guardrail evaluates
• Custom Trigger (when trigger = Custom) — natural-language description of when to evaluate
• Evaluation Criteria (required) — natural-language "what should be checked"
• Pass Criteria (required) — natural-language "conditions for passing"
• Fail Criteria (required) — natural-language "conditions for failing"
• Enabled checkbox

The natural-language criteria mean guardrails are LLM-judged, not deterministic shape checks. Implication: pass/fail is a soft assessment by a grader LLM, not a hardcoded equality test.

Settings panel (captured 2026-05-20)

Toggles + dropdowns that control HOW the guardrail runs:

• Run on test calls (toggle)
• Run on live calls (toggle)
• Evaluation Mode (dropdown, e.g., Text Only — implies other modes exist like Voice or Multimodal)
• Wait for tool call data (toggle) — guardrail evaluation waits until tool call results are available before grading. Important for assertions like "agent invoked transferCall."
• Wait for webhook validation data (toggle) ⭐ — the key finding. When enabled, Hamming fires a webhook to an external endpoint mid-evaluation and waits for the response payload. The returned data feeds into the guardrail's pass/fail decision.
• Use knowledge extraction (toggle) — extracts structured facts from the call (purpose pending verification)
• Tags — for organization / filtering across the workspace

What this resolves

Earlier we believed Hamming guardrails could only evaluate transcript + tool-call sequence — i.e., no post-call state assertions. "Wait for webhook validation data" disproves that. The architecture is:

Hamming finishes the call run
  ↓ guardrail with "Wait for webhook validation data" enabled
Hamming POSTs to our endpoint (URL configured at guardrail or workspace level — to verify)
  ↓ our endpoint queries external state (e.g., mock-AutoOps DB)
  ↓ returns validation payload
Hamming uses payload in LLM-judged pass/fail
  ↓ result lands in the run record automatically, in-band with other guardrails

This collapses the planned "post-run SQL script" workaround for the mock-AutoOps plan into a native Hamming feature.

Open verification needed

• Where is the webhook URL configured? Per-guardrail, per-workspace, or per-agent? Need to find the input field.
• What's in the webhook payload? Hamming likely sends { callId, testCaseId, agentId, transcript, ... }. Our endpoint correlates by testCaseId and queries mock DB.
• What's the expected response shape? Probably JSON the guardrail's LLM can reason over. Maybe also pass/fail booleans?
• Latency tolerance — does Hamming have a timeout? If our endpoint takes 5s to query the mock, does that fail the run?

Agent Tools (inferred — pending UI inspection)

Likely a UI surface listing tools registered against the Hamming agent. Possible interpretations:

• Display-only: "here's what tools this agent has"
• Configurable: "add/remove tools the agent should expose"
• Mock-configurable: "configure mock responses for tools" (would parallel avoca-next's tool_call_mocks system)

avoca-next has a route at /api/hamming/agents/[agentId]/tools/route.ts that returns tool names per agent — suggests Hamming has a native concept of "tools per agent" that this API wraps.

To investigate:

• Is there a tool-mocking feature in this section that's an alternative to avoca-next's tool_call_mocks?
• If yes, does it work for our purposes (per-test-case scoping, response shape control)?
• Strategic implication: if Hamming has native tool mocking, the avoca-next tool_call_mocks system might be redundant for new work (though it stays for backwards compatibility).

Test Hooks (captured 2026-05-20)

Full webhook-driven test orchestration framework built into Hamming. Path: Settings → Test Hooks. Description on landing page: "Run HTTP calls during test execution (before/after run, before/after case) to setup data, validate state, or trigger cleanup."

Per-agent: webhook groups are assigned to a specific Hamming agent.

Webhook Groups (Hook Groups)

Top-level container, created via Create Hook Group button. A group has:

• Name (required)
• Description (optional)
• Environment (optional tag — production / staging / development — for filtering)
• Webhook Configurations — the actual hooks, organized by phase

Four lifecycle phases

Each Hook Group has slots for webhooks at four points in the test lifecycle:

BEFORE RUN  ──→  BEFORE CASE  ──→  📞 Call Executes  ──→  AFTER CASE  ──→  AFTER RUN
   (once)         (per case)        (the actual run)      (per case)        (once)

Phase	Use cases
Before Run	Initialize state once per run: truncate mock DB, seed shared fixtures, fetch auth tokens
Before Case	Per-test-case setup: seed scenario-specific data, snapshot starting state
After Case	Per-test-case validation: query mock DB for the booking, verify shape, log per-case metrics
After Run	Tear down: reset state, send aggregate report, archive recordings

Per-webhook configuration (captured 2026-05-20)

Each individual webhook within a group has:

• Name (e.g., "Notify Slack")
• Phase (dropdown: Before Run / Before Case / After Case / After Run)
• Description (optional)
• Response Key ⭐ — auto-generated from name if blank. Later hooks reference this hook's response via {{<key>.field}} syntax. Hooks chain. One hook's output feeds the next.
• Call Trigger toggle — when on, "this webhook initiates an outbound call before each test case." Inverts the usual flow (instead of Hamming dialing, our hook tells Hamming to make the call).

HTTP Configuration:

• Method (POST / GET / etc.)
• URL (the endpoint Hamming POSTs to)

Payload Template (JSON) with default:

{
  "testRunId": "{{testRunId}}",
  "testCaseRunId": "{{testCaseRunId}}",
  "phoneNumberToDial": "{{phoneNumberToDial}}"
}

Available variables (insertable via UI buttons):

• testRunId — the run we're in
• testCaseRunId — the specific case run
• phase — current phase (before-run, before-case, etc.)
• agentId — the Hamming agent under test
• workspaceId — the Hamming workspace
• phoneNumberToDial — the test phone number
• scenarioFacts — per-scenario facts, accessed via {{scenarioFacts.fieldName}}

Variable syntax:

• {{variable}} — standard interpolation
• {{<key>.field}} — reference a previous hook's response, e.g., {{auth_token.token}}
• {{Get_ENV.winery}} — environment variable access
• {{scenarioFacts.fieldName}} — per-scenario data

Authentication:

• Auth Method (None / and other options)

Behavior:

• On Failure (Continue / Halt / etc.)
• Timeout (default 30000 ms = 30s)
• Enabled toggle

Why this is huge

This is essentially a pluggable test orchestration framework native to Hamming. It collapses substantial planned work for the mock-AutoOps + replica + variant-testing track. Concretely:

Previously planned	Replaced by
Post-run script that queries mock DB and computes pass rate	After Case hook → POSTs to mock's /validate endpoint with the booking data → Hamming uses response in guardrail (via "Wait for webhook validation data")
Seeding mock DB before each test case	Before Case hook → POSTs to mock's /seed?scenario=<id> with scenario facts
Resetting state between runs	Before Run hook → POSTs to mock's /reset
Cleanup / archive after runs	After Run hook → POSTs to mock's /cleanup or to a logging endpoint
Cross-hook data flow (e.g., capture booking ID for later cleanup)	Response Key chaining — hook A captures bookingId, hook B references {{book_response.bookingId}}

Result: mock-AutoOps just needs to expose a few well-named REST endpoints (/seed, /reset, /validate, /cleanup). Hamming's Test Hooks orchestrate the lifecycle without us writing any orchestration code.

Open verification

• Where does the validation payload return into guardrails? Two pathways possible: (a) After Case hooks set a response key that a guardrail's "Wait for webhook validation data" reads, OR (b) hooks just trigger side effects and guardrails fire their own webhook. The former would be more powerful; the latter is simpler. Worth checking how the two surfaces interact.
• Concurrency: if 100 test cases run in parallel, do their Before Case hooks fire serially or in parallel? Matters for our mock DB if state is shared.
• Failure modes: what if our endpoint returns 5xx? "On Failure: Continue" implies we keep running but log the failure — but what does Hamming surface in the run record?

How this changes the testing architecture

The single biggest finding so far is "Wait for webhook validation data." It changes our mock-AutoOps integration plan in three ways:

1. Grading approach simplifies. No post-run script needed. Guardrails with webhook validation are native, in-band, evaluated alongside other guardrails.
2. Mock-AutoOps needs to expose a validation endpoint. Our mock at apps/dashboard/app/api/autoops-mock/... now also needs /api/autoops-mock/validate?callId=X&testCaseId=Y that Hamming POSTs to. The endpoint reads mock DB state (the booking written during the call) and returns a validation payload.
3. Test Hooks might enable seeding + reset too. If hooks can fire pre-run and POST to our mock to seed initial state, we collapse another piece of the planned workflow into native Hamming.

Updates to:

• Mock AutoOps API research — grading-strategy section needs to favor Option 1 (Hamming-native via webhook validation)
• Tool Call Mocks — note that Hamming has its own grading-side webhooks distinct from the in-call mock interception

How to use this doc

Each Hamming feature gets a ## Feature Name heading. Inside, distinguish:

• Captured sections: confirmed via screenshot or UI inspection
• Inferred sections: best guess pending verification
• To investigate sections: explicit open questions

When verifying an Inferred section, change it to Captured + update with screenshot reference + dated finding.

Related

• Tool Call Mocks, the avoca-next-side mocking system
• Mock AutoOps API research, the broader plan
• How-To: Test Transfer Decision, the worked example using guardrails today