Skip to content

Hamming Features (investigation log) ​

A live working doc capturing Hamming dashboard features as we encounter them. Hamming's docs are gated and don't index well in search, so this is our ground-truth record of what each surface does and how it relates to our testing infrastructure.

Working surface

Sandy is clicking through Hamming UI; this page captures findings as we go. Entries marked Captured are confirmed via screenshot or UI inspection. Entries marked Inferred are best guesses pending verification.

Guardrails (Assertions) ​

The grading layer in Hamming. Each guardrail describes a single pass/fail criterion that gets evaluated against a test run.

Creation dialog (captured 2026-05-20) ​

Fields:

  • Name (required) — display name
  • Importance (Low / Medium / High dropdown) — affects how failures are surfaced
  • Description (optional)
  • Category (e.g., General)
  • Trigger (required, dropdown including "Custom...") — when the guardrail evaluates
  • Custom Trigger (when trigger = Custom) — natural-language description of when to evaluate
  • Evaluation Criteria (required) — natural-language "what should be checked"
  • Pass Criteria (required) — natural-language "conditions for passing"
  • Fail Criteria (required) — natural-language "conditions for failing"
  • Enabled checkbox

The natural-language criteria mean guardrails are LLM-judged, not deterministic shape checks. Implication: pass/fail is a soft assessment by a grader LLM, not a hardcoded equality test.

Settings panel (captured 2026-05-20) ​

Toggles + dropdowns that control HOW the guardrail runs:

  • Run on test calls (toggle)
  • Run on live calls (toggle)
  • Evaluation Mode (dropdown, e.g., Text Only — implies other modes exist like Voice or Multimodal)
  • Wait for tool call data (toggle) — guardrail evaluation waits until tool call results are available before grading. Important for assertions like "agent invoked transferCall."
  • Wait for webhook validation data (toggle) ⭐ — the key finding. When enabled, Hamming fires a webhook to an external endpoint mid-evaluation and waits for the response payload. The returned data feeds into the guardrail's pass/fail decision.
  • Use knowledge extraction (toggle) — extracts structured facts from the call (purpose pending verification)
  • Tags — for organization / filtering across the workspace

What this resolves ​

Earlier we believed Hamming guardrails could only evaluate transcript + tool-call sequence — i.e., no post-call state assertions. "Wait for webhook validation data" disproves that. The architecture is:

Hamming finishes the call run
  ↓ guardrail with "Wait for webhook validation data" enabled
Hamming POSTs to our endpoint (URL configured at guardrail or workspace level — to verify)
  ↓ our endpoint queries external state (e.g., mock-AutoOps DB)
  ↓ returns validation payload
Hamming uses payload in LLM-judged pass/fail
  ↓ result lands in the run record automatically, in-band with other guardrails

This collapses the planned "post-run SQL script" workaround for the mock-AutoOps plan into a native Hamming feature.

Open verification needed ​

  • Where is the webhook URL configured? Per-guardrail, per-workspace, or per-agent? Need to find the input field.
  • What's in the webhook payload? Hamming likely sends { callId, testCaseId, agentId, transcript, ... }. Our endpoint correlates by testCaseId and queries mock DB.
  • What's the expected response shape? Probably JSON the guardrail's LLM can reason over. Maybe also pass/fail booleans?
  • Latency tolerance — does Hamming have a timeout? If our endpoint takes 5s to query the mock, does that fail the run?

Agent Tools (inferred — pending UI inspection) ​

Likely a UI surface listing tools registered against the Hamming agent. Possible interpretations:

  • Display-only: "here's what tools this agent has"
  • Configurable: "add/remove tools the agent should expose"
  • Mock-configurable: "configure mock responses for tools" (would parallel avoca-next's tool_call_mocks system)

avoca-next has a route at /api/hamming/agents/[agentId]/tools/route.ts that returns tool names per agent — suggests Hamming has a native concept of "tools per agent" that this API wraps.

To investigate:

  • Is there a tool-mocking feature in this section that's an alternative to avoca-next's tool_call_mocks?
  • If yes, does it work for our purposes (per-test-case scoping, response shape control)?
  • Strategic implication: if Hamming has native tool mocking, the avoca-next tool_call_mocks system might be redundant for new work (though it stays for backwards compatibility).

Test Hooks (captured 2026-05-20) ​

Full webhook-driven test orchestration framework built into Hamming. Path: Settings → Test Hooks. Description on landing page: "Run HTTP calls during test execution (before/after run, before/after case) to setup data, validate state, or trigger cleanup."

Per-agent: webhook groups are assigned to a specific Hamming agent.

Webhook Groups (Hook Groups) ​

Top-level container, created via Create Hook Group button. A group has:

  • Name (required)
  • Description (optional)
  • Environment (optional tag — production / staging / development — for filtering)
  • Webhook Configurations — the actual hooks, organized by phase

Four lifecycle phases ​

Each Hook Group has slots for webhooks at four points in the test lifecycle:

BEFORE RUN  ──→  BEFORE CASE  ──→  📞 Call Executes  ──→  AFTER CASE  ──→  AFTER RUN
   (once)         (per case)        (the actual run)      (per case)        (once)
PhaseUse cases
Before RunInitialize state once per run: truncate mock DB, seed shared fixtures, fetch auth tokens
Before CasePer-test-case setup: seed scenario-specific data, snapshot starting state
After CasePer-test-case validation: query mock DB for the booking, verify shape, log per-case metrics
After RunTear down: reset state, send aggregate report, archive recordings

Per-webhook configuration (captured 2026-05-20) ​

Each individual webhook within a group has:

  • Name (e.g., "Notify Slack")
  • Phase (dropdown: Before Run / Before Case / After Case / After Run)
  • Description (optional)
  • Response Key ⭐ — auto-generated from name if blank. Later hooks reference this hook's response via {{<key>.field}} syntax. Hooks chain. One hook's output feeds the next.
  • Call Trigger toggle — when on, "this webhook initiates an outbound call before each test case." Inverts the usual flow (instead of Hamming dialing, our hook tells Hamming to make the call).

HTTP Configuration:

  • Method (POST / GET / etc.)
  • URL (the endpoint Hamming POSTs to)

Payload Template (JSON) with default:

json
{
  "testRunId": "{{testRunId}}",
  "testCaseRunId": "{{testCaseRunId}}",
  "phoneNumberToDial": "{{phoneNumberToDial}}"
}

Available variables (insertable via UI buttons):

  • testRunId — the run we're in
  • testCaseRunId — the specific case run
  • phase — current phase (before-run, before-case, etc.)
  • agentId — the Hamming agent under test
  • workspaceId — the Hamming workspace
  • phoneNumberToDial — the test phone number
  • scenarioFacts — per-scenario facts, accessed via {{scenarioFacts.fieldName}}

Variable syntax:

  • {{variable}} — standard interpolation
  • {{<key>.field}} — reference a previous hook's response, e.g., {{auth_token.token}}
  • {{Get_ENV.winery}} — environment variable access
  • {{scenarioFacts.fieldName}} — per-scenario data

Authentication:

  • Auth Method (None / and other options)

Behavior:

  • On Failure (Continue / Halt / etc.)
  • Timeout (default 30000 ms = 30s)
  • Enabled toggle

Why this is huge ​

This is essentially a pluggable test orchestration framework native to Hamming. It collapses substantial planned work for the mock-AutoOps + replica + variant-testing track. Concretely:

Previously plannedReplaced by
Post-run script that queries mock DB and computes pass rateAfter Case hook → POSTs to mock's /validate endpoint with the booking data → Hamming uses response in guardrail (via "Wait for webhook validation data")
Seeding mock DB before each test caseBefore Case hook → POSTs to mock's /seed?scenario=<id> with scenario facts
Resetting state between runsBefore Run hook → POSTs to mock's /reset
Cleanup / archive after runsAfter Run hook → POSTs to mock's /cleanup or to a logging endpoint
Cross-hook data flow (e.g., capture booking ID for later cleanup)Response Key chaining — hook A captures bookingId, hook B references {{book_response.bookingId}}

Result: mock-AutoOps just needs to expose a few well-named REST endpoints (/seed, /reset, /validate, /cleanup). Hamming's Test Hooks orchestrate the lifecycle without us writing any orchestration code.

Open verification ​

  • Where does the validation payload return into guardrails? Two pathways possible: (a) After Case hooks set a response key that a guardrail's "Wait for webhook validation data" reads, OR (b) hooks just trigger side effects and guardrails fire their own webhook. The former would be more powerful; the latter is simpler. Worth checking how the two surfaces interact.
  • Concurrency: if 100 test cases run in parallel, do their Before Case hooks fire serially or in parallel? Matters for our mock DB if state is shared.
  • Failure modes: what if our endpoint returns 5xx? "On Failure: Continue" implies we keep running but log the failure — but what does Hamming surface in the run record?

How this changes the testing architecture ​

The single biggest finding so far is "Wait for webhook validation data." It changes our mock-AutoOps integration plan in three ways:

  1. Grading approach simplifies. No post-run script needed. Guardrails with webhook validation are native, in-band, evaluated alongside other guardrails.
  2. Mock-AutoOps needs to expose a validation endpoint. Our mock at apps/dashboard/app/api/autoops-mock/... now also needs /api/autoops-mock/validate?callId=X&testCaseId=Y that Hamming POSTs to. The endpoint reads mock DB state (the booking written during the call) and returns a validation payload.
  3. Test Hooks might enable seeding + reset too. If hooks can fire pre-run and POST to our mock to seed initial state, we collapse another piece of the planned workflow into native Hamming.

Updates to:

  • Mock AutoOps API research — grading-strategy section needs to favor Option 1 (Hamming-native via webhook validation)
  • Tool Call Mocks — note that Hamming has its own grading-side webhooks distinct from the in-call mock interception

How to use this doc ​

Each Hamming feature gets a ## Feature Name heading. Inside, distinguish:

  • Captured sections: confirmed via screenshot or UI inspection
  • Inferred sections: best guess pending verification
  • To investigate sections: explicit open questions

When verifying an Inferred section, change it to Captured + update with screenshot reference + dated finding.