Reply triage & follow-ups

The most expensive minutes in outbound

Today's build: a triage agent classifying your real replies in under 30 seconds, drafting responses in your voice, and waiting for your one-tap approval in Slack - calibrated against your own reply history before it goes live.

You spent two modules earning a reply rate above the ~3.5% average (vendor-reported, 2026). Then the reply lands at 4:47pm on a Thursday, the rep is in a call, and by Monday the prospect's interest has cooled to room temperature. Reply latency is where outbound pipelines quietly leak the revenue all the earlier work earned.

The fix is a triage agent: every reply gets classified within seconds, a response gets drafted in your voice, and a human approves the send from Slack. Production systems running this pattern report over 94% classification accuracy from a single structured Claude call, at under 30 seconds of latency. Interest never sits unanswered overnight again.

The taxonomy: five buckets, five playbooks

Classification only works if the categories are few and the action per category is unambiguous. The standard taxonomy:

• INTERESTED - wants to talk or asks a buying question. Action: draft a reply proposing times (or your booking link), notify the owner immediately, create/update the CRM deal.
• REFERRAL - "talk to my colleague X." Action: draft a thank-you, extract the referred name, queue the new contact through enrichment and into the CRM with the referrer noted.
• NOT_NOW - real interest, wrong timing ("Q4", "after our launch"). Action: draft a graceful close, set a CRM follow-up task dated from their words, tag for the nurture list.
• UNSUBSCRIBE - any flavor of "stop." Action: remove from ALL active campaigns across ALL channels immediately, suppress the address permanently, no reply drafted. This branch is compliance, not courtesy - it's the one fully automatic branch.
• OOO - auto-responder. Action: no human needed; if a return date is parseable, shift the sequence to resume after it.

The architecture: webhook to Slack in 30 seconds

The pipeline is four small pieces, each one already in your toolkit: your sending platform fires a webhook on every reply (Instantly, the course example, does - check your sender's webhook docs); a small FastAPI receiver catches it; one structured Claude call classifies and drafts; Slack presents it for approval.

# One headless call per reply - structured output, not chat.
# schema.json pins the shape so downstream code never parses prose.
#
# claude -p "$(cat triage-prompt.md) Reply: ..." \
#   --output-format json --json-schema "$(cat schema.json)"
# (the flag takes the schema text; read .structured_output)

SCHEMA = {
    "type": "object",
    "properties": {
        "category": {"enum": ["INTERESTED", "REFERRAL",
                              "NOT_NOW", "UNSUBSCRIBE", "OOO"]},
        "confidence": {"type": "number"},
        "reasoning": {"type": "string"},
        "draft_reply": {"type": "string"},
        "follow_up_date": {"type": ["string", "null"]},
        "referred_contact": {"type": ["string", "null"]},
    },
    "required": ["category", "confidence", "draft_reply"],
    "additionalProperties": False,
}
# Below a confidence threshold (start at 0.8), the agent
# doesn't guess - it routes to a human with category UNCLEAR.

1. Write triage-prompt.md as a skill: the five categories with 3 real example replies each (pull them from your inbox history), plus the response playbook per category, plus voice.md for draft tone. Reply-triage skills exist in the libraries (skillsmp.com) - start from one and swap in your taxonomy and playbooks rather than starting blank.
2. Have Claude build the FastAPI receiver: one POST endpoint that accepts the Instantly webhook, calls the classifier, and posts the result to Slack with Approve / Edit / Skip buttons. Run it on a small always-on host - this is the one piece that can't live on a laptop; a one-click PaaS (Railway, Render, Fly.io) deploys it in minutes.
3. Authenticate the webhook before anything else processes: verify the secret or signature your sender includes with each delivery (if it offers none, require a shared token in the webhook URL), reject any post that fails the check with a 401, and log every reject. This is where the receiver's auth lives - nothing unauthenticated reaches the classifier.
4. Wire the webhook in Instantly to your receiver's URL, then send yourself a test reply through a campaign to see the full loop fire.
5. On approve: the reply sends through the sending platform (never directly - same rule as always), the CRM updates, and UNSUBSCRIBE suppression is verified across every active campaign.

Calibrate before you trust

94%+ accuracy is the published bar for this pattern - but verify YOUR accuracy on YOUR replies before relying on it. The calibration run takes an hour and tells you exactly where the agent is weak.

1. Export 20-30 historical replies (any past campaign - the weirder the mix, the better the test).
2. Classify them yourself first - your labels are ground truth.
3. Run the agent over the same set and diff: "Compare my labels to yours; show every disagreement with your reasoning."
4. Fix the pattern, not the instance: most misses cluster (sarcastic not-interested read as interested; referral-plus-interest hybrids). Add the missed cases as examples in the skill and re-run.
5. Ship when you're at or above ~90% on your own data, with UNCLEAR catching most of the rest. Then re-audit a random 10 replies weekly for the first month.

For teams: routing, ownership, and the approval queue

Solo, every approval is yours and the loop is simple. With a team, the question becomes: whose Slack does this reply land in, and what happens when they're heads-down?

Do this now

Sources and further reading

• Instantly API v2 docs (webhooks)
• Claude Code docs - headless & structured output