The first AI agent that scared us

We've built a lot of agents. Most are fine. Boring, useful, predictable.

One we built for internal use stopped us cold for a day and made us rethink some of our defaults. We're sharing the story because the lesson generalized into how we think about agent safety.

What the agent was supposed to do

We wanted an agent that monitored our shared team inbox (founder@avery.software) and:

→ Read every incoming message → Classified it (sales prospect, support question, partnership inquiry, press, recruiting, spam) → Routed to the appropriate person → Drafted a first-pass response in our brand voice → Posted the draft + classification to a private Slack channel for review

Standard inbox automation. Common pattern. Nothing exotic.

We built it in about 90 minutes using Avery NXR. Tested on sample emails. Worked well. We let it run on the real inbox.

What it actually did

Within 6 hours, the agent had drafted responses to 23 messages, classified them, and posted everything to the review channel.

We started reviewing. Mostly good — sales prospect drafts were on-brand, support questions were routed correctly, partnership inquiries were flagged appropriately.

Then we got to message 17.

The message was from someone we'd been talking to about a potential investment. The conversation was sensitive — we'd been deliberately slow-walking to negotiate position. The agent had drafted a response that was technically polite, helpful, and on-brand.

It was also completely wrong for the strategic context. If sent, it would have undermined our negotiating position with a specific signal we'd been carefully NOT giving.

Nobody on our team would have responded to that email with what the agent drafted. Not because the agent did anything obviously bad, but because the agent didn't know the strategic context.

What it taught us

The agent did its job correctly. It classified the message, drafted a response in our voice, didn't say anything wrong. By every metric we had defined, the agent succeeded.

The failure was at a layer the agent couldn't see: the strategic context that lives in human heads, not in past emails.

This made us realize something uncomfortable: as agents get better at the surface tasks, we're going to need to be much more careful about what they have authority to ACT on versus just DRAFT.

What we changed in the agent

After the scare:

→ The agent no longer sends responses directly. Even drafts that look "obviously fine" go to a human reviewer first. → Classification thresholds got tightened. Anything tagged "high-stakes" or "negotiation" or "press" requires explicit human approval, not just review. → The agent now flags messages where it can't determine the strategic context, and explicitly asks for human input rather than drafting. → Audit ledger entries include the agent's confidence level + what it inferred about strategic context.

The agent is more useful now, not less. It's doing more thinking about WHEN to defer, not just WHAT to draft.

The broader pattern we noticed

After this incident, we started looking at our other agents through a "what's the failure mode?" lens. We found:

The competitor monitoring agent could in theory generate alerts that triggered overreactions in our roadmap meetings if presented as more confident than they were. We added confidence levels and explicit "this is one signal, not a directive" framing.

The resume screening agent could in theory perpetuate hiring biases if the JD itself was biased. We added bias check prompts that flag potential issues for the hiring manager to consider.

The pipeline agent could in theory tell a salesperson a deal was "stalled" when actually it was just slow for valid reasons, causing inappropriate escalation. We added context flags so the agent could ask about deal-specific reasons before flagging.

The pattern: every agent has failure modes that aren't about the agent being WRONG, but about the agent being CONFIDENT in a context where confidence is misplaced.

Default settings we now apply to all agents

Based on this learning, our default agent settings include:

→ No auto-send for high-stakes actions. Agents draft. Humans send. → Confidence levels in outputs. Agents say "high / medium / low confidence" so humans calibrate their review. → Explicit escalation paths. Agents have a "defer to human" option, not just yes/no decisions. → Audit ledger entries that include context the agent USED. So when humans review, they can see what the agent knew (and didn't know). → Periodic sampling. Even fully-automated agents get sampled by humans periodically to catch drift.

These defaults aren't features in the marketing sense. They're protections in the safety sense.

Why this matters for the local-first model

This is a place where the local-first architecture matters in an under-appreciated way.

If our agent had been running on a cloud-LLM platform that auto-sends responses, the scary message could have been sent before we noticed. The recovery would have been a multi-day cleanup.

With local-first, the agent runs in our infrastructure with our review gates. We catch issues before they leave the company. The audit ledger gives us full traceability of what the agent saw and decided.

Cloud agent platforms can implement review gates too, but the architectural friction is higher. When the agent and the response infrastructure are owned by different parties, "stop the agent" is harder than "stop the local process."

What we'd tell other agent builders

If you're building agents that take action on your behalf — drafting responses, scheduling things, posting publicly, committing to anything — design the SAFETY layer with the same care you design the capability layer.

The agent that does the work well 99% of the time is also the agent that does the wrong thing well 1% of the time. The 1% can be expensive.

Specific patterns we'd recommend:

→ Default to draft, not send. It's the cheapest safety layer. → Make defer-to-human a first-class action. Not a fallback. → Surface confidence. Help humans calibrate review attention. → Audit everything. When something does go wrong, you need to understand what happened. → Test on REAL inputs that include edge cases. Test sets you control miss the edge cases the world generates.

The agent that scared us is still running

We didn't delete it. We made it safer.

It now processes our shared inbox daily. It drafts responses, classifies messages, routes appropriately. Humans review before anything goes out. The audit ledger logs everything.

Conservatively, it saves us 4-6 hours/week of triage and drafting. The scary moment cost us about 2 hours of work to fix and rebuild. Net positive by a wide margin.

But the experience changed how we think about agent design. Every agent we build now starts with the question: "What's the worst thing this could do, and what stops it?"

That's not a question most agent platforms surface. It should be.

→ avery.software — Free Desktop tier. The platform built by people who learned safety the hard way.