Build a server and endpoint health monitoring agent

Your on-call engineer gets paged at 3 AM for a known issue with a known fix.

They wake up, SSH in, run the same three commands they ran last month, watch the service restart, confirm metrics return to normal, and go back to sleep. This loop has happened every Tuesday for six months. The fix is documented in Confluence. Nobody automated it.

Traditional monitoring tools alert. They don't remediate. You get a Datadog page, the runbook is in Confluence, the commands are SSH commands. Someone has to run them manually.

AIOps tools promise to fix this. Most just add another layer of alert noise. The remediation step never quite materializes because the architecture isn't built for it.

This post is the build for a local-first AI agent that monitors infrastructure, correlates anomalies, auto-remediates known issues with explicit guardrails, and pages humans only when something genuinely new requires attention.

Why traditional monitoring fails the remediation gap

The standard monitoring stack: metrics in Datadog, logs in Splunk or ELK, alerts via PagerDuty, runbooks in Confluence, fixes via SSH.

The gap between alert and fix is a human. The human is the bottleneck. The human is also the cost.

Alert fatigue is the obvious failure mode. Engineers get paged for noise. They stop reading the alerts. The real incident sits in the queue while engineers tune out.

Documentation drift is the subtle failure mode. The runbook from 2023 says to restart service X. The actual fix in 2026 requires also clearing a Redis cache and rotating a credential. The runbook hasn't been updated. New engineers fail the runbook. Senior engineers spend time being the "actual fix" oracle.

The remediation isn't the hard part. The hard part is encoding the institutional knowledge into something that runs autonomously and safely.

What the agent does

The agent fills the gap between alert and fix.

Health checks. The agent runs a continuous schedule of checks against your infrastructure. CPU, memory, disk, network, application-specific metrics. Compares against expected ranges.

Correlation. When an anomaly fires, the agent correlates with other signals. Did the deploy half an hour ago cause this? Did the upstream service start throttling? Is this the Tuesday pattern recurring?

Pattern matching. The agent has a knowledge base of past incidents and their fixes. It matches the current pattern against history. If it's a known pattern, the fix is known.

Auto-remediation. For known patterns with documented safe fixes, the agent executes the fix. Restart the service. Clear the cache. Increase the threshold. With explicit guardrails on what it can and cannot do.

Verification. After remediation, the agent verifies the fix worked. Metrics back to normal? Alert cleared? If yes, log and move on. If no, escalate to human.

Alerting for unknowns. For patterns the agent doesn't recognize, alert a human with full context. The human handles the genuinely new situation; the agent learns the new pattern for next time.

That's the system. Six Avery NXR CRs, four to five days for a typical SRE team.

The Avery NXR build

CR 1: Health check runner.

"Build a scheduled health check system. Each check has: id, name, target_host, command, expected_output_pattern, threshold_min, threshold_max, frequency_seconds. The runner executes commands via SSH (with keys managed in OS keychain), captures output, parses numerical metrics, and compares to thresholds. Store results in HealthCheckResult table with timestamp, value, status (ok, warning, critical). Run continuously."

CR 2: Anomaly detection.

"For each new HealthCheckResult, evaluate against historical patterns. If value is outside thresholds, mark as anomaly. Use a simple statistical model (z-score against 14-day rolling average) for additional pattern detection beyond static thresholds. Create an Anomaly record with type (threshold_breach, statistical_anomaly), severity (info, warning, critical), and link to relevant historical context (similar anomalies in past 30 days)."

CR 3: Incident pattern matching.

"For each Anomaly, search the IncidentPattern knowledge base for matching prior incidents. Match on: affected host, anomaly type, time-of-day patterns, related anomalies in same time window. Local model classifies the match (high, medium, low confidence). For high-confidence matches, surface the prior incident's resolution. For medium, present options. For low, flag as new pattern requiring human investigation."

CR 4: Safe auto-remediation.

"Build an AutoRemediation system. Each pattern in the knowledge base can specify a remediation: command (from a strict allowlist), expected_outcome, verification_check, rollback_command (if needed). When a high-confidence pattern match occurs and AutoRemediation is enabled for that pattern, execute the remediation. The allowlist constraints: cannot delete data, cannot modify production configuration outside an approved change window, cannot take destructive actions ever. Anything outside the allowlist triggers human escalation."

The guardrails matter. Auto-remediation without strict boundaries is how outages get worse.

CR 5: Verification and rollback.

"After auto-remediation, run the verification check. If verification passes, log success and close the anomaly. If verification fails, run the rollback command if defined, then escalate to a human. Either way, log the entire flow: original anomaly, remediation attempted, verification result, final state."

CR 6: Alerting for unknowns.

"For anomalies that don't match known patterns or for patterns where auto-remediation is disabled, page a human via PagerDuty. The page includes: the anomaly details, the related context (other anomalies in the same window, recent deploys, recent config changes), the most-similar past incidents (with confidence scores), and suggested investigation steps based on the pattern category."

CR 7: Pattern learning.

"After a human resolves an incident, capture the resolution in the IncidentPattern knowledge base. The human marks: the root cause, the fix command, whether this should be eligible for auto-remediation, and the conditions under which to apply the fix. Future similar anomalies can match this pattern."

CR 8: Dashboard.

"Build /ops/dashboard showing: current anomalies, auto-remediations in last 24h with outcomes, patterns by frequency, human escalations in last 24h. Allow filtering by host, severity, time range. For each remediation, show the full audit log of what was checked, what was done, what was verified."

Eight CRs total. Four to five days for the basic build.

The safe auto-remediation pattern

The most important part of this system is what the agent CAN'T do.

The allowlist for auto-remediation commands:

Restart services from a defined list of restartable services.

Run safe diagnostic commands (status checks, log tail, metric queries).

Clear caches (Redis, application caches with documented purge procedures).

Rotate credentials in approved key management systems.

Scale up replicas within a defined range (autoscaler-equivalent within bounds).

The denylist is the more important list:

No deletion of data. Ever. No DROP TABLE, no rm -rf, no log purges that lose forensic data.

No modification of production configuration outside an approved change window.

No actions on systems flagged as critical/sensitive without explicit per-incident human approval.

No actions that affect customer-visible behavior (no traffic redirects, no feature flag toggles, no customer notification triggers).

No actions on systems the agent hasn't been onboarded to.

The agent's superpower is that it never gets bored or makes mistakes from inattention. The constraint is that it should never take actions whose blast radius isn't bounded.

A common failure mode in AIOps deployments: the team gives the agent too much autonomy. The agent makes a wrong call. The blast radius is huge. Trust evaporates.

Better pattern: start with very narrow auto-remediation (the obvious Tuesday issue), expand the allowlist as the agent earns trust, never let it touch the high-blast-radius actions.

Real numbers from a deployment

A SRE team at a fintech company running about 100 services in production:

Pre-agent: 47 on-call pages per week on average. Median time to resolve: 32 minutes. On-call engineers had 4-hour blocks of degraded productivity after night pages.

Post-agent (after 6 weeks of tuning): 18 on-call pages per week. The agent auto-remediated 65 percent of pages that would have been generated. Median human resolution time for the remaining pages: 22 minutes (because the agent provided rich context).

On-call quality of life improved dramatically. The on-call engineer wasn't fixing the Tuesday issue at 3 AM anymore. The agent did it.

The team's morale improved. Retention of senior SREs improved. Recruiting for SRE roles improved because "low-noise on-call" became a recruiting talking point.

The build took about 80 engineering hours. The ongoing maintenance is a few hours per week of pattern tuning. The pages avoided per week translate to real money in engineering productivity.

What this is NOT

Worth being explicit about the boundaries.

This is not a replacement for SRE judgment on novel incidents. The agent handles patterns it's been taught. New patterns require humans.

This is not a replacement for proper monitoring infrastructure. The agent sits on top of Datadog/Grafana/whatever you use. You still need the underlying observability.

This is not an excuse to stop investing in reliability. The agent is a backstop for known issues. It doesn't fix the architectural debt that makes Tuesday issues happen.

This is not a tool that "thinks." It's a pattern-matching layer with execution capability. Set expectations accordingly.

When this fails

The honest failure modes:

The first month is painful. The agent doesn't know your patterns yet. Engineers feel like they're babysitting it. This phase ends when the knowledge base reaches critical mass (typically 30-50 documented patterns).

Pattern matching across complex distributed systems is harder than single-server health checks. Multi-service correlation is its own subproblem.

The agent over-relies on recent patterns. New failure modes that look superficially similar to old ones can get misclassified. Periodic human review of auto-remediations catches this.

The agent doesn't learn from contextual signals well. "Don't restart this service during the marketing campaign launch window" is harder to encode than rule-based remediation. Human judgment still wins for these.

For these failure modes, the answer is humility: keep humans in the loop for novel things, audit auto-remediations periodically, expand the allowlist gradually.

Integration patterns

The agent integrates with your existing observability stack rather than replacing it.

Datadog/Grafana for metrics. The agent reads via API. Doesn't replace the visualization or alerting infrastructure.

PagerDuty/Opsgenie for paging. The agent reduces the page count rather than replacing the paging system.

Splunk/ELK for logs. The agent queries via API to fetch context.

Slack for human-facing notifications and quick acks.

GitHub/GitLab for runbooks (if you keep runbooks as code). The agent reads the runbooks and uses them as input to pattern matching.

The integration story is "we plug into your stack" not "rip and replace."

The v1.1 additions

After the core system is shipping:

Capacity planning. The agent observes long-term trends and predicts when you'll need to scale up specific services.

Cost optimization. The agent identifies over-provisioned resources during quiet periods and (with appropriate guardrails) scales them down.

Change impact assessment. Before a deploy, the agent predicts which patterns are likely to fire based on the changeset.

Cross-incident correlation. The agent connects related incidents across services (this database slowdown caused this API timeout caused this frontend error).

Postmortem assistance. The agent generates a timeline of an incident from its logs, sufficient for the human to write a quality postmortem in minutes instead of hours.

Each adds value. None are urgent for the initial deployment.

The economic argument

Cost of an unplanned on-call page at 3 AM (engineer time, productivity impact next day, indirect morale cost): $200 to $500 depending on company.

At 47 pages per week pre-agent: $10K to $25K per week in absorbed cost.

After the agent reduces pages to 18 per week: $4K to $10K per week in absorbed cost.

The savings per week alone justify the engineering investment in weeks.

The harder-to-quantify benefits: better SRE retention, faster incident resolution, better company reputation for reliability. All real, all hard to quantify, all positive.

The closing thought

Monitoring without remediation is half a solution. Traditional tools have prioritized alerting because alerting is easier to build. The remediation gap has stayed open because the architecture wasn't right.

Local-first AI agents close the gap. The agent runs where the infrastructure runs. The pattern knowledge stays local. The remediation has tight guardrails. Auto-remediation works for known patterns; humans handle novel ones.

If your on-call team is spending its time on issues that have known fixes documented in your runbook, this build is the priority. The fixes were already automatable. The agent just makes them autonomous.

avery.software