# AI Incident Response Agent

> An on-call SRE agent: correlates alerts with metrics, logs, and recent deploys, proposes safe mitigations, drafts status updates, escalates SEV1 — with approval gates.

- **Category:** DevOps & SRE
- **Difficulty:** Advanced
- **Setup time:** 50 min
- **Stacks:** Claude, LangGraph, OpenAI
- **Version:** 2.0.0 (updated 2026-06-21)
- **Source:** https://agent-kits.com/kit/incident-responder

## Overview

Alert → correlate → hypothesize → mitigate → communicate: a full first-responder loop grounded in your real telemetry.

Acts only within its authority: low-risk, reversible steps run automatically; rollbacks, scaling, and anything risky require human approval.

Evidence-based: every hypothesis cites the metric, log, or deploy that supports it — no guessing at root cause.

Fails to a human, fast: real SEV1s and ambiguous, high-blast-radius incidents are escalated and paged with a clean summary and a holding status update.

## When to use it

- You run on-call and want faster triage on the flood of alerts, especially the repetitive, well-understood ones.
- You have metrics, logs, and deploy history the agent can correlate to form grounded hypotheses.
- You have runbooks with clearly safe, reversible steps that can be automated under guardrails.
- You want consistent, timely status updates drafted during an incident.
- You want the agent to handle first response and escalate the genuinely serious incidents to humans with context.

## When not to use it

- You have no observability data for the agent to ground hypotheses in — it would be guessing.
- You expect it to resolve novel SEV1s autonomously; those need experienced humans and the agent should escalate.
- Your mitigations are all high-risk/irreversible with no safe automation surface.
- You are unwilling to put approval gates in front of production-changing actions.

## Architecture

- **Alert intake & dedup** — Ingests the alert from the monitoring/paging system, deduplicates against active incidents, and normalizes it with the affected service, signal, and threshold breached.
- **Telemetry correlation** — Queries metrics, logs/traces, and recent deploys/changes around the alert window to find what changed and when — the raw evidence for any hypothesis.
- **Hypothesis engine** — The model proposes the most likely cause strictly from queried signals, assigns severity and a blast-radius estimate, and cites the evidence behind each claim.
- **Guarded action layer** — A deterministic executor classifies each proposed action by risk and reversibility against an allow-list; only low-risk, reversible, allow-listed actions run autonomously, everything else needs approval.
- **Communication layer** — Drafts and posts concise status updates to the incident channel and stakeholders, honest about what is known and unknown, with no invented ETAs or root causes.
- **Escalation & paging** — For SEV1/SEV2, broad blast radius, or low confidence, pages on-call and hands over a structured summary with evidence and suggested next steps.
- **Postmortem feed** — Logs the full timeline — signals, hypothesis, actions, outcomes — to seed the postmortem and to tune which patterns are safe to auto-mitigate.

## Files in this bundle

- `system-prompt.md` — the agent's system prompt
- `tools.json` — tool/function definitions the agent expects
- `workflow.md` — step-by-step run loop
- `.env.example` — environment variables to fill in
- `kit.json` — full structured metadata
- `starters/` — minimal starter snippets
- `LICENSE` — Apache-2.0 (code, configuration, and schema)
- `LICENSE-CONTENT` — CC BY 4.0 (specification text and documentation)
- `NOTICE` — attribution and trademark notice
- `run.py` — **runnable demo**: real agent loop with mock tools (`pip install anthropic`, set `ANTHROPIC_API_KEY`, `python run.py`)
- `agentaz.json` / `agentaz.schema.json` — AgentAz™ governance spec + schema

## Implementation notes

- Define the autonomous action allow-list narrowly: only reversible, low-blast-radius steps. Everything that changes production state should be propose-and-approve, enforced outside the model.
- Never let the agent silence an alert it can't prove is non-actionable. Suppression needs cited evidence and a bounded duration, or it becomes a way to hide real incidents.
- Always require a blast-radius estimate before any action; broad scope or critical/customer-facing services automatically disqualify autonomous action.
- Start in copilot mode (proposes only). Review proposals for a few weeks, then enable responder mode for the allow-listed actions you trust.
- Status updates should state impact and the next update time, never an invented ETA or unconfirmed root cause — over-promising during an incident destroys trust.
- Log the full timeline (signals, hypothesis, actions, outcome) for every incident; it seeds the postmortem and shows which patterns are safe to automate next.
- Dedup and signal-gathering run on a cheaper model; the strong model handles the hypothesis and severity decisions.

## Variations

- **Basic — Triage co-pilot:** Correlates the alert with metrics, logs, and deploys and posts a severity, hypothesis, and suggested actions to the incident channel. Proposes only — humans act. The safe default.
- **Advanced — Guarded first responder:** Auto-executes allow-listed, reversible mitigations for known patterns, stages risky actions (rollback, scale-down) for one-click approval, and drafts status updates, with SEV1 auto-escalation.
- **Enterprise — Org-wide incident commander:** Adds service-aware policies and ownership routing, multi-signal correlation across teams, audited approvals, automatic postmortem timelines, and tuning of auto-mitigation patterns from incident outcomes.

## FAQ

**Will it take actions on production by itself?**

Only narrow, reversible, allow-listed ones (like restarting a stateless pod or silencing a proven-false alert). Anything that changes production state — rollbacks, scaling down, traffic shifts — is proposed for human approval, never executed autonomously.

**How does it decide the root cause?**

Strictly from telemetry it queries — metrics, logs, traces, and recent deploys — and every claim in its hypothesis cites the specific signal. If the evidence is missing or conflicting, it lowers confidence and escalates instead of guessing.

**Can it hide a real incident by silencing alerts?**

No. It can only suppress an alert it can prove is non-actionable, with cited evidence and a bounded duration, and it never downgrades severity to keep dashboards clean.

**What happens on a real SEV1?**

It declares the severity, pages on-call, posts an honest holding update, takes no risky automated action, and hands over a structured summary with evidence and suggested next steps — including what not to do.

**How do we adopt it safely?**

Start in copilot mode where it only proposes, replay past incidents to validate its reasoning, and enable guarded auto-mitigation for specific allow-listed actions once you trust its proposals.

**How is cost controlled during an incident?**

It queries the minimal set of signals needed to test a hypothesis rather than pulling every dashboard, caps its tool calls, and stops investigating once it can decide — escalating with current evidence if it hits the cap.


---

© 2026 AgentKits · www.agent-kits.com

**Open source.** Code, configuration, and the AgentAz™ schema are licensed under
**Apache-2.0**; specification text and documentation under **CC BY 4.0** (with
attribution to AgentKits, link to https://www.agent-kits.com). See LICENSE, LICENSE-CONTENT,
and NOTICE in this bundle. "AgentAz" and "AgentKits" are trademarks of AgentKits.

**Flagship reference blueprint.** View the source, AgentAz™ spec, and frozen v1.0 schema
in the public registry: https://github.com/agent-kits/agentaz
Original: https://agent-kits.com/kit/incident-responder
