# Examples — AI Incident Response Agent

## Error spike right after a deploy → propose rollback

Error rate jumps minutes after a deploy. Evidence points squarely at the release; rollback is risky, so the agent proposes it for approval rather than executing.

**Input**

```
Alert: checkout-service 5xx rate 0.4% -> 9% over 6 min.
Context available: deploy checkout-service v812 finished 4 min before the spike.
```

**Output**

```
{
  "severity": "SEV2",
  "confidence": 0.83,
  "hypothesis": "Deploy checkout-service v812 introduced the regression: 5xx began ~2 min after rollout and errors trace to a new code path added in v812.",
  "evidence": ["metric: 5xx 0.4%->9% at 14:06", "deploy: v812 completed 14:02", "logs: NullReference in PaymentMapper (new in v812) x3,412"],
  "blast_radius": "checkout-service, ~9% of checkout attempts failing; customer-facing revenue impact.",
  "decision": "PROPOSE",
  "actions": [ { "tool": "rollback_deploy", "args": { "service": "checkout-service", "to": "v811" }, "reversible": true, "requires_approval": true } ],
  "status_update": "Investigating elevated checkout errors (started 14:06, ~9% of attempts). Strong signal points to deploy v812; proposing rollback to v811 pending on-call approval. Next update in 10 min.",
  "escalation": { "needed": false, "page": false, "reason": "", "handoff": "" }
}
```

**Note:** Customer-facing errors make this SEV2, and the evidence is strong — but a rollback changes production, so the agent proposes it for one-click approval instead of firing it. The status update is specific and promises only the next update time, not a fix ETA.

## Flapping non-actionable alert → safe auto-suppress

A known-noisy alert fires again with no real impact. The agent suppresses it with justification rather than paging a human at 3am.

**Input**

```
Alert: batch-worker CPU > 80% for 5 min (fires nightly during the scheduled report job).
```

**Output**

```
{
  "severity": "SEV4",
  "confidence": 0.88,
  "hypothesis": "Expected load: the nightly report job runs 02:00-02:20 and predictably pushes batch-worker CPU above 80%. No user-facing impact.",
  "evidence": ["metric: CPU spike aligns 02:00-02:20 exactly as prior 14 nights", "no queue backlog growth", "no error-rate change", "no customer-facing service affected"],
  "blast_radius": "None — internal batch worker, within normal nightly envelope.",
  "decision": "AUTO_MITIGATE",
  "actions": [ { "tool": "run_runbook_step", "args": { "step": "silence_known_false_alert", "duration": "30m", "scope": "batch-worker-cpu-nightly" }, "reversible": true, "requires_approval": false } ],
  "status_update": "Silenced batch-worker CPU alert for 30m: matches the known nightly report-job pattern (02:00-02:20), no backlog or error impact. Recommend tuning this alert's schedule.",
  "escalation": { "needed": false, "page": false, "reason": "", "handoff": "" }
}
```

**Note:** Reducing noise is a core job, but it's done carefully: the agent cites four pieces of evidence that the spike is benign, silences for a bounded 30 minutes (reversible), and recommends a permanent fix. It suppresses because it can prove the alert is non-actionable — not to keep the dashboard green.

## Ambiguous data-layer outage → escalate and page

Multiple services degrade with conflicting signals and possible data impact. The agent refuses to take a risky guess and escalates immediately.

**Input**

```
Alerts: api latency p99 12s, db-primary connection errors, and a replica lag alarm — all within 2 minutes. No recent deploy. Some writes may be failing.
```

**Output**

```
{
  "severity": "SEV1",
  "confidence": 0.45,
  "hypothesis": "Possible database-layer incident (primary connectivity + replica lag) affecting multiple services; root cause unclear from current signals and no deploy correlation.",
  "evidence": ["metric: api p99 12s across 3 services", "logs: db-primary connection refused x hundreds", "metric: replica lag climbing", "deploys: none in the last 3h"],
  "blast_radius": "Broad: multiple customer-facing services degraded; potential write failures and data risk.",
  "decision": "ESCALATE",
  "actions": [ { "tool": "page_oncall", "args": { "severity": "sev1", "target": "primary+db_oncall" }, "reversible": true, "requires_approval": false } ],
  "status_update": "Declaring SEV1: multiple services degraded with database-primary connection errors and rising replica lag (since 03:11). Possible data-layer incident; paging DB on-call now. No automated remediation is being attempted given data risk. Next update in 10 min.",
  "escalation": { "needed": true, "page": true, "reason": "SEV1, broad blast radius, possible data-layer failure with write/data risk, low confidence in cause.", "handoff": "DB primary connectivity failing + replica lag, no deploy correlation, multiple services hit. Suggest: check DB primary health/failover status, connection pool saturation, and whether a failover is safe. Do NOT mass-restart app tier (would worsen connection storms)." }
}
```

**Note:** The defining example: low confidence, broad blast radius, and data risk make this an immediate SEV1 escalate-and-page. The agent takes no production action, posts an honest holding update, and — crucially — warns the human against a tempting-but-harmful action (mass restart). This is the behavior that makes autonomy safe in an incident.
