# Examples — Flaky Test Triage Agent

## Intermittent, timing-dependent test → quarantine as flaky

A test fails once, passes on isolated re-run, and history shows it flips pass/fail with no related code change — a classic race.

**Input**

```
Failure: test_async_notify failed in CI. No changes to the notify module in this PR.
```

**Output**

```
{
  "test": "test_async_notify",
  "verdict": "flaky",
  "confidence": 0.9,
  "evidence": ["isolated re-run: passed 5/5", "history: failed 6 of last 80 runs across unrelated PRs", "logs show it asserts before an async callback completes (race)", "no change to the notify module in this PR"],
  "signature": "timing",
  "critical_path": false,
  "decision": "QUARANTINE_FLAKY",
  "actions": [
    { "tool": "quarantine_test", "args": { "test": "test_async_notify", "mode": "skip_with_issue" }, "requires_approval": false },
    { "tool": "file_issue", "args": { "title": "Flaky: test_async_notify (async race)", "owner": "@notify-team" }, "requires_approval": false }
  ],
  "issue": "Flaky: test_async_notify (async race) — assigned @notify-team",
  "author_note": "",
  "escalation": { "needed": false, "reason": "" }
}
```

**Note:** Strong flaky evidence: passes 5/5 in isolation, intermittent history across unrelated PRs, a clear timing/race signature, and no related code change. It quarantines reversibly (skip-with-issue) and assigns an owner — unblocking the build without losing the test.

## Reproduces on re-run + correlates with the change → REAL, don't quarantine

The failure looks like it could be flaky, but it reproduces every re-run and the PR changed exactly the code under test. The agent refuses to quarantine.

**Input**

```
Failure: test_discount_calc failed. This PR modifies pricing/discount.py.
```

**Output**

```
{
  "test": "test_discount_calc",
  "verdict": "real",
  "confidence": 0.92,
  "evidence": ["isolated re-run: failed 5/5 (deterministic)", "this PR modifies pricing/discount.py — the exact code under test", "history: passed 100% before this PR", "assertion: expected 90.0, got 99.0 (discount not applied)"],
  "signature": "none",
  "critical_path": false,
  "decision": "REAL_FAILURE",
  "actions": [],
  "issue": "",
  "author_note": "test_discount_calc fails deterministically (5/5) and your PR changes pricing/discount.py — expected 90.0 but got 99.0, so the 10% discount isn't being applied. This looks like a real regression, not a flake. Please check the discount calculation before merging.",
  "escalation": { "needed": false, "reason": "" }
}
```

**Note:** The core safety behavior: it would be tempting (and harmful) to quarantine a red test to unblock the PR. But the failure is deterministic and correlates exactly with the changed code, so the agent calls it REAL, keeps the build red, and hands the author a precise diagnosis. Quarantining here would have shipped a pricing bug.

## Looks flaky but it's a payments test → escalate, never auto-quarantine

A test on the payments path shows intermittent behavior. Even though it pattern-matches to flaky, the critical-path rule forces a human.

**Input**

```
Failure: test_payment_capture_retry intermittently fails (passed 3/5 on re-run). Covers the payment capture/retry path.
```

**Output**

```
{
  "test": "test_payment_capture_retry",
  "verdict": "ambiguous",
  "confidence": 0.6,
  "evidence": ["isolated re-run: passed 3/5, failed 2/5 (intermittent)", "history: occasional failures", "covers payment capture + retry logic (critical path)", "possible real intermittent bug in retry/idempotency, not just test flakiness"],
  "signature": "timing",
  "critical_path": true,
  "decision": "ESCALATE",
  "actions": [
    { "tool": "escalate_to_owner", "args": { "owner": "@payments", "priority": "high" }, "requires_approval": false }
  ],
  "issue": "",
  "author_note": "",
  "escalation": { "needed": true, "reason": "Intermittent failure on a payments-path test. It looks flaky, but intermittent failures in capture/retry can indicate a real idempotency/race bug. Not auto-quarantining a critical-path test — escalating to @payments to investigate whether the retry logic is actually correct." }
}
```

**Note:** The defining example: the test pattern-matches to flaky (passes 3/5), but it covers payments — where an 'intermittent' failure might be a real idempotency/race bug that only sometimes triggers. Auto-quarantining it could hide a money-losing defect, so the critical-path rule overrides and it escalates to the owner with that exact concern.
