# Flaky Test Triage Agent

> A flaky-test triage agent that re-runs failures in isolation, uses history to separate flakes from real regressions, and quarantines proven flakes. Never masks a bug.

- **Category:** QA & Testing
- **Difficulty:** Advanced
- **Setup time:** 45 min
- **Stacks:** Claude, LangGraph, OpenAI
- **Version:** 2.0.0 (updated 2026-06-21)
- **Source:** https://agent-kits.com/kit/flaky-test-triager

## Overview

Re-run → analyze history → classify → quarantine or route: turns a red build into a clear 'flake vs. real bug' verdict.

Evidence-based: it isolates the failure and weighs failure history and patterns (timing, ordering, environment) before judging.

Quarantines proven flakes behind a gate to unblock the pipeline — and routes real regressions straight to the code author.

Defensive: it never masks a real bug, needs repeated confirmation to call a test flaky, and escalates ambiguous or critical-path failures.

## When to use it

- Flaky tests are eroding trust in your CI and engineers are re-running builds or ignoring reds.
- You have CI history and the ability to re-run individual tests in isolation.
- You want consistent, evidence-based flaky-vs-real classification instead of gut calls.
- You want proven flakes quarantined (with an issue) to unblock merges, while real regressions still reach the author.

## When not to use it

- You can't re-run individual tests or lack failure history — classification would be guessing.
- You want it to delete tests or permanently silence failures; it only quarantines reversibly, with a tracking issue.
- Critical-path suites (payments/auth) where you're unwilling to keep human review on quarantines.
- Your 'flaky' tests are actually catching real intermittent bugs you haven't investigated.

## Architecture

- **Failure intake** — Receives the failing test from CI with its logs, the diff/commit under test, and the job/environment context.
- **Isolated re-run harness** — Re-runs just the failing test in isolation (and a few times within a cap) to see whether the failure reproduces deterministically or intermittently.
- **History & change correlation** — Pulls the test's recent pass/fail history and checks whether the failure correlates with a recent change to the code under test — the key flaky-vs-real signal.
- **Signature analysis** — Looks for flakiness signatures — timing/races, test-ordering dependence, shared state, environment/network — to support (or refute) a flaky verdict.
- **Verdict & quarantine gate** — The model classifies on the evidence; a deterministic gate blocks quarantine for critical-path tests and for anything that reproduces or correlates with a code change.
- **Quarantine & tracking** — Reversibly skips a proven flake with an auto-filed tracking issue and owner, and routes real failures to the author — never deleting tests.
- **Feedback & flake registry** — Maintains a flakiness registry and logs verdict outcomes so repeat offenders are spotted and misclassifications tune the thresholds.

## Files in this bundle

- `system-prompt.md` — the agent's system prompt
- `tools.json` — tool/function definitions the agent expects
- `workflow.md` — step-by-step run loop
- `.env.example` — environment variables to fill in
- `kit.json` — full structured metadata
- `starters/` — minimal starter snippets
- `LICENSE` — Apache-2.0 (code, configuration, and schema)
- `LICENSE-CONTENT` — CC BY 4.0 (specification text and documentation)
- `NOTICE` — attribution and trademark notice
- `run.py` — **runnable demo**: real agent loop with mock tools (`pip install anthropic`, set `ANTHROPIC_API_KEY`, `python run.py`)
- `agentaz.json` / `agentaz.schema.json` — AgentAz™ governance spec + schema

## Implementation notes

- Require positive, multi-signal evidence to call a test flaky (isolated pass + intermittent history or a clear signature). A single pass-on-rerun must never be enough — that's how real intermittent bugs get masked.
- Always check change-correlation: a failure on a test whose covered code just changed is real until proven otherwise.
- Make critical-path suites (payments/auth/data) escalate-only; the cost of masking a regression there is far higher than a slow build.
- Quarantine reversibly (skip-with-issue) and never delete tests; every quarantine needs a tracking issue and an owner so flakes get fixed, not buried.
- Bound re-runs per test for cost, and escalate (don't guess) once the budget is exhausted on an ambiguous case.
- Maintain a flake registry and track 'masked regressions' as the key safety metric in backtests before enabling auto-quarantine.
- Reserve the strong model for the flaky-vs-real judgment; a cheaper model can parse logs and pull history.

## Variations

- **Basic — Flake classifier:** Re-runs the failure, analyzes history and signatures, and reports a flaky/real verdict with evidence and a recommendation for an engineer. No auto-quarantine.
- **Advanced — Guarded auto-quarantine:** Auto-quarantines proven non-critical flakes (skip-with-issue + owner) to unblock CI, routes real failures to authors, and escalates critical-path and ambiguous cases.
- **Enterprise — Org-wide flake management:** Adds a cross-repo flake registry, trend analytics, CODEOWNERS routing, quarantine SLAs and auto-expiry, and threshold tuning from masked-regression metrics at scale.

## FAQ

**Will it hide real bugs by quarantining tests?**

No — that's its primary safety constraint. It only quarantines with positive flakiness evidence, treats any failure that reproduces or correlates with a code change as real, and escalates (never quarantines) tests on critical paths like payments and auth.

**How does it decide flaky vs. real?**

It re-runs the failing test in isolation, checks its pass/fail history, looks for flakiness signatures (timing, ordering, shared state, environment), and correlates against recent code changes. Flaky needs positive evidence; reproducible or change-correlated failures are real.

**Does it delete flaky tests?**

Never. Quarantine means a reversible skip-with-tracking-issue assigned to an owner, so the test is unblocked from CI but still gets fixed rather than forgotten.

**What about tests on critical paths?**

Tests covering payments, auth, or data integrity are escalate-only — it won't auto-quarantine them even if they look flaky, because an 'intermittent' failure there can be a real bug.

**How is the cost of re-running controlled?**

It re-runs only the single failing test in isolation, up to a configured cap, rather than the whole suite — and if a case is still ambiguous after the cap, it escalates instead of burning more runs.

**How do we adopt it safely?**

Start in advise mode where it only recommends, backtest on recent failures to confirm it would mask zero regressions, then enable auto-quarantine for non-critical proven flakes.


---

© 2026 AgentKits · www.agent-kits.com

**Open source.** Code, configuration, and the AgentAz™ schema are licensed under
**Apache-2.0**; specification text and documentation under **CC BY 4.0** (with
attribution to AgentKits, link to https://www.agent-kits.com). See LICENSE, LICENSE-CONTENT,
and NOTICE in this bundle. "AgentAz" and "AgentKits" are trademarks of AgentKits.

**Flagship reference blueprint.** View the source, AgentAz™ spec, and frozen v1.0 schema
in the public registry: https://github.com/agent-kits/agentaz
Original: https://agent-kits.com/kit/flaky-test-triager
