Flaky test

A flaky test is an automated test that passes and fails on the same code without any change. Common causes are timing issues, shared test data and unstable environments. Flaky tests erode trust in a suite, so teams track, quarantine and fix them rather than simply re-running them.

How to spot a flaky test

Run the same test several times on an unchanged build. If the result changes, the test is flaky. Most CI tools can rerun failures automatically and report which tests only passed on a retry.

Typical fixes

• Replace fixed sleeps with explicit waits for the element or state you need.
• Give each test its own data instead of sharing records between tests.
• Mock slow or unreliable external services.
• Run tests in a clean, isolated environment.

Why it matters for AI testing

LLM outputs vary from run to run, so exact-match assertions on AI answers behave like flaky tests. Score the answers with metrics and thresholds instead, as described under LLM evaluation.

Read the full guide: AI Test Automation vs Traditional Test Automation — The 2026 Reality Check