Every tester knows the sinking feeling of a flaky test. Now picture a system where every test is a little flaky by design. That is what testing AI-enabled features feels like, and it is the problem I'll dig into at TestCon 2026 in my session, Taming the Beast: Testing the Unpredictable with Confidence.
Most teams building with AI today aren't training models. They're integrating them: a support chatbot, a RAG-powered product assistant, a summarization feature, an agent that can issue refunds. The engineering feels familiar. The testing doesn't. "Assert the output equals the expected string" stops working the moment one input produces three different, equally reasonable answers.
Here's the good news after 30-plus years in software quality: this is still software testing. The fundamentals hold. What changes is the toolkit, and more importantly, the mindset. Below is a preview of what I'll cover.
Ask an AI feature to summarize the same research article three times and you'll get three summaries that differ in structure, style and tone. If every output is different, how do you decide which ones are "correct"?
Non-determinism in AI models stems from several factors, and each source calls for a tailored testing approach:
The practical consequence is a shift from binary to statistical thinking. A single green check tells you very little. A distribution of results across repeated runs and perturbed inputs tells you a lot.
A common trap is writing evaluation criteria before anyone has actually read the outputs. Specs tell you what to check first. The traces tell you what those checks will miss.
Start with what the requirements already give you. For a product support assistant, that might be: every answer includes at least one citation, citations link to published documentation, responses stay under the length limit, and no internal document IDs appear in the reply. Those become cheap, deterministic assertions.
Then do the error analysis nobody wants to do:
In an illustrative run of that exercise, the top failure modes might be stale sources cited, invented product details, partial answers and out-of-scope answers. None of them would show up in a spec. That's the point: the failures that hurt you most are usually the ones no requirement predicted.
No single technique tames the beast. The modern AI test stack is layered, and each layer trades cost for judgment. From cheapest and most deterministic to most expensive and most subjective:
How you use the stack depends on where you are. When getting started, work top-down: write deterministic checks from the spec, read traces to find what they missed, then build judges. In steady state, run bottom-up, so cheap filters catch the obvious problems before the expensive layers ever see them.
Two habits keep this sustainable. Separate traditional tests from tests that call a model, so you control cost. And log every input, output and evaluation, so a human can review what the automation decided.
LLM-as-a-judge is powerful: it can be orders of magnitude cheaper than human review. Early work (Zheng et al., 2023) found GPT-4 judges reached over 80% agreement with human preferences, the same level humans reach with each other, but later studies show reliability varies sharply by task and judge. That's enough to catch regressions. It is not enough to support an absolute quality claim. Treat the judge as software that can have bugs.
Judges fail in predictable ways. They show position bias in pairwise comparisons, prefer longer answers, favor outputs from their own model family, flip verdicts on small formatting changes, and perform near chance on adversarial safety inputs. The mitigations are just as concrete:
Most importantly, validate the judge before you trust it. Split your human-labeled traces into a dev set (to tune the judge prompt) and a test set (measured once, at the end). Then track two numbers, not one: the share of real failures the judge catches, and the share of real passes it confirms.
Why not a single agreement percentage? On 100 traces with 8 real failures, a judge that passes everything agrees with humans 92% of the time and catches none of the failures.
The same discipline applies when you report results. Take an illustrative example: if release N-1 scored 91% and release N scored 88% on a 200-case eval set, that looks like a regression. Pair the cases and measure the change, though, and the interval around the -3.0 point drop crosses zero. You may be looking at noise. A judge is a test instrument with a known error rate; report it the way you'd report any measurement.
Agentic systems raise the stakes. A single LLM call becomes a chain of 5 to 50+ calls, state propagates from step to step, and one judge score is no longer enough. Consider a refund agent that tells the customer, "Your refund of $89 has been processed." The reply looks perfect. The database shows a refund of $890.
That's why I grade agents on three things:
A customer only ever sees one attempt. Reliability means every attempt works.
Security deserves the same rigor. Refusal tests tell you whether the model resisted a prompt injection. Just as important is testing what happens when an injection gets through. Don't wait for the model to be fooled: plant the payload or force the call, then assert deterministically on the trace. No call to an unapproved tool. No outbound call containing private content. Agent credentials that can't read another customer's records. Rendered links and images that resolve only to approved domains. Refunds, sends and deletes above a limit that require confirmation. These tests are deterministic and cheap.
Not every AI feature needs the full stack. The right depth depends on what a failure costs:
In high-risk use cases, a missed failure costs more than a false alarm, so tune your checks to catch failures and budget human review for the extra false alarms. Set thresholds from your own observed distributions; published numbers don't transfer across embedding models or domains. And remember that regulatory classification can set the floor, so thresholds need to hold up in an audit.
Finally, close the loop. Traditional QA already reviews production bugs for the regression suite, or at least it should. In AI QA, that loop becomes automated and continuous. Production traces flow into an observability platform. Implicit user signals such as abandoned conversations, escalations and repeated rephrasing flag obvious failures, though only a small fraction of users ever signal. Automated scoring covers the rest by sampling 5–10% of traffic plus 100% of high-risk categories. Humans confirm and categorize what's flagged, and it goes straight into the regression suite.
The same scorers protecting your CI pipeline are now sampling production. Failures surface proactively, and your test suite grows continuously instead of bug by bug.
I'll be presenting Taming the Beast: Testing the Unpredictable with Confidence at TesCon 2026 on Wednesday, Oct 21st at 10:10. If you're a tester, QA lead or engineering leader shipping AI-enabled features, you'll leave with:
You don't need to become an ML engineer to test AI well. Curiosity, analytical rigor and solid test design still do most of the work; the new layer is learning to measure behavior instead of matching strings. It's still software testing. It just asks more of us.
If you'll be at TesCon, I hope you'll join the session. If you can't make it, let's continue the conversation: reach me at lee.barnes@fortegrp.com, or on LinkedIn.
83% faster delivery. Under 10% rework. See exactly how Xceptor got there.
