Why non-determinism breaks traditional testing
AI systems are fundamentally non-deterministic: the same input can produce slightly different outputs from one run to the next. A classic test is deterministic, it asserts that a calculation always returns exactly the same result. A sentiment-analysis test, however, can legitimately return "POSITIVE", "positive", "Positive", or even an emoji depending on the run: a simple strict equality check no longer cuts it, and it's the whole foundation of software testing that needs rethinking.
Comprehensive testing of an AI system covers seven complementary dimensions: data quality, model performance, behavior, fairness and bias, robustness, end-to-end integration, and production monitoring treated as continuous testing. None of these dimensions is enough on its own, a model that performs well on degraded data, or that is fair but vulnerable to prompt injection, remains a production risk.
The shift in mindset is simple to state: you replace exact expected values with thresholds, tolerance bands, and general properties that must hold true. Accepting non-determinism doesn't mean giving up rigor, it means moving rigor from outputs to distributions, behaviors, and invariants.
The stakes go beyond engineering: AI systems now make decisions that commit the business, granting a loan, prioritizing a ticket, answering a customer. Without a tooled-up testing strategy, every hallucination or silent drift is discovered in production, at the cost of an incident, customer harm, or regulatory exposure. Frameworks such as the European AI Act impose explicit robustness, traceability, and data governance requirements on high-risk systems: testing is no longer just good practice, it's a compliance obligation.
Testing data quality and data drift
Data is the fuel of AI: poor-quality data produces poor-quality models, no matter how much engineering effort goes in downstream. Tools like Great Expectations or Soda formalize these checks as automated expectations: the presence of expected columns, type consistency, plausible value ranges, no null values on required fields, identifier uniqueness, and statistical distribution consistency.
Catching data drift before it degrades the model
Beyond point-in-time quality, you also need to detect gradual data drift. Tools like Evidently compare the distribution of training data with what's observed in production, column by column, and trigger an alert as soon as statistically significant drift appears, a signal that the model risks drifting out of sync with the real world well before its business metrics visibly collapse. These checks run continuously, not only at training time.
Evaluating model performance with thresholds, not exact values
Rather than expecting an exact output, you set minimum thresholds to meet: accuracy of at least 0.85, an F1 score of at least 0.80, an AUC-ROC of at least 0.90. These thresholds become test assertions executed on every training run: if the new model falls below the bar, the pipeline fails and the deployment is blocked, exactly the way a red unit test blocks a software release.
An equally important point is checking performance consistency across population segments, age brackets, regions, acquisition channels, making sure the accuracy gap between the best- and worst-performing segments stays under a reasonable threshold, 10% for example. A flattering global metric can hide a catastrophic blind spot on a minority segment; only a per-segment breakdown reveals it.
Finally, these thresholds fit into a non-regression logic: every candidate is compared with the champion model in production on the same frozen evaluation set, and only replaces it if it does better, or at least as well, on every critical metric. Shadow deployments, where the challenger receives real traffic without its predictions being used, make it possible to verify that verdict on live data before switching over.
Behavioral testing for LLMs: judges, properties, and golden sets
For LLMs, you test expected behaviors rather than a literal output: does the model correctly identify a sentiment, does it refuse to repeat a credit card number a user provided, does it decline dangerous requests, does it stay in its assigned role when faced with a hijacking attempt, and does it respect the requested output format, structured JSON, for example?
Property-based testing and LLM-as-a-judge
Instead of checking isolated examples, you formulate general properties that must hold across a large number of generated inputs: a summary must always be shorter than the source text, a classification must always return one of the valid categories. The evaluation frameworks have matured: DeepEval works like a Pytest specialized for LLM outputs, RAGAS remains the reference for evaluating RAG pipelines, Promptfoo excels at cross-model matrix testing, and Arize Phoenix covers tracing and observability. All of them rely on LLM judges (LLM-as-a-judge) that score responses against custom criteria, faithfulness, relevance, tone.
Evaluation now extends to agents: beyond the final answer, you verify the full trajectory, tool selection, call ordering, respect for the token budget and permissions. Standardized OpenTelemetry traces make these trajectories observable and comparable, turning every agent session into a potential test case for the evaluation suite.
Keeping evaluations stable in CI
To avoid flaky builds, mature teams apply three rules: tolerance bands rather than exact thresholds, a judge model pinned to a fixed version, and a golden set of examples that is stable and versioned like code. Many combine two tools, an evaluation framework at development time and an observability platform in production, because no single one covers both needs well. The composition of evaluation sets deserves the same care as code: representativeness of real cases, coverage of edge cases, strict separation between training and test data, and continuous enrichment from failures observed in production, a golden set that never grows becomes an exam the system learns to pass without improving.
Fairness, robustness, and adversarial red teaming
Testing that a model doesn't discriminate means comparing its outcomes across demographic groups using dedicated metrics, supported by libraries like AIF360 or Fairlearn: disparate impact, which should stay close to 1.0 (typically between 0.8 and 1.2), and equal opportunity difference and average odds difference, which should stay close to 0. These metrics turn a fairness intuition into testable, automatable thresholds that stand up in an audit.
Red teaming and prompt injection attacks
Robustness is tested by deliberately subjecting the system to perturbations and adversarial prompts: noise added to features, attempts to ignore instructions, injections of new directives, requests to reveal the system prompt. The model must resist without ever exposing its instructions or its training data. Tools like Promptfoo automate these red-teaming campaigns aligned with the OWASP Top 10 for LLM applications, which ranks prompt injection as the leading risk, and these adversarial suites run in CI, like any other security test.
From the ML testing pyramid to continuous production monitoring
Test the full system, not just the model: an end-to-end test follows the whole business flow, a user event arrives, it's enriched by the feature store, the model produces a prediction, a business action fires if the risk crosses a threshold, and the prediction is logged for later analysis. Every link in the chain must be validated, not just the model's final output.
The ML testing pyramid and the CI/CD pipeline
Unlike the classic software pyramid, the AI version rests on a wide base of data tests: typically 20% data quality tests, 30% behavioral tests, 30% model performance tests, 15% integration tests, and only 5% end-to-end tests. In the CI/CD pipeline, every trigger chains together data quality, training, performance, behavior, fairness, robustness, and then integration, with deployment happening only once every step has passed.
Testing then continues in production: a Kolmogorov-Smirnov test compares the distribution of the last hour's predictions with that of the past week, a high rate of uncertain predictions signals a model hesitating unusually often, and a latency P95 above 500 ms reveals an operational problem. Monitoring acts as a test suite that never stops running, and its alerts feed back into the CI's golden sets and thresholds.
Deployment strategies are part of the testing toolkit too: a canary release exposes the new model to a fraction of traffic with automatic rollback thresholds, while an A/B test measures the real business impact, conversion rate, resolution rate, beyond the technical metrics. It's this complete loop, from offline testing to online validation, that separates teams that endure their models from teams that steer them.
Building measurable trust in AI systems
Testing AI systems requires a radically different approach from traditional software testing, but it comes down to five principles: test the data as much as the model, use thresholds and tolerances rather than exact values, test behaviors rather than implementation, automate everything that can be automated, and treat production monitoring as continuous testing.
At Adservio, we help teams put these evaluation strategies in place end to end, from framework selection to CI/CD integration and monitoring. The conclusion is the same everywhere: trust in AI doesn't come from eliminating uncertainty, it comes from the ability to measure, monitor, and manage that uncertainty continuously. That capability is built incrementally, one metric, one golden set, one alert at a time, and it compounds with every release.
STAY POSTED
Get our next analyses and field notes straight to your inbox.
