Generating test cases with AI: promise versus measured reality
Creating test cases remains a critical, time-consuming and skill-intensive activity. Large language models (LLMs) promise to automate this highly manual work performed by QA engineers, but between the seductive demo and industrial practice, the gap is real, and the true effectiveness of these tools is rarely measured in controlled experimental settings.
That is precisely what we set out to do: a controlled experiment comparing manual test case creation to AI-assisted generation, on real user stories, with metrics defined in advance and a solid human baseline. This article presents the protocol, the quantified results, and what they mean in 2026, now that completion assistants have become autonomous agents capable of executing the tests they write.
The stakes go beyond academic curiosity. In organizations where QA represents a significant share of the delivery cost, industrializing test case generation can transform the economics of a project, provided you know precisely what you gain, what you lose and what needs to be controlled. Without measurement, teams oscillate between naive enthusiasm and rejection on principle, two postures that are equally costly.
Research questions and hypotheses of the QA experiment
The central challenge was to determine whether AI tools can genuinely assist QA professionals in creating comprehensive and accurate test cases, while maintaining quality standards and reducing design time.
Four questions structured the study: can AI-generated test cases achieve accuracy comparable to manual creation? What impact does AI assistance have on creation time and coverage? How does prompt optimization affect the quality of the results? And what are the limitations of AI tools on complex testing scenarios?
These questions are anything but rhetorical: most published feedback on AI test generation rests on impressions or hand-picked demonstrations, rarely on a direct comparison with the work of seasoned QA engineers on the same requirements. Establishing this human baseline, with shared metrics, was the precondition for producing figures that can be debated, in the good sense of the word.
Our starting hypothesis
We hypothesized that assisted generation would bring measurable improvements across several dimensions: a reduction in test case creation time, better coverage, particularly of edge cases and complex scenarios, structural consistency superior to manual creation, and the need for iterative prompt refinement to reach optimal results.
Protocol: nine user stories, three QA engineers, two AI tools
Three experienced QA engineers, each with three to five years of professional testing practice, took part in the study. All were familiar with manual test case creation and standard methodologies. The tools compared were GitHub Copilot as the primary tool and ChatGPT as the secondary comparison, the dominant assistants at the time of the experiment.
A standardized user story format and test template
To guarantee reproducibility and consistency of results, every user story followed a structured format: concise title, detailed description of requirements, explicit acceptance criteria, edge cases and constraints. Every test case followed a common template, title and scenario, preconditions, steps, expected results, environment specification, identical for the human baseline and for the AI, in order to normalize evaluation between the two approaches.
Five phases of controlled comparison
The methodology chained five phases: manual baseline creation on nine user stories of varying complexity (simple, medium, hard); AI generation on the same stories; comparative analysis on eight defined metrics, including accuracy measured as the ratio of correct cases to the manual total, the redundancy rate and the time saved relative to manual creation; iterative prompt optimization based on the initial results; then a full final evaluation with the optimized prompts.
Results: 80% time saved, 27% residual ambiguity
Prompt refinement proved decisive. The initial prompt, "Generate test cases for this user story",produced fast but incomplete results. The optimized prompt, "Acting as a senior QA engineer, generate comprehensive test cases following the specified template; include functional, boundary-value, error-handling and edge case scenarios; use the Given/When/Then format where appropriate",improved key metrics by an average of 67.78%.
With the optimized prompts, the final scores across the nine user stories speak for themselves: average time efficiency of 80.07%, structural consistency of 96.11% (4.22/5), and coverage ranging from 50% to 99% depending on scenario complexity, with some scores even exceeding the reference when the AI generated more comprehensive cases than the manual baseline.
The tool comparison also delivered a lesson: ChatGPT outperformed GitHub Copilot on scenarios requiring detailed prompt interpretation. Tool choice is therefore not neutral: it must be evaluated according to the context, the types of tests targeted and the way teams formulate their requirements.
Strengths and limitations observed
On the strengths side: initial drafts produced in seconds rather than hours, high structural consistency across cases, effective coverage of explicit functional requirements and minimal redundancy on simple scenarios. On the limitations side: heavy dependence on input quality and specificity, difficulty interpreting complex business logic, advanced testing techniques rarely applied spontaneously, non-functional coverage absent without explicit instruction, and above all 27.22% of ambiguous cases requiring human clarification before being usable.
What 2026 changed: from the prompt to the autonomous testing agent
Since this experiment, the landscape has shifted. Completion assistants have given way to coding agents, Claude Code, Cursor, GitHub Copilot in agent mode, that no longer merely generate test cases from a user story: they read the repository's code and specifications, write the tests, run them, analyze failures and fix them in a loop until the suite passes.
Separating the agent that codes from the agent that tests
The practice taking hold in 2026 is to run a dedicated testing agent on every pull request, independent of the agent that wrote the code. This is separation of concerns applied to AI: generation optimizes for the plausibility of the code, testing optimizes for finding failure conditions, asking an agent to validate its own output amounts to letting the author grade their own paper. Through the Model Context Protocol (MCP), these testing agents also access browsers, databases and real execution environments.
Our 2025 results nonetheless remain relevant on two essential points: the quality of the framing, yesterday the prompt, today the specification and context given to the agent, remains the primary quality factor of the tests produced, and the residual ambiguity rate still demands human review before production integration.
The study's limitations must also be kept in mind: only nine user stories, specific tools and versions, a degree of subjectivity in the qualitative evaluation and no measurement of long-term impact on productivity. These caveats do not invalidate the trends observed, but they invite teams to replicate the measurement in their own context rather than transposing the figures as they are.
Human oversight, the non-negotiable condition of quality
Despite good performance on basic scenarios, more than a quarter of the generated cases in our study showed ambiguity. QA teams must therefore institutionalize structured review steps: manual validation against acceptance criteria, flagging of unclear logic, explicit verification of edge case coverage, before adopting any generated tests into production workflows.
This requirement aligns naturally with shift-left: the earlier generated tests are reviewed in the cycle, the less the ambiguities cost to fix. The tester's role evolves accordingly, toward context engineering, the design of genuinely testable acceptance criteria and the critical evaluation of what the agent produces, skills that gain value with AI, far from disappearing because of it. It is also a compliance matter: in regulated sectors, the traceability of testing decisions remains an audit requirement that AI does not make disappear.
Overseeing does not mean rereading everything line by line: mature teams equip the review with automated evaluations, mutation testing to verify that the tests actually catch regressions, coverage metrics per requirement, sampling of generated cases for human audit. The goal is to concentrate human attention where ambiguity is most likely.
AI in QA: a productivity lever under human direction
This experimental study suggests that AI tools clearly improve the efficiency and structure of test case creation in QA workflows, particularly for initial drafts: 80.07% average time savings and 96.11% structural consistency. For the industrial organizations supported by Adservio, these gains represent a significant lever for optimizing quality processes.
The return on investment, however, plays out over time: speed gains are only worth it if the maintenance of the generated suites remains under control. A test suite bloated with redundant or brittle cases costs more than it delivers; review discipline and traceability between requirements and test cases remain the best defenses against this silent inflation.
But these benefits are neither immediate nor automatic. They depend on investment in prompt and specification design, our optimization process alone generated a 67.78% average improvement, and on rigorous validation maintained over time. The conclusion fits in one sentence: LLMs are most effective as assistants, and now as agents, that accelerate test creation under human direction; they augment testers, they do not replace them.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Adservio.
STAY POSTED
Get our next analyses and field notes straight to your inbox.
