Putting vibe coding to the production-quality test
The idea of letting an AI write production-quality code sparks both fascination and doubt. Some see the promise of near-instant productivity, code in a single prompt, while others fear unleashing legions of barely readable, unmaintainable scripts into their codebases. As practitioners who have spent countless hours refining standards for "good" code, we approached this debate with a mix of curiosity and caution, and a simple hypothesis to test: can an AI build a non-trivial application from scratch, without a single line written by us, and produce something humans can maintain?
To make the test concrete, we chose to build a System Update Planner: a tool for managing software updates and patch rollouts across a fleet of devices. It lets users define which packages need updating, schedule staged rollouts in batches, and track execution status down to individual devices. Neither "Hello World" nor a massive enterprise system: just enough versioning, state management, and real-time tracking to realistically assess the quality of generated code.
We ran three experiments with very different expectations. In the first, a free-improvisation style, what Andrej Karpathy popularized as vibe coding, focused on functionality, expressing nothing about the expected structure. In the others, we were far more deliberate: prescribed design heuristics, modularity and testability requirements, and quality reinforced through continuous feedback. This contrast sheds light on the role of human intent and technical discipline in AI-created systems.
Defining production quality: criteria and measurable signals
"Production quality" is not a universally defined term, and if a single definition existed, many real-world codebases would struggle to meet it. In practice, the term reflects a combination of technical judgment, organizational context, and hands-on experience. We do not use it to mean perfection, but software that is robust, maintainable, and ready to be deployed and evolved responsibly under real-world conditions.
Six criteria, from qualitative signals to quantitative thresholds
Our grid combines six criteria, each paired with observable signals and numeric benchmarks. Behavior: key workflows are verified by fast automated tests, with no regressions in basic usage, pass rate near 100%, mutation score above 80%. Testability: the design supports targeted, isolated unit, integration, and end-to-end tests, unit coverage above 90%, no flaky tests. Readability: modular, idiomatic code that others can safely modify, with low cognitive complexity. Non-functional concerns: performance, security, and operational robustness anticipated, without excessive coupling. Diagnosability: structured, contextualized logs, traceable failures, alerts on critical paths. Finally, engineering practices: version control, CI, static analysis, frequent commits, and clean lint runs.
This is not a checklist for perfection, but it is roughly the benchmark we have in mind when we ask: can vibe coding produce production-quality software?
Tools and setup: agentic IDEs, frontier models, and MCP servers
We primarily used Cursor in agent mode, which gives the AI the ability to run command-line programs, interact with Git, and leverage tools exposed via Model Context Protocol (MCP) servers, Sequential Thinking, Git, GitHub, and Memory. This agentic mode, since generalized across the whole ecosystem (Cursor's cloud agents, Anthropic's Claude Code, and their equivalents at JetBrains and Windsurf), lets the AI explore the codebase, read documentation, run commands, and reason across multiple steps.
On the model side, we started with Claude Sonnet for its reasoning capabilities, then adopted Gemini Pro almost exclusively, impressed by its understanding of functional requirements and its engagement in architectural discussions. Generations succeed one another quickly, Claude Opus 4.8 and Sonnet 5 at Anthropic, Gemini 3.1 Pro at Google as we update this article, but our observations are less about a specific model than about how to guide it: the conclusions hold from one generation to the next.
Three experiments, three levels of constraint
Experiment 1: a functional overview, then total freedom, the AI built the application autonomously, mostly in JavaScript. Experiment 2: the same functional requirements, plus strict implementation rules such as test-driven development, the AI chose TypeScript and Prisma as the ORM. Experiment 3: MCP servers disabled and a conversational style close to human collaboration, encouraging the AI to ask clarifying questions, the final implementation was in Python. To simulate a brownfield context, we started new chat sessions: the AI would "forget" it had created the codebase, and the Memory MCP server did not notably mitigate this behavior.
Experiment 1: pure vibe coding, fast but fragile
In our first experiment, the approach was entirely hands-off: a high-level functional description, then the AI at the wheel. The initial result was impressive, a nearly functional application generated in a single pass, demonstrating the models' ability to quickly translate specifications into code.
The limits appeared as soon as the system needed to evolve. Adjusting output formats, introducing a menu-based interactive mode, integrating a database migration tool, adding tests: each incremental change caused regressions and required manual intervention. This is now a widely documented finding: without structure, generated code ages badly, and technical debt accumulates at generation speed, large-scale code quality analyses measure rising duplication and code rewritten shortly after creation in heavily AI-assisted repositories.
Still, we must acknowledge what this approach changes: it radically lowers barriers to entry. Less technical profiles prototype ideas autonomously, and teams accelerate their proofs of concept. The ability to quickly produce functional software is a real advance, as long as it is not confused with the ability to produce durable software.
Experiment 2: imposed discipline, TDD, TypeScript, and mutation testing
Our second experiment inverted the logic: technical discipline from the outset. Explicit expectations, TDD, small incremental changes, regular commits, modularity, and type safety, which led the AI to TypeScript and Prisma. Early results were promising: clean domain models, scaffolded unit tests, a trunk-based workflow held with reasonable consistency. We added coverage thresholds and Stryker for mutation testing; some parts of the codebase reached mutation coverage close to 100%.
Despite repeated guidance, the AI would still regress to its old habits, writing production code first, adapting tests afterward. This behavior, still observable with today's models, most likely reflects a training bias: rigorously test-first codebases remain underrepresented in training data. The imbalance showed in our commit history, where test-line growth trailed production growth until we enforced mutation checks.
The Prisma incident: when a small change goes off the rails
One incident sums up the risk. After a clean test run, we asked for the removal of a few leftover console.log statements. A minor change, in theory. What followed was a cascade of breaking changes, test regressions and, ultimately, a spontaneous, unjustified downgrade of the Prisma dependency by two major versions. That the AI could trace the problem back to the version mismatch was impressive; that it introduced the change without cause or warning was concerning. The lesson fits in one sentence: the more complex the change, the more essential the guardrails, static analysis, test enforcement, mutation thresholds, human review, are to reliability.
Experiment 3: conversational collaboration, as with a human peer
For the third experiment, we changed strategy: no more MCP servers, a single model, and rich architectural conversations, exactly as with a human collaborator asked to critically evaluate our assumptions.
When the AI asks the right architecture questions
The most striking moment came while discussing the design of device snapshots. Rather than mechanically implementing an API, the AI raised thoughtful questions on its own: should snapshots capture the full state of installed packages at a given point in time? How should discrepancies between reported and stored states be handled? Should submitting a snapshot overwrite the current state, or record a historical version for audit purposes? It proposed API designs, debated trade-offs, eager versus lazy loading, and exposed the gaps in our original assumptions.
These exchanges amount to genuine architectural foresight, not mechanical generation. Despite some residual inconsistencies, the resulting code, a Python/FastAPI system, was the cleanest, most modular, and most RESTful of the three experiments. It also prefigures what is now called context engineering: the quality of the conversation and the context provided directly determines the quality of the system produced.
Lessons and recommendations for teams in 2026
Rated against our six criteria, the three exercises draw a clear hierarchy: pure vibe coding excels at speed and fails at maintainability; imposed discipline produces the most verifiable code; conversational collaboration produces the best architecture. Three cross-cutting lessons emerge. Tooling choice matters, but how you guide it matters more. Context awareness remains limited: even with MCP servers, every new session required re-explanation. And production quality requires deliberate oversight, testability, maintainability, and diagnosability only improve when they are explicitly demanded and verified.
What each role can do right now
Developers should learn to guide these tools: frame requests with design patterns, testing strategies, and coding principles, and treat the AI as a fast but inexperienced pair programmer. Tech leads and architects should define the guardrails, starter templates, reference repositories, governance policies for assisted development. Testers and security analysts can use AI to explore test scenarios, edge cases, and attack surfaces. IT leaders, finally, must prepare the organization: new capabilities, new risks, and new costs, since agentic IDEs now run on monthly credit systems that frontier models burn through quickly at team scale.
So, can AI-assisted coding produce production-quality software? Not consistently, not yet. But the gap narrows with every model generation. As they progress, a shift is taking shape: smaller, modular systems that fit within a model's context window, regenerated rather than endlessly patched. In that world, maintainability will no longer just mean writing code that lasts, but writing code that is easy to replace. Organizations that invest now in the art of guiding, governing, and integrating these tools will be tomorrow's leaders. Disclaimer: the statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Adservio.
STAY POSTED
Get our next analyses and field notes straight to your inbox.
