Black-box reverse engineering

SECTION 1

Introduction

It's a scenario that sends a chill down every CTO's spine: a business-critical application is running, but the source code is gone. Maybe a vendor relationship has soured, leaving you with a functional but opaque binary. Or imagine you do have access to the source code, but it's so messy that both humans and AI struggle to understand and describe what the codebase actually does.

Generally, one of our approaches to accelerating legacy system modernization with AI is to use it to speed up the reverse-engineering part first, feeding the AI the existing code, then letting it help us produce a complete description of the application's functionality, which can be used for forward engineering.

But what happens if we don't have the code, or it's so messy it's unusable? How can generative AI accelerate reverse engineering in that case?

Recently, we explored this question at Adservio: the result was a "black-box reverse engineering" experiment. We wanted to find out whether, by combining AI-driven navigation with data-capture techniques, we could produce a detailed functional specification of an existing system and use it to build a modern replacement from scratch.

This is the story of how we did it, the obstacles we ran into, and the powerful lessons we learned about the future of legacy system modernization.

SECTION 2

Setting up our AI experiment

As our test subject, we chose Odoo, an open-source ERP platform that's easy to set up and run on our machines. The application's back-end code was strictly off-limits to our AI setup, our sources of information were limited to what could be observed from the outside:

The user interface. The AI could see and interact with the application exactly as a user would, observing static elements, dynamic behaviors, and click paths. The database. While the back-end code was hidden, the schema and existing data were allowed. Network traffic. We also let the AI inspect the network requests flowing from the browser to the back-end server as an additional data source.

The scope of our experiment was to:

Reverse-engineer the functionality of a single, foundational user journey: creating a new opportunity in the Odoo CRM's sales pipeline. The deliverable for this step was a specification document describing all available information about the application's behavior, including screenshots. Explore the new opportunities generative AI offers for building a "functional parity test" that can run against both the old and new applications.

SECTION 3

Experimental iteration one: reconnaissance

Our first attempt was a broad exploration.

### AI as a user

We sent an AI agent with access to the Playwright MCP server to navigate the application. The prompt asked it to "discover the user journey for creating a new opportunity in the sales pipeline." The agent clicked through the interface, took screenshots, and generated a description of every dialog with its fields and the dynamic behaviors it could observe (for example, "when an entry is chosen from this dropdown, the form fields are automatically populated"). We also asked for a flowchart describing the click path it found. The flow it found was a very linear "happy path", almost suspiciously simple. No real-world application is that straightforward; we knew there was more work to do to get the AI to uncover the more complex reality of the paths.

### Change data capture

After getting an initial description of the paths, we asked the AI to navigate again and check the database changes after each click. This was made possible by setting up database triggers that logged every INSERT and UPDATE operation on the relevant tables. We built a small custom MCP server that let the AI agent query this audit log after each interaction. As a result, our functional specification was enriched with the database operations that occurred after each click step. This change data capture (CDC) approach proved to be a powerful and low-cost way to dissect queries from the outside.

@cite:le-protocole-model-context-au-dela-de-la-tendance

SECTION 4

Experimental iteration two: refining the specification

### Running the agent multiple times to converge on a reliable flow

With the insights from our first attempt, we improved the workflow and prompts. First, we refined our click-path discovery prompt, asking it more explicitly to find and traverse every possible user action on each screen. And instead of a single run, we ran the agent multiple times, since we assumed the AI would struggle to find every path branch in one go. We were right: each run uncovered slightly different branches and variations.

We then used another AI step to "converge" these different versions, analyze and reconcile the discrepancies, and produce a single consolidated map of the user flow. The resulting diagrams were far more representative of the application's actual behavior.

### Attempting to reverse-engineer network traffic

In this iteration, we also attempted to add another layer of data collection by capturing network traffic. The idea was to reverse-engineer the back-end API by analyzing the JSON payloads passing between the front end and the back end.

While we did eventually manage to generate an API Swagger document, this setup was far harder to work with than change data capture.

We also concluded that, for this specific scenario, where the API is an internal implementation detail, not a public contract, the back-end API information really didn't give us anything new that we didn't already know from the data structures.

### Validation through reconstruction

The ultimate test was to use our AI-generated artifacts to build a new application. Our goal in this experiment wasn't forward engineering, so we didn't want to spend too much time on this step.

Still, we needed to verify whether the specification was actually useful, so we decided to feed it into one of the rapid AI application-generation tools.

First, we fed our detailed specification, the collection of Markdown files, screenshots, and database logic descriptions, into a series of prompts that generated epics, user stories, and an implementation order. We then fed the first set of stories into Replit.

Here are a few observations from this validation phase:

Prototyping as a validator. Watching the AI build the prototype was a very effective way to validate and improve the specification. When the prototype diverged from the original, it was sometimes easier to spot a flaw or an ambiguity in our specification document than it was when reading the specification itself. The AI struggles with planning. LLMs often struggle to create a logical, incremental implementation plan; this was the case here too. Its initial instinct was to build the entire complex pipeline view all at once, rather than starting with the simplest pieces and building up progressively. Good AI workflow design and human oversight matter for guiding the build strategy in a real-world scenario. No demonstration of query fidelity. Unfortunately, Replit ignored our inputs about the database schema and the exact queries we wanted to run. Since we didn't want to spend too much time on forward engineering, this meant we couldn't fully demonstrate that the new application could be pointed at the existing database. However, generative AI is generally very good at retrieving specific examples and schemas and using them in code; it's not a big leap to think the specific database queries we collected could be reproduced by a coding agent with dedicated prompts.

SECTION 5

Can AI help test for parity?

While the generated application wasn't perfect, it gave us the opportunity to run a second experiment: how could we create an automated test suite that could run against both the old application and our slightly different new prototype?

The challenge is that, while we expect the same general user controls, the new application might look slightly different, and will certainly have different DOM selectors. AI offers some opportunities here to make UI instructions vaguer, which is an advantage in this use case. It can use visuals as information and react to vaguer descriptions of UI elements. For example, you might say something like "look for a + icon," and it could work whether that's implemented as an image icon or a text representation.

We used a framework that lets you add natural-language test instructions to a Playwright test. Instead of writing a selector like find('button#id-123'), we could instead write instructions like await ai('Click the + icon to create a new opportunity'). This way, the test wasn't tied to a specific DOM structure, meaning it was high-level enough to run against two different implementations of the same interface.

When we built a test suite that ran against both applications, we mostly ran into the usual challenges of end-to-end browser testing, like adding wait times. Layering AI non-determinism on top of the well-known flakiness of these tests creates a new set of challenges. This is a promising use case, but one that requires careful implementation to avoid generating more noise than signal.

Prerequisites and limitations, This approach relies on having a safe, isolated test environment available where AI agents can explore freely and where data-capture scripts can run without impacting live operations.

Depending on the type and complexity of the application, we'd expect the resulting reverse-engineered specification document to have varying levels of gaps that would require subject-matter-expert input, for example, if there's more complex logic that isn't observable from a user interface. We also didn't explore techniques for landscapes involving service-to-service communication or other back-end side effects. Deeper network data capture could be useful there.

SECTION 6

Conclusions and key takeaways

Our experiment showed us there's real potential in using AI to reverse-engineer an application without access to its source code.

Here are three key takeaways:

AI is an accelerator, not an automator. A recurring theme across all our experiments is that, while AI is a powerful force multiplier, it doesn't replace human expertise. It handles the first pass, generates baseline documentation, and frees up subject-matter experts to focus on refinement and validation. The process is a human-AI collaboration. Iterate and decompose. Breaking the application down into user journeys, and the reverse-engineering process into small iterative steps (discover, converge, add detail), helps produce higher-fidelity results. Validate continuously. Using techniques like rapid prototyping and parity testing lets you build early feedback loops.

@cite:de-la-boite-noire-au-plan-technique-comment-nous-avons

Disclaimer: the statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Adservio.

FAQ

Frequently asked questions