Enhance AI Trust on IBM i With the 7 Levels of Test Automation

Pascal Polverini, founder of Polverini & Partners and co-author of a new IBM i modernization Redbook, explains the significance of independent, deterministic validation

Generating code is no longer the bottleneck on IBM i. Trusting it is. An AI assistant can refactor a 20-year-old RPG program in minutes, but nobody can tell you in minutes whether order entry, invoicing and month-end close still behave exactly as they did yesterday.

That gap was already expensive before AI. Developers spend roughly 30% to 50% of their time on testing, and the new IBM Redbook “Modernizing IBM i Applications” estimates that up to 88% of that effort can be automated. AI widens the gap: Code now arrives faster than any team can check it by hand.

So it is no surprise that “deterministic” validation and “AI-ready” testing have become common phrases. What matters is what actually gets automated. Fortunately, IBM i now has a public yardstick for exactly that.

The Yardstick: Three Laws and Seven Levels

The testing chapter of the Redbook starts with a definition. A test must obey three laws, and anything that breaks one of them is not a formal test:

• Autonomous: Isolated from outside events that could change the result.
• Reproducible: Run from identical initial conditions with the same sequence of events.
• Observable: Able to prove the final state matches the expected state.

This is what “deterministic” really means. Same start state, same inputs, same verdict, every time. A process that cannot reset data or compare database end states can use the word, but it cannot deliver it.

The Redbook then defines seven levels of automation.

Level	What is automated	Why it matters on IBM i
1	Replay with data initialization and comparison	Delivers the Reproducible and Observable laws
2	Script generation from user or system activity, or from prompts	Nobody hand-writes test scripts
3	Batch running of interactive processes	5250 and web scenarios run unattended in CI/CD
4	Data subset extraction	Tests run on only the records they need, in minutes not hours
5	Code coverage reporting and merging	Proves which lines ran, across every variation of a scenario
6	Security attribute integration during replay	Ownership and authorities survive a restore, so security is tested too
7	Workload with differentiated record flow	Parallel scenarios on different records, without lock conflicts

The Redbook adds two more ladders: operability across jobs and platforms, and AI integration, which runs from tests feeding AI, to AI guiding tests, to autonomous testing, where an AI agent calls the test platform itself through model context protocol (MCP).

Where Automation Usually Stops

The Redbook is diplomatic: “most tools address only parts of the process.” In my experience, the upper levels are where most automation efforts stall.

The reason is architectural, not cosmetic. Levels one to three can be reached from the outside: Capture a screen flow, replay it, diff the output. Levels four to seven have to work inside the system. They must know which records a scenario touched, trace which lines of RPG actually ran, restore object authorities after a data reset and route parallel jobs to different records.

The practical rule is simple: Ask for a demonstration level by level, on your own application. The Redbook recommends exactly that kind of proof of concept.

Beyond Coverage: The Exact Logical Flow

Level five asks which lines ran. For AI-assisted modernization, that is not enough: You also need to know in what order they ran and how long each one took. Coverage data with a count or a timestamp for every executed line gives the exact path through programs, modules and procedures.

This is gold for AI: It restores the exact business flow of a dynamic process. Static analysis shows an LLM the code that exists; timestamped coverage shows the code that actually ran for a real transaction, in order. That is the context an AI assistant needs before it rewrites legacy logic, and afterwards a deterministic check that the new program follows the same business path.

No AI Required: The Business Case for Legacy Testing Alone

AI makes test automation urgent, but it is not what makes it pay. Automating regression testing on the legacy code you already run can pay for itself within a year, even for a shop with a single developer.

The arithmetic is simple. Applying the Redbook’s figures at the low end (30% of time, 88% automated) to a typical 1,600 productive hours gives back about 420 hours per developer every year.

Team size	Testing hours freed per year (30% × 88%)	Equivalent
1 developer	about 420	10 working weeks
5 developers	about 2,110	1.3 full-time people
10 developers	about 4,220	2.6 full-time people

Those are only the hours. Automated regression also catches defects before production, when they are cheapest to fix, and cuts the downtime and audit effort that follow a bad release.

The savings grow with every developer, while the tests are built once. Every OS upgrade or PTF cycle then reuses them.

The Top Rung: An Independent Gate

The highest AI integration level in the Redbook is autonomous testing: The AI agent calls the test platform itself. That raises a governance problem, not just a technical one.

An LLM is probabilistic. It answers “probably” or “almost certainly.” A business needs a yes or a no before code reaches production. And asking the same AI to judge its own output is like letting a butcher grade his own meat. Unit tests the AI wrote alongside the code share its blind spots.

The answer is an independent, deterministic control layer outside the model, which agents can call through MCP. The loop works like this:

1. The AI agent modernizes or generates a program, and it compiles.
2. The agent calls the gate, which replays recorded real-world scenarios against the new code and compares the outcomes with legacy behavior or target specifications.
3. The gate returns YES or NO.
4. On NO, structured feedback goes back to the agent: the failed comparisons plus contextual logs, workflows, screen flows and the execution flow, so the agent can correct and retry until the gate passes.
5. On YES, the program is production-ready and the agent moves on, leaving auditable evidence behind.

This makes testing a governance artifact. The EU’s DORA and AI Act require traceability, human oversight and operational resilience; in the US, Sarbanes-Oxley demands that every change to a financial system is tested, approved and documented. A gate records who changed what, what was tested and why it was authorized.

Seven Questions for Your Next Proof of Concept

Use these on your own application:

1. Reset the database, replay and compare every changed table.
2. Generate a test script from a recorded 5250 or web session, not from a template.
3. Run that interactive scenario in batch, with no one at the keyboard.
4. Build a test data set containing only the records the scenario touched.
5. Merge code coverage across three variations of a scenario, then show the order in which lines ran.
6. Restore the data, then prove object authorities are still what production uses.
7. Run 10 copies in parallel, each on different records, without lock waits.

Then one more: Can an AI agent call the test platform and get a deterministic yes or no it cannot argue with?

Automation Is the Mechanism of Trust

AI will keep making code cheaper to write. The value moves to proving it works. On IBM i, where one program can carry decades of business rules, that proof has to be autonomous, reproducible and observable, at every level of the ladder.

The Redbook gives the industry a shared yardstick. Use it. When anyone says “deterministic,” ask which of the seven levels are automated, and ask to see it on your own system.

---