Overcoming the Semantic Gap in Mainframe-to-AWS Schema Mapping
Connecting enterprise data to cloud platforms is the first step—understanding what it means is the next challenge, Tim Bond of Adeptia and Rohan Gupta of R Systems explain
It can be tempting for enterprises to assume that once they connect mainframe data to cloud platforms like AWS, the hardest part of modernization is complete.
With cloud service providers offering tools like Amazon Simple Storage Service (S3) for moving and storing data, migration now presents less of a bottleneck. But there is another challenge that teams may not account for in modernization planning: schema discovery—figuring out what a field actually means and identifying inconsistencies, duplication and missing details in the data.
A field labeled “CUST-BAL-AMT-3” may be usable in an established system, but without context it tells a cloud analytics platform little about what it is or how it should be used.
“The semantic gap between mainframe data and cloud-native schemas is significant and routinely underestimated,” Tim Bond, chief product officer at data automation firm Adeptia, warns. The gap is becoming more significant as more and more data is pushed into AWS to support AI and analytics applications.
An IBM survey of chief data officers found that 78% cite leveraging proprietary data as a top strategic objective to differentiate their organization in the market. But that value depends on correctly mapping and orchestrating how the data flows.
Why Mainframe Data Might Not Translate Automatically
“Migration is engineering; mapping is archaeology,” Rohan Gupta, VP of cloud, security and DevOps at R Systems, an AWS Advanced Consulting Partner, tells TechChannel. “The real challenges are semantic interpretation, reconciliation, lineage and operational orchestration. None are solved by the connectivity layer.”
Before data moves onto AWS, it needs to be validated, cleansed of duplicates, aligned with current business definitions and reconciled with the systems it will integrate with. Cryptic naming conventions, packed decimals, legacy formats and embedded business rules do not automatically carry over to cloud platforms, and misinterpretation can lead to incorrect data analytics and AI outputs.
Bond explains that the semantic gap operates at two levels.
At field level, legacy data is often encoded in formats such as COBOL packed decimal, EBCDIC and fixed-width records, with meaning embedded in copybooks and implied scales. At the conceptual level, different systems may use the same label—such as “customer”—to represent entirely different things, from a billing entity to a user account. Without understanding these distinctions, combining datasets can produce results that appear valid but are fundamentally incorrect, highlighting how easily meaning is lost when data is moved without proper context.
Bond cites the example of a mainframe field called TXN-AMT, stored as a COBOL packed decimal with an implied two-place decimal. The number $1,234.56 sits in storage as the digits 123456, with no decimal point, no scale indicator and no currency context.
“If the extraction layer doesn’t carry the copybook definition forward, the cloud target sees an integer (123456) and loads it as $123,456, a hundred-fold error,” Bond explains. “Multiply that by a million transactions and a downstream AI model is now learning patterns that don’t exist.”
He adds that concept-level mismatches are just as common—for example, a STATUS-CD field where the value “1” means “active” in the customer system and “pending” in the policy system. “Same column name, opposite meaning. Without a shared data dictionary and semantic layer sitting above the physical schemas, you don’t catch these. AI gets confidently wrong answers, and so do your people,” Bond says.
These simple errors mean that schema mapping can unexpectedly slow down modernization projects more than the initial data migration. “Programs often double or triple their mapping timelines not because the team is slow, but because the unknowns were unknowable at planning time,” Gupta says.
There are three main sources of delay. The proprietary data that is so valuable as a competitive differentiator is often undocumented institutional knowledge that exists in the heads of employees and has to be reconstructed. Each mapping decision has downstream consequences, from breaking reports by truncating a customer ID to creating compliance issues by incorrectly translating a regulatory field.
And without formal data contracts between producers and consumers, each mapping integration becomes a bespoke negotiation. “The 10th integration is hard. The hundredth is impossible if every one of them rediscovers what the fields mean from scratch,” Bond says.
Orchestrating Data Flows Across Systems
Mapping data correctly is only part of the challenge. Ensuring that data moves, transforms and arrives consistently requires orchestration. This means managing how data is extracted, standardized into usable formats, checked for accuracy and delivered to downstream applications.
“Once data is flowing into AWS, you’re not done; you’re in operations,” Bond says. Orchestration ties extraction, format translation, schema mapping, validation and delivery into a single observable flow, with lineage captured at every step.”
AWS services like Glue provide the backbone for ETL and pipeline management, with Kinesis enabling real-time data streaming once it is generated.
The value of orchestration is consistency. “Orchestration is what turns a collection of pipelines into a system you can run a business on,” Gupta says. It provides a coordination layer that supports pipelines to handle growing volumes of data without manual intervention, preventing well-mapped data from losing integrity and becoming fragmented.
Orchestration “manages mainframe-to-cloud dependencies, replay, reconciliation and SLA monitoring,” Gupta adds. “Without it, late batches silently break downstream SLAs, CDC streams drift from the mainframe undetected, two consumers compute different totals from the same dataset, and lineage becomes a slide deck instead of a system.”
Can AI Solve the Gaps in Semantic Understanding?
IT teams are increasingly using AI to help tackle the challenge of semantic understanding, particularly in speeding up the task of schema discovery and mapping.
Rather than having senior engineers analyzing data patterns, inferring field types and relationships and drafting transformation logic for cloud-native schemas, teams can use platforms like Amazon Bedrock to leverage large language models (LLMs) for generating code, metadata and data definitions.
As Bond notes, “AI is excellent at pattern recognition, and most schema mapping is variations on patterns that have been solved thousands of times before.”
However, AI augments rather than replaces human understanding. AI can misinterpret context, such as which fields are relevant to specific compliance reporting obligations, unique structures that do not appear in training data and semantics that only certain team members would know.
“Humans still own business-rule interpretation, regulatory classification and accountability. AI can suggest the mapping,” Gupta asserts.
Engineers and data specialists need to confirm that those AI mappings reflect business intent and the way definitions are used in practice.
As Bond points out, “anyone selling fully autonomous integration today is overstating where the technology actually is. … Production data is messier than demos.”
From Lift-And-Shift to Data Reset
While accurately mapping data across systems can pose a challenge, migration should be treated as an opportunity rather than just a transport exercise, Bond argues, to avoid carrying problems into the new environment.
“Done well, the modernization effort produces a cleaner, more consistent data set than the organization has had in years, and every downstream use case (analytics, AI, real-time decisioning) gets the benefit. Done poorly, you lift and shift the mess, and now it’s running on more expensive infrastructure with the same trust problems attached.”