Got IBM i Data? Simplify Data Replication, Migration, Data prep and Data Motioning with IBM’s DataMigrator for IBM i

DB2 Web Query DataMigrator ETL (Extract, Transformation, Load) Extension supports the creation, population, and maintenance of database tables from one or more data sources and targets. The low-cost, IBM i centric ETL tool can be used for motioning of data for many purposes:

• Consolidate, cleanse, and optimize the data for analytics into a data mart or data warehouse infrastructure (with the added benefit of isolating large, complex queries from your production system).
• Read data from database logs (Journal Receivers) for near real time replication/transformation.
• Automate feeding of data real-time from Db2 for i into Cloud based databases or servers.
• One time, or ongoing dual maintenance of synchronizing data across databases during a data migration project.
• Manage file/table changes to stage an altered (think modernized) file/table for near instant deployment with minimal disruption.
• Automate the consolidation of data from multiple servers, including data from non Db2 sources such as Microsoft SQL Server, Oracle, Postgres, et.al.
• Automate enhancing your data with demographics, geospatial, or weather data from the cloud to make more informed business intelligence decisions.
• Manage the data preparation and feeding of data into Advanced machine learning analytical tools such as IBM’s Watson Studio or Python based ML models.
• Integrate with open-source solutions such as Apache Kafka to facilitate motioning data into almost any target (AWS, Azure, et.al.).
• Create a repeatable process to securely FTP data into a targeted server.
• Schedule “flows” based on your IBM i job scheduler or create event driven initiation of a data flow.
•  Leverage existing data preparation/transformation programs you might already have, for instance, you can integrate an (externalized) RPG program or SQL Stored Procedure to reflect the source data in a DataMigrator “flow.”
• Supports the Db2 family and non-Db2 relational databases, along with Apache Kafka, a popular open-source integration product.
• Scrape Journal Receiver data for simplified viewing or auditing purposes.

How it works

DataMigrator includes the ability to perform both bulk loads and incremental updates of data, making it ideal for maintaining a reporting database such as a data mart or data warehouse, or pushing data into some Cloud service or database. “Data flows” can be created that define where the data comes from, how it should be transformed, and into what files the resulting data should be loaded. Once defined, data flows can be run immediately or scheduled to run once or on a recurring basis. As many data flows can be created as necessary to support an environment. Multiple data flows can optionally be controlled by one or more “process flows,” which orchestrate when and how data flows run including ability to set up multiple parallel streams).

Also  known as, its all about the T (Transformation) in ETL. The true value of a tool such as DataMigrator is in its ability to take transactional data and untangle it into meaningful information in a format that is consumable on the target. It is not unusual for an analytics or business intelligence type project to require as much as 70-80% of time in just getting the data right. DataMigrator (and the entire Db2 Web Query for i suite of products) is designed with that in mind. The meta data abstraction layer provides a simplified view of complex data structures to the data consumers. Graphical interfaces to define data (and process) flows simplifies the design of abstraction layer including transformation rules, creating of derived fields, consolidation of data, data type conversions (turn a packed 8 decimal DATE field into a true DATE data type), filtering of rows, and aggregating the data.

DataMigrator is an IBM i Licensed Program Product, part of the Db2 Web Query family of products that provide an IBM i centric foundation for analytics and data motioning.