Rabobank Transforms Its Payments System With DevOps on IBM Z
When global financial services provider Rabobank decided to renovate its payments system four years ago, it took a broad view of the project. The bank assessed its hardware, its payments processing requirements and the need for end-to-end testing of its processes. The result was a payments system built on IBM Z* using DevOps to provide process flexibility. Rabobank has designed an architectural landscape that’s reaping big benefits.
Netherlands-based Rabobank is at the forefront of using DevOps on IBM Z. The bank uses DevOps to support its payments system in a more agile and automated way. A key component of this initiative was enabling DevOps on IBM Z to automate Rabobank’s end-to-end tests.
Rabobank had compelling reasons to embrace DevOps, according to Ralph van Beek, the bank’s DevOps architect for the payments domain. The bank’s outdated application landscape for payments needed renovation. Mobile banking, the API economy and the entrance of fintech companies in the financial market pressured Rabobank to become more flexible with its applications and front-end functionalities.
Revamping Payments
Payments processing is the core of the bank and responsible for account statements, balances and payment validation. It combines systems of record with transaction processing. Scalability and reliability of the bank’s systems are paramount to keeping Rabobank’s current account system up and running as roughly a third of the Dutch economy depends on the bank.
This need for reliability is underscored on the last Saturday in November each year—when the majority of the Dutch shop for Santa Claus presents. This day is equivalent to Black Friday in the United States. During this busy time, Rabobank processes several thousand transactions per second. Each transaction involves dozens of I/O transactions, including checking the customer’s bank account balance and validating the customer’s bank card. If a mainframe application crashes in one of Rabobank’s data centers during account processing, the bank must switch to another data center without losing a single transaction.
To keep up with growing demand, the bank needed to replace its entire payments applications landscape. The previous landscape was a combination of systems that were several decades old.
Rabobank envisioned how its future landscape would look and function. It wanted to be able to maintain performance, scalability, stability and data consistency. After extensively investigating the payment platforms used by other financial organizations, Rabobank concluded that the IBM Z platform remained the best choice for its transaction processing needs.
As a result, the bank committed to large-scale greenfield applications built on IBM Z with state-of-the-art technology. The system would be web service-based and secured by IBM DataPower. Rabobank chose a DevOps approach to foster agility, stability and collaboration.
The bank saw a great opportunity to make choices about which workloads might remain on or be migrated to IBM Z. To remain on Z, workloads had to meet four criteria: high availability, high consistency, high security and high scalability. “If the answers to all four questions are yes, then the workload is allowed to run on Z,” van Beek explains.
Even though the bank is forging ahead with revamping systems, it relies on COBOL for many applications. While Rabobank uses Java and .Net, it uses COBOL for its efficiency in running transactions. It may take more time to develop code in COBOL. However, the bank views it as a necessary investment to get high-performance transaction processing.
While the old payments applications were run with COBOL on HP NonStop and IBM Z, Rabobank now has a hybrid setup. The Current Account System, which makes up the core of payment processing, runs on IBM Z. The bank uses a collection of package-based solutions and Java applications to run the less time-critical and more product-specific tasks. For instance, an ACI Worldwide solution is used for card processing, and IBM Financial Transaction Manager handles internally initiated payment transactions.
DevOps Culture
Rabobank realized that it also had the opportunity to change its approach to development. The need for agility and faster development influenced the bank’s decision to use a DevOps model. “If you’re going to build 12 million lines of code, then you need to accelerate your development,” notes van Beek.
The desire to keep up with market demand for mobile and website functionality was another reason Rabobank chose DevOps. “We can’t allow ourselves to slow down our product innovation because our back end isn’t able to match the speed of our front end,” he says. That goal encouraged IT to be more agile and use smaller-scale development releases.
DevOps is a model that Rabobank has been implementing for some time. Several years ago, the bank created its “One Team” initiative to bring development and operations personnel together under the same governance and at the same location.
Next, the bank started to apply DevOps processes and development technology to integrate and automate its work processes. Rabobank’s “Radical Automation” initiative focuses on the automation of its entire development process and looks at IT from a business viewpoint. The initiative also searches for process waste points and ways that IT can accelerate development and testing.
For instance, handover points from developer to an operations engineer in the past were a source of waste because they involved steps being redone. When integration and automation were brought into the IT process, handover points were eliminated. “That’s when you start to accelerate processes and make them more reliable and efficient,” notes van Beek.
Despite these achievements, cultural change at the bank hasn’t been easy. A number of people were initially skeptical about the changes, especially regarding automation or moving away from the green screens. The bank started with the teams that were eager for change. When these teams discovered that DevOps made their jobs easier and less time consuming, the skeptics took note. This made it simpler to get everyone on board with the DevOps approach.
The Need for End-to-End Testing
Even as automation increased, the bank’s complex payment processes made it necessary to continue end-to-end testing. The bank sought ways to speed up testing while maintaining quality and accuracy.
A Java developer might contend end-to-end testing isn’t necessary when the interfaces are designed well, thoroughly tested and all of the developers adhere to the conventions. While that may be true for developing a web app, banks have different requirements.
Back-end processing, especially for banking applications, is the point where all the process flows come together. This convergence brings a much higher level of complexity and error proneness. On top of that, failure of back-end banking applications has a much greater impact.
For instance, Rabobank’s payment processes serve dozens of distribution channels, which handle hundreds of product varieties. Such complexity creates room for human error, which can seriously impact the Dutch economy.
Rabobank’s systems were redesigned to be more service-oriented and modular. However, strong non-functional requirements such as performance and consistency need an integrated architecture. The new architecture reduced the end-to-end testing to some extent, but Rabobank’s need for these tests will never disappear.
“We’re able to considerably reduce the number of end-to-end tests, but they will never go away,” notes van Beek. “Given that end-to-end tests are here to stay, we’ve chosen to completely embrace them and fully invest in automating them,” he says.
Rabobank is using four components to automate end-to-end tests:
- Automated service virtualization (stubbing). Automated stubbing allows the bank to reduce the size of the end-to-end test by virtualizing the rest of the environment.
- Automated delivery of the test data. The bank requires a highly automated master functionality to create end-to-end test data that’s both highly secure and high quality. This automation step also fits in nicely with the new European Union (EU) General Data Protection Regulation (GDPR). GDPR has strict requirements for anyone hosting or processing data of EU citizens.
- Automation of deployments. This automation must be done in an integrated manner, which means deploying a consistent chain of applications together with their end-to-end relationships through web services and IBM MQ connections.
- Automated execution of end-to-end tests. This step includes automated analysis of the test outcomes.
This four-part approach has worked well for Rabobank. “You’re better off automating end-to-end testing, and these four parts are crucial to achieve results,” says van Beek.
Partnering With IBM
It takes good partnership to help produce change. Rabobank’s active partnership with IBM has enabled the bank to explore and provide input into many IBM technologies.
Rabobank is piloting IBM IGNITE, which uses artificial intelligence (AI) to accurately identify, classify and allocate defects. The bank believes AI can be used to generate more reliable test cases.
The bank has also partnered with IBM to enhance Rational* Integration Tester for performance and stress tests. “The results are impressive,” van Beek notes. Rabobank is performing 200 million service virtualization calls each month for load and stress testing purposes—a volume that speaks to the tool’s scalability.
Thanks to a recent IBM Design Workshop, Rabobank has identified a series of Rational Team Concert Enterprise Edition and UrbanCode* product enhancements to get real DevOps continuous integration to the mainframe. The idea is to reduce the number of home-breed add-ons to the product and enhance the out-of-the-box support for IBM Z. Similar improvement initiatives are underway to expand the InfoSphere* Information Server and Rational Integration Tester.
Business Benefits
Rabobank is gaining many benefits by automating its DevOps approach on IBM Z. Quality issues are identified much earlier in the process. Delays due to deployment issues have been reduced.
The four-part approach to automation has cut Rabobank’s lead time for an end-to-end test from a month to an average of four hours. The bank runs 50 to 60 end-to-end performance tests each month using a single production-like environment. Performance tests run at 30 percent of production capacity, giving the bank a sufficiently reliable view of how the system would run in production. “We’ve been doing this for 18 months and it’s proven to yield very relevant test results,” van Beek says.
The bank has also gained the ability to test only those applications affected by a change. For example, if there’s a new product that will affect just two applications out of 12, the bank can now limit the load and stress test to those two applications. This saves Rabobank time, as the scope of the test is smaller and there’s less time spent in troubleshooting.
Success doesn’t just happen; it’s planned. Rabobank’s achievements are due to a very strong vision, a belief in that vision and steadfast support from senior management. This kind of change takes years. The bank began the process in 2014 and work continues today. “It’s a process of continuous improvement that has become part of our way of working,” van Beek says.
The DevOps initiative at Rabobank greatly benefited from the decision to build a new system, which gave the bank a new application landscape. The decision validated the budget needed to automate processes.
Looking ahead, Rabobank expects to finish onboarding the new IBM Z development teams by the end of this year. The focus will then move to extending functionality of the bank’s deployment and test tools. The bank is moving toward the cloud and the benefits it will provide.
DevOps is an approach and an attitude. Rabobank’s experience shows that very complex operations running on IBM Z can and will benefit from a DevOps focus.