Application Performance and Tuning

As the original author of “CICS Performance Bulletin” published by IBM Dallas System Center, and co-author of the subsequent IBM “CICS TS Performance Guide” in conjunction with IBM Hursley CICS TS Development, performance and tuning has been a big part of my job. I’ve performed more CICS performance studies than I can count, and almost invariably, those studies expanded to IMS DB/DC, Db2, ACF/VTAM, z/OS and other OS software products, plus telecommunication networks.

While hardware and software performance management have evolved and expanded over the past half-century, application performance management—homegrown or vendor-supplied—has advanced slowly. Business solutions rather than technology optimization has been their charter, so it doesn’t really fit with an application programmer’s role. Tools and monitors are mundane, they’re difficult to standardize and the effort is sizable. Application development’s primary objective is to automate business processes and increase employee productivity—a priority that outweighs an application’s consumption of processor and disk resources.

Yet in many cases where response time or job turnaround is unacceptable, the primary cause is application design, program language selection (Java, .NET, COBOL, AIX, C++, application generators, etc.), and programming techniques, which when misused can create significant performance problems. Typically, application construction hasn’t been considered significant in performance degradation, because it’s unfeasible. Application rewrites are prohibitively expensive, compounded by a lack of standards or documentation.

Application tuning differs from hardware or software tuning. Software has tuning parameters that can be varied to optimize performance, and hardware has configuration options like dataset placement, prioritizations or workload managers to modify resource usage. The biggest difference is that application performance is partially dependent on how users vary in numbers and the different ways they utilize transactions. For example, a user may scan an entire massive database looking for a customer during a period of peak activity. Similarly, systems may surge and ebb on short notice based on when users are scheduled. Most systems have nothing to deal with things like this.

Organizational Components

While system performance tuning is often an exercise in modifying parameters, reallocating resources, adjusting priorities and removing optional features (like trace, for example), a company’s infrastructure and philosophy can play an important role in determining how well a company’s applications will perform. A company’s infrastructure and philosophy should focus on:

• Standards. Over time, application staff members develop performance improvements for code they write and support. Once an application is in production, performance bottlenecks and excessive resource consumers reveal themselves and must be addressed. These constraints and solutions must be documented and organized in a readily available standards manual. Many coders bring standards with them when they join; add those to the manual. New performance improvement techniques, necessitated over time, continue to improve the standards manual—the “bible” of programming guidelines.
• Training. All the standards in the world are pointless unless programmers and analysts are trained in what they are and how to use them. Performance techniques, tools and facilities are of little value unless used properly and where needed. Verbal instruction should emphasize their importance, and training exercises are recommended. The standards manual should be the source of guidance.
• Documentation. As programs are written and implemented, they should be thoroughly documented in the code with comments as well as externally regarding performance, purpose, files, etc. Most programmers don’t like documenting code, and most managers discourage it because it takes time, but for someone new, it’s a godsend. As turnover and promotion rearranges staff, documentation becomes invaluable in finding and addressing performance issues.
• Oversight and walkthroughs. The design phase is a marvelous opportunity for senior members to mentor less experienced members, to jointly review flowcharts and compare them to code, to verify inputs produce desired results, and to measure the object’s resource consumption in a helpful way. If not, the program, transaction or unit of work can be examined in detail—an approach that both gathers performance data and allows senior staff members to tutor junior staff in the object’s optimization.

Performance Tuning Techniques

Once a performance concern has been identified, numerous tuning options will usually be available to reduce or eliminate it. While environments and products vary widely, and are implemented differently, here are some available actions:

• Preprocessors are provided with many programming languages that optimize code before compiling it. An Optimization and Tuning Guide is offered for C/C++. Similar manuals exist for other languages.
• Start by understanding bottlenecks and identifying constraints. When performance degradation is having a derogatory effect on end users and business operations, it’s necessary to first determine the cause and location, the nature of the performance impairment and a method to eliminate or curtail it. Knowing this, it often becomes possible to alter or reverse the mechanism, whereupon resolution becomes a matter of implementing the solution.
• Customized, streamlined subroutines can outperform code within application programs. Subroutines optimized for performance—possibly implementing system functions unavailable to application programmers—can be created to replace programmer-written logic and help perform complex numeric functions (e.g., date arithmetic), data manipulation (specialized sorts or data sifts), operating modes, etc. These tailored subroutines are optimized to outperform standard application coding. These functions are usually invoked via CALLs with parameters. This both simplifies coding and performs better than normal programming.
• Tune using callable services that accompany most programming languages and application generators. Once known as linkage editors, these products may be packaged with programming languages or separate but required products. An example is IBM’s Language Environment that supports C/C++, COBOL, PL/I, Fortran, and Assembler. These products provide enhanced language function and aid development or productivity; many also contain parameters which affect program performance.
• Minimizing I/O overhead and read/write time reduces program processing time. Although normally considered system tuning, there are times this tuning is targeted at a specific application whose performance limitations are due to the time and overhead of reading and writing data from disk, tape or other storage. Increasing buffer (storage reserved for file records) allocations in processor and disk controllers allows the OS data access method to prefetch data (anticipate I/O requests and preemptively read them) via algorithms, keeping records likely to be read again where they’re at while flushing unused records to disk. Increasing blocksize results in more records being read at once.
• Network performance affects application performance in distributed networks. In widely-distributed processing, network performance can drastically increase response time beyond other any factor, for three related reasons: transmission speed, error rate and availability. Today’s hard drive transfer rates are measured in gigabits per second, while network transmission rates are measured in megabits per second, a thousandfold less. It should be no surprise that data moves slower in a network than a processor. Multiple links can help, but keeping data and processing together as much as possible and designing a network with sufficient bandwidth provides the best option.
• Adding processing power might solve the problem. This is only true if the processor and the application are cycle constrained, and it’s expensive. Unless the processor utilization is consistently high and tuning options have been exhausted, it should be the last option.

Performance Means Productivity, Responsiveness and Usability

The best tuned, most efficiently performing IT system cannot provide users with premium response time, high availability and consistent business systems unless application programs are also well-tuned and carefully-maintained with a focus not only on function but also on performance. Both performance and function need thorough nurturing along with the facilities an application is designed for, namely to enable users to accomplish their business duties. To assume the application programming staff’s only responsibility is to produce code is a big mistake. If the application once put into production performs poorly and prevents employees from performing their job in a superior manner, the application isn’t providing value nor is it enabling the people who work with it.

---