Establishing an Infrastructure for Tuning a Distributed Network

Just as with system or application tuning on a computer, network tuning is a discipline involving hardware and software, but components and methodology differ significantly. There are also commonalities, like the need to collect performance data from various sources, the need to understand interdependence and interactions involved in changing software, hardware or both. Product knowledge is vital to both methodologies, but product sets differ drastically; that’s where the distinction lies. Both disciplines depend on gathering performance data, analyzing data, then making changes.

I got involved in network performance early in the game, when my CICS clients—mostly insurance companies— were starting to provide remote offices access to mainframe applications. Because of a network design tool, I was able to assist my clients to deploy several primitive networks, but they hadn’t been in place for very long before performance became an issue. A small print job could tie up the line for 15 to 30 minutes and response time was measured in minutes.

So I got the “opportunity” to learn network tuning via the school of hard knocks, but my clients were thankfully patient. Telecom companies weren’t very helpful, partly because network tuning was new to them too, and they were responsive to my calls because it was better me than them. We converted multipoint to point-to-point, allowing us to use full-duplex transmission, we shifted print functions to after-hours, increased bandwidth by upgrading modems with faster ones. It would get a whole lot more sophisticated, but the art of network tuning had arrived.

Times Keep Changing

Early network tuning was pretty straightforward because it was simple: terminals were on one end of a link, a computer on the other. It was rare for a computer to be networked to another computer, although that would soon change due to innovation and the advent of CICS intercommunication. This article is entitled ”Distributed Network Tuning” rather than “Network Tuning” because the very structure of networks has changed, much richer in function, connectivity, transmission protocols, data types, media and connecting devices. Network tuning has become a fine art, with immense challenges addressed by numerous, productive tools.

Know Your Network

The first step in network tuning is to understand all network elements. For me, the starting point is to acquire or create a network diagram, usually in geographical format. Lines represent telecommunications links, various icons represent routers, repeaters, communications controllers, intelligent network nodes (e.g., mainframes, midrange, servers), LANs, SANs, etc. This pictorial diagram is accentuated with annotations such as media (e.g., copper wire, fiber optics, coaxial cables, etc.), nodes, locations and supported functions. Volumes or utilizations—both measures of activity—should be regularly monitored and recorded on the diagram in colors to show when activity is becoming excessive, accompanied with reports showing previous time periods and trends. This diagram can be computerized, and often a large printed or projected version is useful.

Having useful network diagrams and displays doesn’t by itself produce deep knowledge of the network; weekly network status and performance meetings are an even more effective way to develop in-depth knowledge. Since most networks are supported by multiple people at various geographical sites, tele-meetings allow group interactions that reveal the network’s personality (e.g., nuances, quirks, critical and trivial components, operational characteristics and dependencies). Just as with people, the way to gain in-depth network knowledge is to spend a lot of time with it.

This network status meeting should focus on places where established performance thresholds are threatened or exceeded, information that can be used to preventatively anticipate and resolve issues before they occur, or fix things before any significant degradation has occurred. A log of previous tuning should be maintained, because this can be extremely helpful in determining trends, and of getting a historical perspective on previous performance issues, capacity increases and other modifications. Experience is the greatest teacher, and that not only applies to staff members but also the network operation.

The network status meeting’s scope should be expanded to include problem tracking and resolution, because network problems cause performance issues as well as affecting network availability, and the overall infrastructure accommodates both disciplines quite well. Many monitoring products can collect information on both functions, so reports can be designed to include both types of data, and many staff members get involved in both practices. The training and skills are different, so specialization is called for, but there’s also crossover between the skills.

Monitoring and Performance Data Collection

No network tuning can be performed until sufficient information is collected, summarized and massaged in various ways. It can produce reports with insights into how the network is performing, why it’s performing that way, where it’s performing insufficiently or incorrectly, and where it’s not performing at all. While some network aspects are unique, my recommendation is to implement a network performance monitoring and analysis vendor product. There are some excellent products that provide more extensive and in-depth function than could be developed in-house.

Setting up a network monitoring and performance analysis product isn’t as simple as performing the installation procedure; that’s just the starting point. An entire infrastructure is needed, involving organizational as well as software tailoring and procedures establishment, designing reports, assigning responsibilities, determining thresholds, careful study of network diagram(s) to determine features to implement and incorporating the diagram into the product. Planning sessions produce guidelines, specifications, report and display formats, and notification procedures are integrated into a project plan that guides the metamorphosis of the network tuning infrastructure.

Five major network performance indicators should be central to the design:

• Latency: The time to move data from one point to another (e.g., one way, round trip, multi-point)
• Throughput: The quantity of data transmitted within a given unit of time
• Utilization: The percentage of available bandwidth consumed over a given amount of time
• Error Rate: Number of network errors per unit of time
• Packet Loss: Number of packets lost per 100 sent

Tuning

Key tuning options to integrate into the infrastructure include:

• Increase speed: Upgrade to higher speed network links or devices. Simple, most effective and may increase cost.
• Capacity tuning: The simplest form of tuning by adding bandwidth at points of network congestion, or adding memory or processing power to network devices like communications controllers, routers, firewalls, repeaters, modems, switches or bridges, or adding some network devices
• Software tuning: Various software (e.g., TCP/IP, ACF/VTAM, etc.) parameters, like IOBUF (e.g., transmission buffer size), RUSIZE (e.g., size of transmission units) or PACING (e.g, rate of transmitting packets), can be altered to improve network performance
• Data compression: Implement data compression on application and/or network levels. There’s a tradeoff between processor overhead and network traffic, but compression reduces the size of data streams.
• Traffic scheduling: By characterizing network traffic (e.g., online/interactive, file transfer, print, video, audio, etc), traffic can be scheduled based on importance (e.g., online in prime time, print during night hours, etc.). Traffic prioritization can augment scheduling.
• Error reduction: Network errors consume bandwidth by instigating retransmissions, degrading efficiency and throughput. Minimize them.
• Applications tuning: Applications can send large quantities of unnecessary data. Correct these applications.
• Reconfigure network components (e.g., LANs/LAN Adapters, firewalls, back plane, routers, etc.): This is the most work but provides a big payback. Networks are more than interconnected telecommunication links, they include a vast menagerie of devices and connectors that need to be optimized based on their environment.

Setting the Table

Network tuning will be haphazard and ineffective until the necessary network knowledge is collected, a high-quality data analysis tool is deployed, and an effective network tuning infrastructure that includes a comprehensive monitoring and data collection facility, plus an effective organizational structure, is put in place. There are upfront costs of time, effort and money. The payback far exceeds the initial investment, both in network cost savings and in significantly improved response time and availability, hence employee productivity.

Tuning is an ongoing process, a constant vigil turning outages and congestion into a smoothly running network, serving the purpose of organizational objectives.

---