Distributed Network Testing: The Key to Productive Network Management Infrastructure
Network testing was a pretty straightforward process when IBM assigned me to network implementation and management in my job as a Systems Engineer. There weren’t many network components: Point-to-point or multipoint modems, a channel connection between mainframe and 3705 communication controllers with associated network interfaces (which was admittedly complex), the analog telecommunications link, a 3270 communications controller network interface and cables connecting each 3270 “green screen” display.
Even though the connections were simple, connecting everything together was no guarantee of functionality, and my team quickly learned that once everything was customized and attached, it also had to be verified. Not surprisingly, many errors or omissions were resolved before going online, which avoided end-user dissatisfaction and lost productivity. Many of the testing techniques we developed were also helpful when a connection either intermittently or permanently failed. Little did we know we were part of a growing genesis of network testing methodologies and disciplines.
The complexity and sophistication of network testing has not just grown by leaps and bounds; it’s grown by magnitudes and variations, methods and techniques, tools and providers. Links are no longer just analog, but also digital, wireless or broadband—and most often a mixture of all possibilities. Testing tools, testing procedures, test-based vendors, validation techniques and network testing methodologies have all blossomed in numbers and approaches. Network testing has become a complex and elaborate information technology endeavor.
Know Your Network
The first step in establishing a network testing methodology is to understand all network elements. The initial task is to acquire or create a topological network diagram, where lines represent telecommunications links, various icons represent routers, repeaters, communications controllers, intelligent network nodes (processors), LANs (Local Area Networks), SANs (Storage Area Networks), etc. This pictorial diagram is enhanced with annotations such as media (copper wire, fiber optics, coaxial cables, etc.), nodes, locations and supported functions. Measures of activity—such as volumes or utilizations—should be color-coded on the diagram to show activity by time of day, complete with reports showing previous periods and trends. This diagram can be computerized, projected or printed.
Having useful network diagrams and displays alone doesn’t produce deep network knowledge; weekly network status and performance meetings greatly enhance in-depth knowledge. Since most networks are supported by groups at various geographical sites, tele-meetings allow group interactions that reveal the network’s “personality”—its nuances, quirks, critical and trivial components, operational characteristics and dependencies. Just as with people, the best way to gain deep knowledge about a network is to spend a lot of time with it.
Tailoring Network Testing Methodology
Once knowledge of workloads, availability standards, response time requirements, equipment, protocols, links, applications, timings, facilities and unique network characteristics is known, it’s viable to customize a network testing methodology to specific network characteristics. This includes procedures, techniques, hardware testing devices and all necessary testing tools to troubleshoot pertinent components. In this way, determining required diagnosis hardware and software becomes a relatively simple matter; you just have to match them to the network structure.
In addition to acquiring and/or developing necessary testing tools, the tools themselves need to be tested. While they’re being vetted, procedures must be written and exercised to verify all aspects are functioning as expected, and the results are correct. Testers and procedure developers (who are often the same individuals) then need to train the rest of the staff, both in terms of reviewing the materials with them and providing hands-on experience with many of the developed measures. An ongoing training plan also needs to be established.
Another important network testing aspect (one which varies with the organization) is vendor support. In most IT networks, a portion—sometimes a very large portion—of the network is owned by vendors rather than clients, and as such some testing must be performed by the vendor (albeit with vendor approval, a client may also be empowered to perform the same procedures and tests). Almost every client-based network tool or facility is offered by more than one vendor, so it’s crucial to determine which product or vendor is most appropriate. In addition to product quality and applicability, vendor support is a key selection criteria in determining the best network solution.
Implementing the Strategy
When a network testing methodology or a new network is implemented, all network testing facilities need to be in place. As mentioned, network testing has two primary purposes: Validation of new network components, and identification, diagnosis and resolution of network errors. A structure for error reporting, problem tracking and problem resolution needs to be in place. When a problem occurs, the following data collection inquiries must be answered:
- Is the problem intermittent or permanent?
- How severe of an impact does this problem have on users’ ability to do their jobs?
- How many individuals are affected?
- What symptoms are being experienced?
- When did the problem occur?
- Where is the reporter located?
- Various other items as appropriate.
After a problem is reported and information is gathered, a network analyst gets assigned to the incident, and the next step is to identify and locate the problem. This is where testing tools become vital contributors to a solution, ranging from something as simple and easy as the “ping” command—which tests for connectivity—to a multi-function network tester or micromapper. Once the failing component(s) are located, the responsible party must be identified and contacted if it’s a vendor. An owner—someone who will drive the problem through to resolution—should be assigned and held responsible.
Maintaining and Modifying the Strategy
A network isn’t static; it’s a living, evolving and growing entity—an asset controlled by no single owner. Because of this, any network testing methodology that’s established needs to be pliable and flexible. Network testing tools and procedures must evolve with network hardware, software, protocols, etc. Existing tools may or may not validate new hardware. New vendors mean new tools, procedures and training, especially when it comes to interaction between new and old facilities. Management needs to keep things current to assure continuity.
The external world isn’t the only thing that’s continually changing. The internal organization, systems, applications and business processes are all in a constant state of flux, driven by business objectives and user demands. Automating business functions and enabling user-friendly, responsive applications is vital, but only if the applications are adapted to the changes affecting them. Network testing must adapt to change, and can help other IT components do the same.
Keeping an Eye on Things
Network testing can be automated in various ways, which can simplify problem determination by automatically executing command lists in response to error codes or other malfunction indicators such as high transient error rates or excessive utilization indicators. In addition, an automation product can issue commands to narrow down an error’s nature and location, and sometimes can even take the necessary actions to resolve the problem, like activating a wireless link that occasionally fails due to interference.
Where Does It Break?
Another form of testing that deserves mention is stress testing. Stressing a network is a complicated and challenging exercise that drives a large amount of simulated work through a network. That is, stress testing drives data of various forms through a network and monitors the network performance and integrity as it deals with varying, controlled volumes of work via monitoring products, statistical analysis and other measurements. Scheduling and logistics can be challenging, but it’s rewarding, too. Having participated in several network/central processing/website stress tests, I will say that you can learn more about a network in one night of stress testing than you would learn across several months of traditional network activity.
Testing Means Integrity
Network testing is a vital, fundamental part of an efficacious and productive network management infrastructure, and is mandatory for a network problem identification, evaluation and resolution methodology. Network errors are a corrupting influence on performance, and are the primary detractors of online availability. They degrade vital business processes for employees and customers, and hinder business functions that are essential to organizational survival. Although there are up-front costs of time, effort and money, network testing’s payback far exceeds its cost—especially when you consider network integrity along with superior response time and availability, resulting in employee productivity. Network testing is an ongoing process that turns outages and congestion into a smoothly running network, and fully serves an enterprise’s objectives.