Understanding Power Supplies for Power Systems
In 2018, I published an article on planning for power that explored the requirements for power connections and the rules about power distribution units (PDUs) when implementing IBM servers. IBM publishes maximum power usage for each of their servers in the technical overview manuals’ operating environment sections. Whenever installing new servers, it’s important to reassess the current power committed through the PDUs. Many new servers require significant power, and I’m running into power issues regularly.
Most confusion about electrical power is centered around what’s being measured—and about why IBM requires so many PDUs when the measurements show less power is actually being used. To clarify these things, we’ll cover some terminology and then discuss PDUs, power cords and other items to think about when planning for power.
Terminology
The IBM technical overview and introduction Redbooks provide the maximum power consumption for the various servers. Measurements will show you’re using less than that, but you still need to plan for the maximum in case you end up using all the resources. Measurements provided typically consist of watts (W), kVA (kilovolt-amps) and voltage (V). Depending on the server, these may be per server or per node. The same power planning needs apply to your intel servers and storage subsystems.
Volts is a measure of an electrical source’s strength at a given amperage, like water pressure in a pipe. Amperes, or amps, is a measure of the total electrical current that can flow, like a water current. kVA, 1000-volt amps, is the key number typically provided by vendors for power planning. A volt amp is the unit used to measure the apparent power in an electrical circuit. Volt amps are used for AC (alternating current) power. Other calculations are available for DC (direct current) power.
I normally plan using kVA, but if you want to plan using amps, you can use the following calculations:
Single-phase amps = ([kVA x 1000]/voltage) or (watts/[voltage x power factor])
Typically, we divide by 208 for voltage. Resources like rapidtables.com can provide calculations for both single- and three-phase power.
In addition to power, you’ll also need plans for air conditioning and UPS systems. Air conditioning uses the energy unit BTU (British thermal unit), and UPS systems use watts and kVA. All of these are published in the various IBM Redbooks. I recommend putting together a spreadsheet for each rack in your data center that includes kVA, amps, watts and BTUs. That way, you can ensure you have the information you always need. It also helps to keep a map of how server power supplies are connected to the various PDUs. You’ll also need to note the kind of PDU plug required for different systems. For example, the POWER E850 and E950 are C19/C20 power cords, but the smaller POWER9 servers, the E980 or E1080, need C13/C14 power cords.
Lately, I’ve been running into an issue with hosting sites: Many of them only allow up to 10kVA per rack, which is basically 2 x 30-amp PDUs (4.99 usable kVA each). The Lenovo 3950 HANA server with 1400W power supplies can draw up to 7.01 kVA. Each E1080 node can draw up to 4.6 kVA, and the E950 has a maximum draw of 4.3 kVA. You can rapidly run out of power on a pair of 30-amp PDUs, which is why power planning is critical. Later, I’ll talk about how you double the PDUs to support redundant power requirements.
PDUs and Line Cords
It’s critical to understand the National Electrical Code’s impact on power. Because of past issues with overloading, fires, etc., PDUs in the U.S. are derated to 80% of their maximum capacity. This means that a single-phase 208V 60-amp PDU is only allowed to have up to 48 amps (9.98kVA) of equipment connected to it, and a single-phase 208V 30-amp PDU is derated to 24 amps (4.99 kVA). A three-phase 30-amp PDU can use up to 8.65 kVA (see Figure 1).
Current PDUs from IBM can support up to 60 amps (48 derated) depending on the line cord, which determines the amperage. IBM currently provides multiple PDUs—the 9188/7188, the 5889/7109, the 7196+ and the new intelligent PDUs (they all start with EP). The first four PDUs—9188/7188/5889/7109—have six pairs of IEC320-C13 outlets rated at 200-240 volts. Each outlet is rated at 10 amps. Each pair of outlets is connected to a 20-amp circuit breaker in the PDU, with each circuit breaker derated to 16 amps. The full amperage rating for the PDU (derated) is 16 amps, 24 amps or 48 amps, depending on the line cord. The difference between the PDUs is that the 5889/7109 combination provides power monitoring capabilities, and the 9188/7188 doesn’t. The 7196+ PDU only has a fixed IEC60309 three-phase 60-amp power cord. It has six outlets that are IEC 320-C19, each of which is derated to 16 amps. There’s also an HVDC 240- to 380-volt DC 90-amp PDU, but I’ve never seen this used in the U.S.
The newer PDUs from IBM are:
- EPTG/EPTJ – 16 to 63 amp 200-240v single- or three-phase PDUs that include 9 x IEC320-C19 and 3 x IEC320-C13 plugs. The C13 plugs are difficult to get to.
- EPTK/EPTL – 208v 60-amp three-phase PDUs with 9 x IEC320-C19 and 3 x IEC320-C13 plugs. Again, the C13 plugs are difficult to get to.
- EPTM/EPTN – the most common; 16 to 63 amps 200-240v single- or three-phase; 12 x IEC320-C13 receptacles.
- EPTP/EPTQ – same as the EPTM/EPTN but only 60-amp 208v three-phase.
Some of these PDUs can be vertically mounted, and some can only mount horizontally. Additionally, certain high-end servers require horizontal mount PDUs, so pay attention to the PDUs’ requirements.
The two most common line cords to the PDUs in the U.S. are:
6492 14' 200/240V/48A UTG0247 IEC309 60A
6654 14' 200/240V/24A UTG0247 L6-30P 30A
The 6492 uses a plug type 363P6W which means the electrician needs to provide a 60-amp circuit with a 363P6W receptacle. The 6654 uses a NEMA L6-30P plug, which requires a 30-amp circuit and a NEMA L6-30R receptacle. There are multiple other line cord options, so it’s important to discuss these with your electrician and data center personnel to ensure you order the correct ones. If you’re using non-IBM PDUs, you’ll need to know the voltage and derated amperage of the PDUs to plan for your power needs. You may also need to get horizontal PDUs for some servers. Details and pictures can be found at the line cord references and in Figure 1 below.
Figure 1. Usable kVA per PDU pair for redundant power:
Designing for Power
Much of the confusion comes from the way redundant power supplies work and the requirements this puts in place for additional PDUs. For example, the E950 has four power supplies set up as two redundant pairs. This means that two cords should go to one PDU and two should go to another, so that the server can continue to run if a circuit or PDU fails. For the most part, two of the power cords may never draw power. However, we don’t get to choose which ones will draw power. This means that redundant power supplies can hide the problem of accidentally exceeding the 4.99 kVA loading on a 30-amp PDU in normal operation, since we’ll never see one of the PDUs in the pair being used. However, if one PDU in a redundant pair happens to fail—for example, if a PDU cord is unplugged or an input circuit fails—all devices will be forced to the remaining PDU, which will likely become overloaded and fail. At that point, one of several things will happen: the circuit breaker will trip and power everything off, the equipment on that PDU may power off or you could have a fire if the circuit breakers don’t do their job. I’ve seen overloaded PDUs several times and the result is never good.
So, how can you design for Power and avoid overload?
First, you need to determine the amperage and derated amperage of your circuits. POWER8, POWER9, Power10 and x3850/3950 HANA servers draw plenty of power, so it’s recommended you go with 60-amp circuits wherever possible. If you remain with 30-amp circuits, you’ll need to provide a lot more PDUs. Once you determine the amperage and voltage, it’s a simple case of planning connections so you don’t exceed the rules for power usage.
Here’s an example for a system consisting of an E950, a two-node E1080 and an HMC and monitor:
Using the maximum use numbers from the Technical Overview Redbooks, each 1080 node is 4.3 kVA and 4500W, the E950 is 4.3 kVA and 4220W and the 7063-CR2 HMC is about 0.48 kVA and 432W. The monitor requirements are negligible (.01 kVA and 11W for a 7316-TF5). For planning purposes we need:
| kVA | Power Plugs | Type | |
| Two-node E1080 | 9.20 | 8 (two pairs per node) | C13 |
| E950 | 4.30 | 4 (two pairs) | C19 |
| 7063-cr2 HMC | 0.48 | 2 (HMC) | C13 |
| 7316-TF5 monitor | 0.01 | 1 | C13 |
| TOTAL | 13.99 kVA |
If we’re using 30-amp (derated to 24-amp) PDUs then we can only support up to 4.99 kVA on a PDU, so the configuration above would require six PDUs (three to support the primary needs and three for failover). With 60-amp PDUs, we can support up to 9.98kVA, so we would only need four PDUs (two PDUs for primary and two for failover). If you switch to three-phase PDUs, you can reduce this a little more. This doesn’t include any storage or other devices you may need to plan for. When doing these calculations, don’t forget to include everything in the rack—including switches.
Summary
IBM has taken great care to ensure their support statements adhere to the electrical codes. Even though you can use the energy estimator to determine numbers closer to what you would really use, it’s still important to plan for enough power to support servers as if they have all resources activated or in use. I normally plan for power using both the energy estimator and the published maximums, and I use the larger of the two numbers. I don’t expect to draw that amount of power, but I must plan for the potential additional power to get proper support and comply with electrical codes. Plus, I also make sure to consider the requirements for redundancy and follow IBM’s rules for maximum servers per PDU. This ensures that the servers are fully supported and that PDU failure doesn’t cause servers to go down—or worse, cause fires.
Redundancy requirements—along with the regulations—mean that you need to plan for what you could use, not just what you normally draw. Therefore, a review of the PDU layouts and usage should be done any time a server, I/O drawer or something similar is added. A preinstall planning meeting should also happen before ordering servers or upgrades.
Taking the time to plan before making changes to your environment can help you to adhere to standards, avoid outages and reduce the risk of costly complications. I see many locations that were planned out correctly, but over time, additional equipment crept into the rack, and no one realized the power draw that some of this equipment had—especially the new x3850 and x3950 HANA servers.
References
For more information on electrical power, check out the following materials:
Useful Power Formulas:
http://www.rapidtables.com/electric/ampere.htm
https://www.rapidtables.com/calc/electric/index.html
Amps to kVA Calculator
Supported PDUs and PDU Power Cords (IBM):
Power10 PDU Power Cords
Power9 PDU Power Cords
Power8 PDU Power Cords
Lenovo and IBM Intel Power Requirements
E1080 Technical Overview and Introduction
E980 Technical Overview and Introduction
E950 Technical Overview and Introduction
S922, S914 and S924 Technical Overview and Introduction
2018 Article on Understanding Power for POWER9