More Power to You: A Closer Look at the Power10 E1080 Server

On September 8, 2021 IBM announced the first server in the Power10 range—the E1080 with an order number of 9080-HEX. The general availability (GA) date for the system is September 17, 2021, and this system is the next generation of POWER servers.

Hardware Technology

The E1080 is similar to the E980 in that it uses up to four 5U CPU drawers along with a 2U SCU (system control unit) in a 19” rack. But there are some significant differences, especially when you look at performance and energy use.

Each CEC (or CPU drawer) has four Power10 sockets, 64 memory DDIMMs. Eight PCIe gen 5 slots, four NVME SSD disk slots, five fans and four power supplies. There are three different core options for the sockets, 3.9ghz 10 cores, 4.15ghz 12 cores or 4ghz 15 cores. With a fully populated 15-core system you can have a total of 240 SMT8 cores available for use. Each Power E1080 server requires a minimum of 16 permanent processor core activations, using either static activations or Linux on Power activations. This is a per-system requirement.

The Power10 is now using DDIMs (differential DIMMs) with the Open Memory Interface. The DDIMMs provide and enterprise buffer, N=1 power management and spare DRAM support. They also provide up to 1.8x the memory bandwidth of the e980 memory. There are 16 DDIMM slots per socket, allowing for 64 DDIM slots per drawer. DDIMMs are purchased in groups of 4 and are either 32GB (3200Mbps), 64GB (3200Mbps), 128GB (2933Mbps) or 256GB (2933Mbps) in size. With the largest DDIMMs, each drawer can have up to 16TB of memory with a total of 64TB for a fully populated server using 256GB DDIMMs. IBM also stated that they plan to carry forward the DDIMMs into the Power11 systems when they come out.

A couple things you should note about memory. You need to install a minimum of eight memory features (32 x DDIMMs) in a server and you cannot mix DDIMM features) within a socket. You can mix DDIMM features within a node, but it’s not recommended for performance reasons. In order to get the best throughput and performance, it’s recommended that you populate all the DDIMM slots. At least 50% of the memory on a server must have permanent memory activations which can be static, mobile, or Linux on POWER. At least 25% must be static activations or Linux on POWER activations.

From a performance perspective each drawer has the potential to provide for up to 1636 GB/sec in memory bandwidth and up to 576GB/sec in IO bandwidth.

The SCU (system control unit) is the controller for the nodes. It has the redundant service processor (FSP) cards, a front accessible USB port, support for the operations panel, system vital product data (VPD) and the external DVD. The DVD drive does not come by default but can be configured with the SCU.

Each drawer comes with nine PCIe gen 5 blind swap low profile slots that support most of the current POWER9 adapters.

If you need more adapters than the slots allow for you can add up to 16 EMX0 I/O expansion drawers (four per CEC drawer). Each EMX0 drawer has one or two EMXH PCIe six-slot gen3 fanout modules with slots that support full height cards. Current EMX0/EMXH drawers will work on the E1080 but will require a new cable card. If you need additional disk you can add up to 168 EXP24S disk drawers for a total of 4032 drives. However, there’s a maximum of 16 x EXP24S drawers per PCIe I/O drawer due to cabling considerations.

The system is designed to fit in a standard 19” rack and is specifically certified for the IBM 42U enterprise rack (7695-S42), which has optimized cable routing. This rack includes a rear rack extension that provides the space to hold the cables, keeping the center clear for cooling and serviceability.

The E1080 is installed by IBM and offers multiple hardware maintenance options. These include:

• 24-7 with two-hour service objective—Optional
• 24-7 with four-hour service objective—Standard
• 9-5 with four-hour service objective
• 9-5 next-business-day

Hardware Management Console (HMC)

The E1080 requires a management console (HMC) and it has to be at V10R1M1010, which will be available September 17, 2021. This version of the HMC code will support the firmware (FW1010) required on the E1080 server. V10R1 will be supported only on 7063-CR1, 7063-CR2 and Virtual HMC appliances, and it’s not supported on 7042 Machine types. HMC with V10R1 will not be able to manage POWER7 processor-based systems. It only supports POWER8 and above. Since the 7063-CR1 has been withdrawn, I highly recommend buying a pair of 7063-CR2 HMCs when you get your first E1080. This will let you set up a completely fresh environment at the latest levels without impacting your current environment. You can migrate any POWER8 or higher systems to the new HMCs and leave any older systems on the old HMCs. As with the 7063-CR1 the 7063-CR2 uses 1Gb/sec RJ45 ethernet connectivity on the private network. It’s a single socket 1U server with one six-core POWER9 chip and 128GB of memory. It has two 1.8TB disk drives in a RAID-1 configuration. There are four RJ45 ethernet ports (used for public and private networks) and a separate dedicated IPMI port to use for the base management controller (BMC). The BMC can be cabled separately or can share ETH0 if that is configured on the public network. I typically use the shared option as it means I only need two cables to each HMC (one public and one private) rather than three. There are also two USB3 ports in front and two in the back. There is no DVD drive so if you need to use DVD then you will need to buy the external DVD. The 7063-CR2 is compatible with flat panel console kits 7316-TF3, TF4, and TF5 but all except the 7316-TF5 have been withdrawn.

The other option to the physical HMC is a virtual HMC (which runs on a POWER server). I recommend reading all the details around the virtual HMC prior to ordering one. As an example, the vHMC cannot manage the system it is hosted on.

Software Technology

Exciting as the hardware is the full potential is only realized when the correct software is running. As with earlier POWER servers, the E1080 has some prerequisites. The required operating system minimums depend on whether the client LPAR is running under the control of a VIO server or as a standalone LPAR. So let’s look at the prerequisites.

Apart from the HMC the VIO server needs to be running POWERVM 3.1.3.10 or later. That version became available September 10, 2021 and can be downloaded from ESS or from Fix Central. Each E1080 comes with PowerVM Enterprise edition entitlement.

Minimum operating system levels are detailed in the announcement and also in the E1080 Technical Overview and Introduction Redbook. As an example AIX 7.2 needs to be at 7.2.5.1 or 7.2.4.1 in order to run on the E1080 under a VIO server. AIX 7.1.5.5 will also run under a VIO , but it will be in POWER8 mode. Without a VIO server, AIX needs to be at 7.2.5.3 (9/10/2021) or 7.2.4.5 (planned for 2/11/2022). There are also specific minimum levels for IBM i and Linux (RHEL and SUSE).

For performance reasons, IBM also recommends upgrading any Java applications to at least Java7, preferably Java8.

Update Access Keys (UAKs)

IBM has been using Update Access Keys (UAKs) for system firmware updates since POWER8. Firmware updates can be downloaded, but at application time a check is done against the UAK expiration date. If the firmware update is more recent that that expiration date then the updates can not be applied. By default, newly delivered systems have an UAK that usually expires after three years. Thereafter, the UAK can be extended every six months, but only if a maintenance contract exists. You can request updated UAKs at the ESS site and then apply them from the HMC.

On Power10, AIX 7.1 and later now include a UAK. Starting with the September 2021 AIX Service Packs, all AIX updates will now require a valid UAK. The initial key expiration date aligns with the software maintenance agreement (SWMA) for that system. The AIX UAK is checked during the install of updates, partition boot and LPM operations. There’s a new command (lparstat -u) that allows you to check the key expiration date. ESS is being updated to better manage keys for AIX and firmware. It will allow you to request multiple keys and emergency keys.

TDAs

A TDA is a technical assessment that the solution will work and it’s mandatory on the E1080. I always do TDAs, even when not required. This is where you pick up issues around power, networking, SAN, etc.

For the Power10 E1080 an expert level pre-sale TDA is mandatory as is an expert level pre-install TDA. Even though they take time, it’s worth the time saved by avoiding errors.

Significant Technology and Performance Enhancements

The E1080 is the latest evolution in IBM’s POWER line of servers. It offers significant enhancements in technology and performance and is generating a great deal of excitement in the technology community. Be sure to look at Nigel Griffith’s videos below on his top 10 facts as well as the other information in the references. I can’t wait to install the new VIO server and AIX levels and am really looking forward to working on an E1080 or one of the smaller Power10 servers which will hopefully be out early next year

References 

• e1080 Datasheet
• Article by Austin Fowler
• Power10 Home Page
• Redbook: Power E1080 Technical Overview and Introduction
• Software PVUs
• Nigel Griffiths Youtube video on E1080 Top 10 Facts and Highlights
• Nigel Griffiths Youtube video on E1080 Top 10 Technical Fast Facts
• Nigel Griffiths: First look at the HMC CR2
• E1080 Announcement
• Virtual HMC Overview
• IBM ESS
• Fix Central
• IBM Prerequisites Site

---