PowerVM Upgrades and Top 12 Tips for Maintaining Your VIO Servers

Jaqui Lynch June 15, 2021

PowerVM v3 has now been out for some time. In this article, I’ll go over what you should think about when maintaining VIO servers. I’ve noticed that once a VIO server is installed, people tend to ignore it until they are forced to upgrade it—this is not recommended.

So, here are my top 12 tips for your VIO servers:

1. Stay Current

This means regularly upgrading and having a patch plan. When I’m doing patching I do it in the following order:

Read the readme files
HMC
Server firmware
NIM server (if there is one) plus it’s I/O firmware
VIO servers and I/O firmware
Client LPARs

I prewrite my documentation on the upgrade steps so I don’t miss anything.

Stay current if you want IBM to be able to provide you with support. As of November 30, 2021 only VIO v3.1.1.10 and higher will be supported:

All version 2 VIO levels have been out of support since at least September 30, 2020. You can check for current support here.

Planning for your upgrade

Your upgrade steps are going to depend on what level you currently have installed. Previous articles have discussed upgrading from v2.2.6.32 (that is the minimum) to v3. The latest level is v3.1.2.21 (AIX 7.2.5.2) and it requires that your NIM server be at AIX 7.2.5.2 if you plan to use your NIM server for backup and restore or for the upgrades/updates.

Getting the code

The base code is downloaded from the IBM entitled software site (ESS). When you go to download it from your entitled software you will see it listed as 5765-VE3 PowerVM Enterprise ED V3 or 5765-VS3 PowerVM Standard Edition or 5765-VL3 PowerVM Linux Edition. Technology levels and service packs are downloaded from Fix Central as per normal and, as of June 14, 2021 there is a service pack for v3.1.2 available (3.1.2.21) available. I went to ESS and just downloaded the flash image as it is a full PowerVM 3.1 image that has been updated to the latest service pack 3.1.2.21. This saves an update step later. The current image is:

Virtual_IO_Server_Base_Install_3.1.2.21_Flash_042021_LCD8250307.iso

If you’re on v2 you need to upgrade to at least v3.1.0.0 before you can update to 3.1.2.10 and then 3.1.2.21. I’ve been able to upgrade directly from v2 to 3.1.1.25 using the flash image and then upgrade to v3.1.2. A minimum of 3.1.2.0 or 3.1.2.10 is required prior to updating to 3.1.2.21.

This is one of the reasons that updating regular is so important. Not only does it ensure you are running a supported copy, but it also ensures that you don’t have to do a multistep update. Today an update from v2.2.6 would require at least three, possibly four, update steps.

2. Install Hiper Fixes and Update SSH, SSL and Java

As of June 14, 2021 there are two Hiper fixes for 3.1.2.21. The first is IJ31191 (which goes on as IJ31604) and the second is IJ31936. IJ31936, in particular, fixes a critical issue with NPIV. These patches are downloaded from the IBM ftp.software.ibm.com website and are installed using the emgr command (as root). An example is:

cd /patches/ij31936
emgr -p -e IJ31936s2a.210414.VIOS3.1.2.21.epkg.Z

The line above does a preview install then you can do it for real.

emgr -e IJ31936s2a.210414.VIOS3.1.2.21.epkg.Z

Both of the patches identified require a reboot of the VIO server.

The necessary patches are identified by using FLRTVC which provides descriptions and the necessary links.
You can use “emgr -P” and “emgr -l” to check what is installed.

SSH and SSL updates are downloaded from the IBM Web downloads page. The latest version of openssh is 8.1.102.2103 and the latest version of openssl is 1.0.2.2102. Java updates are found at Fix Central. When updating ssh, ssl and Java I put them all in a directory and run inutoc. I then use updateios to install these patches, not smitty.

updateios -accept -install -dev /usr/local/soft/javasshssl

VIO 3.1.2 no longer requires Java6 so you can remove it as follows:
updateios -remove Java6
updateios -remove Java6_64

As of VIO 3.1.0.21 you can also remove Java7 the same way.

3. Keep It Simple and General

PowerVM is an incredible piece of software that allows you to perform very complex functions. However, just because you can does not mean you should. I tend to keep things as simple as possible so I can rebuild quickly if possible. Some basic rules for setup though are:

Ensure LMB (memory region size) is the same on all servers if you want to use LPM
Use hot pluggable adapters
All adapters should be desired, not required
Test
Fix page spaces—I make sure I have just one page space of at least 4GB
Configure dump devices so they are big enough in case you need them
Configure logging. I add a filesystem called /usr/local/logs and I add logging to the VIO server that goes to that filesystem.

4. Document

Documentation is critical, especially when you use network aggregates and have complex SEAs. I use the HMCScanner to document everything before and after updates, but I also keep a separate spreadsheet documenting everything including customer numbers and system numbers.

5. Use Dual VIO Servers

Dual VIO servers allow you to update one VIO server while the other VIO server runs the environment. It’s important to ensure that zoning and mapping is done correctly so no LPAR is dependent on one VIO server. If you don’t use dual VIO servers then you will have to shut down all your LPARs whenever you want to work on your VIO servers. If you’re ordering a smaller server make sure it has a split backplane so each VIO can have half the disks. You can have more than two VIO servers, but the minimum I recommend is two.

When using dual VIO servers it’s critical that the VLANs configured on the trunk adapters on each VIO server are identical. If they don’t match then the second VIO server will not boot.

6. Use VIO Commands for Maintenance

Whenever possible, use VIO commands for maintenance. It’s not recommended that you become root and use smitty to install software. You should always try to use updateios instead. The two exceptions are installing HIPERs and ifixes which are installed using emgr as root and updating I/O firmware which uses the diag command.

7. Backup Regularly

When planning for VIO servers it’s important to plan for how you’re going to back them up and restore. You can back up to a NIM server, tape, DVD, to an NFS server (for the HMC to use later) or to Tivoli Storage Manager. Backup and restore with USB sticks is not supported. Learn more here.

If your HMC is at 950 or higher there is a new backup and restore option available. You can now backup your VIO server IO configuration and your actual VIO server to the HMC and restore it later from the HMC.

Prior to any update I use alt_disk_copy to clone rootvg so that I have a fast fail back if there are issues.

I also make sure to back up the virtual definitions daily as these are not backed up with a mksysb style backup. If you issue the following command as padmin (once) a backup will be done daily.

viosbr –backup -file vio1 –frequency daily numfiles 7

You can view the backups take using “viosbr -view”

VIO servers should also be rebooted regularly. You don’t want to find out that there’s an issue with your boot image or your bootlist during an emergency. I have seen VIO servers that have not been rebooted for over 1000 days—it’s great the VIO is so stable, but it’s terrifying if you’re the person that has to reboot that VIO.

8. Allocate Plenty of Resources

The most important LPARs in your system are your VIO servers. They should run at the highest weight (i.e. 254) if they’re in a shared pool. They also should be given plenty of memory and CPU. If the VIO servers are paging or short of CPU it will affect every one of the client LPARs as well.

It’s important to never run at 100% of entitlement on your VIO servers. If you’re consistently running above entitlement, then it impacts your shared ethernet adapter (SEA) performance—so make sure you have enough entitlement.

You will probably need to tune the virtual buffers on your virtual ethernets that you set up. If you have a busy network the defaults are too low.

Finally, you want to make sure you tune your HBA setting (num_cmd_elems and max_xfer_size) to ensure the VIO servers can handle the SAN load. No client LPAR can have these set to a value higher than the VIO server is using.

9. Check Storage Zoning and Mapping

Zoning is when the switch is configured to allow the switch port to talk to the storage and the WWPN for the client LPAR or server. Mapping (or masking) is when the storage subsystem is updated to allow a host (LPAR or server) WWPNs access to specific disk LUNs. Disk LUNs must be provisioned/created at the storage, then mapped and zoned before they can be used in an LPAR. For direct attach LUNs we zone and map the WWNs for the real adapters, for NPIV LPARs we use the WWPNs for the virtual adapters for that LPAR. Physical WWNs tend to start with 10 or 10 and NPIV WWPNs start with C0.

When using NPIV each client LPAR will have two WWPNs for each virtual adapter. The first WWPN is the one normally used, the second is used initially by LPM. Prior to an LPM the first WWPN is used, after the WWPN the second WWPN is used, and for the next LPM it switches back to the initial WWPN. The only exception is if you perform an inactive LPM where it stays on the default WWPN. It’s important to zone and map both WWPNs if you plan to use LPM.

10. Have a Second Disk

If the VIO is running on internal disk then rootvg should be mirrored. If it’s on the SAN there is no need to mirror rootvg, but you should have a second LUN/disk available to take an alt_disk_copy before any maintenance.

If you want to use FBO (file backed optical) add a third disk in its own volume group so that rootvg doesn’t get too big. You should be keeping rootvg as small as possible.

11. Use NTP

I highly recommend implementing NTP on your VIO servers. It’s not recommended to allow them to get out of sync with each other. If you ever have to do a security investigation you will find that having the time on all the LPARs and VIO servers will make your life much easier. If you don’t have NTP servers in your own datacenter you can always point directly to a public pool. You can find the list here.

12. Update Your I/O Firmware

Firmware should be kept up to date and I/O firmware is no different. You download the I/O firmware from Fix Central. In order to do that, you need to know the feature codes for your adapters. Below is an example of updating my 32Gb fibre adapter. During the update the HBA may be taken offline and your LPARs may lose half their paths (assumes your zoning and mapping are correct).

cd /usr/local/soft/patches/en1b

Above is where I uploaded the firmware. The rpm below expands the firmware and putsitin /etc/microcode then the diag command installs it.

rpm -ihv –ignoreos df1000e314101506.00012000040025700030.aix.rpm
diag -T download -d fcs0

The above did both fcs0 and fcs1 as it was a multiport card.

Some of the newer firmware updates require minimum levels of firmware on your adapters or they will not boot so it is important to keep these up to date. You can check the current levels using “lsmcode -A.”

Keep VIO Up to Date

Maintaining a VIO server isn’t that different to maintaining any production LPAR. It’s important to keep the VIO up to date and backed up as it’s a critical part of your infrastructure. If you take the tips in this article and modify them to fit your environment you should have a robust, healthy and recoverable VIO environment.