PowerVM Upgrades and Top 12 Tips for Maintaining Your VIO Servers
It’s important to keep VIO up to date and backed up as it’s a critical part of your infrastructure. Jaqui Lynch provides 12 tips for maintaining your VIO servers here.
By Jaqui Lynch06/15/2021
So, here are my top 12 tips for your VIO servers:
1. Stay CurrentThis means regularly upgrading and having a patch plan. When I’m doing patching I do it in the following order:
- Read the readme files
- Server firmware
- NIM server (if there is one) plus it’s I/O firmware
- VIO servers and I/O firmware
- Client LPARs
Stay current if you want IBM to be able to provide you with support. As of November 30, 2021 only VIO v126.96.36.199 and higher will be supported:
All version 2 VIO levels have been out of support since at least September 30, 2020. You can check for current support here.
Planning for your upgrade
Your upgrade steps are going to depend on what level you currently have installed. Previous articles have discussed upgrading from v188.8.131.52 (that is the minimum) to v3. The latest level is v184.108.40.206 (AIX 220.127.116.11) and it requires that your NIM server be at AIX 18.104.22.168 if you plan to use your NIM server for backup and restore or for the upgrades/updates.
Getting the code
The base code is downloaded from the IBM entitled software site (ESS). When you go to download it from your entitled software you will see it listed as 5765-VE3 PowerVM Enterprise ED V3 or 5765-VS3 PowerVM Standard Edition or 5765-VL3 PowerVM Linux Edition. Technology levels and service packs are downloaded from Fix Central as per normal and, as of June 14, 2021 there is a service pack for v3.1.2 available (22.214.171.124) available. I went to ESS and just downloaded the flash image as it is a full PowerVM 3.1 image that has been updated to the latest service pack 126.96.36.199. This saves an update step later. The current image is:
If you’re on v2 you need to upgrade to at least v188.8.131.52 before you can update to 184.108.40.206 and then 220.127.116.11. I’ve been able to upgrade directly from v2 to 18.104.22.168 using the flash image and then upgrade to v3.1.2. A minimum of 22.214.171.124 or 126.96.36.199 is required prior to updating to 188.8.131.52.
This is one of the reasons that updating regular is so important. Not only does it ensure you are running a supported copy, but it also ensures that you don’t have to do a multistep update. Today an update from v2.2.6 would require at least three, possibly four, update steps.
2. Install Hiper Fixes and Update SSH, SSL and JavaAs of June 14, 2021 there are two Hiper fixes for 184.108.40.206. The first is IJ31191 (which goes on as IJ31604) and the second is IJ31936. IJ31936, in particular, fixes a critical issue with NPIV. These patches are downloaded from the IBM ftp.software.ibm.com website and are installed using the emgr command (as root). An example is:
emgr -p -e IJ31936s2a.210414.VIOS220.127.116.11.epkg.Z
The line above does a preview install then you can do it for real.
emgr -e IJ31936s2a.210414.VIOS18.104.22.168.epkg.Z
Both of the patches identified require a reboot of the VIO server.
The necessary patches are identified by using FLRTVC which provides descriptions and the necessary links.
You can use “emgr -P” and “emgr -l” to check what is installed.
SSH and SSL updates are downloaded from the IBM Web downloads page. The latest version of openssh is 22.214.171.1243 and the latest version of openssl is 126.96.36.1992. Java updates are found at Fix Central. When updating ssh, ssl and Java I put them all in a directory and run inutoc. I then use updateios to install these patches, not smitty.
updateios -accept -install -dev /usr/local/soft/javasshssl
VIO 3.1.2 no longer requires Java6 so you can remove it as follows:
updateios -remove Java6
updateios -remove Java6_64
As of VIO 188.8.131.52 you can also remove Java7 the same way.
3. Keep It Simple and GeneralPowerVM is an incredible piece of software that allows you to perform very complex functions. However, just because you can does not mean you should. I tend to keep things as simple as possible so I can rebuild quickly if possible. Some basic rules for setup though are:
- Ensure LMB (memory region size) is the same on all servers if you want to use LPM
- Use hot pluggable adapters
- All adapters should be desired, not required
- Fix page spaces—I make sure I have just one page space of at least 4GB
- Configure dump devices so they are big enough in case you need them
- Configure logging. I add a filesystem called /usr/local/logs and I add logging to the VIO server that goes to that filesystem.
4. DocumentDocumentation is critical, especially when you use network aggregates and have complex SEAs. I use the HMCScanner to document everything before and after updates, but I also keep a separate spreadsheet documenting everything including customer numbers and system numbers.
5. Use Dual VIO ServersDual VIO servers allow you to update one VIO server while the other VIO server runs the environment. It’s important to ensure that zoning and mapping is done correctly so no LPAR is dependent on one VIO server. If you don’t use dual VIO servers then you will have to shut down all your LPARs whenever you want to work on your VIO servers. If you’re ordering a smaller server make sure it has a split backplane so each VIO can have half the disks. You can have more than two VIO servers, but the minimum I recommend is two.
When using dual VIO servers it’s critical that the VLANs configured on the trunk adapters on each VIO server are identical. If they don’t match then the second VIO server will not boot.
6. Use VIO Commands for MaintenanceWhenever possible, use VIO commands for maintenance. It’s not recommended that you become root and use smitty to install software. You should always try to use updateios instead. The two exceptions are installing HIPERs and ifixes which are installed using emgr as root and updating I/O firmware which uses the diag command.
7. Backup RegularlyWhen planning for VIO servers it’s important to plan for how you’re going to back them up and restore. You can back up to a NIM server, tape, DVD, to an NFS server (for the HMC to use later) or to Tivoli Storage Manager. Backup and restore with USB sticks is not supported. Learn more here.
If your HMC is at 950 or higher there is a new backup and restore option available. You can now backup your VIO server IO configuration and your actual VIO server to the HMC and restore it later from the HMC.
Prior to any update I use alt_disk_copy to clone rootvg so that I have a fast fail back if there are issues.
I also make sure to back up the virtual definitions daily as these are not backed up with a mksysb style backup. If you issue the following command as padmin (once) a backup will be done daily.
viosbr –backup -file vio1 –frequency daily numfiles 7
You can view the backups take using “viosbr -view”
VIO servers should also be rebooted regularly. You don’t want to find out that there’s an issue with your boot image or your bootlist during an emergency. I have seen VIO servers that have not been rebooted for over 1000 days—it’s great the VIO is so stable, but it’s terrifying if you’re the person that has to reboot that VIO.
8. Allocate Plenty of ResourcesThe most important LPARs in your system are your VIO servers. They should run at the highest weight (i.e. 254) if they’re in a shared pool. They also should be given plenty of memory and CPU. If the VIO servers are paging or short of CPU it will affect every one of the client LPARs as well.
It’s important to never run at 100% of entitlement on your VIO servers. If you’re consistently running above entitlement, then it impacts your shared ethernet adapter (SEA) performance—so make sure you have enough entitlement.
You will probably need to tune the virtual buffers on your virtual ethernets that you set up. If you have a busy network the defaults are too low.
Finally, you want to make sure you tune your HBA setting (num_cmd_elems and max_xfer_size) to ensure the VIO servers can handle the SAN load. No client LPAR can have these set to a value higher than the VIO server is using.
9. Check Storage Zoning and MappingZoning is when the switch is configured to allow the switch port to talk to the storage and the WWPN for the client LPAR or server. Mapping (or masking) is when the storage subsystem is updated to allow a host (LPAR or server) WWPNs access to specific disk LUNs. Disk LUNs must be provisioned/created at the storage, then mapped and zoned before they can be used in an LPAR. For direct attach LUNs we zone and map the WWNs for the real adapters, for NPIV LPARs we use the WWPNs for the virtual adapters for that LPAR. Physical WWNs tend to start with 10 or 10 and NPIV WWPNs start with C0.
When using NPIV each client LPAR will have two WWPNs for each virtual adapter. The first WWPN is the one normally used, the second is used initially by LPM. Prior to an LPM the first WWPN is used, after the WWPN the second WWPN is used, and for the next LPM it switches back to the initial WWPN. The only exception is if you perform an inactive LPM where it stays on the default WWPN. It’s important to zone and map both WWPNs if you plan to use LPM.
10. Have a Second DiskIf the VIO is running on internal disk then rootvg should be mirrored. If it’s on the SAN there is no need to mirror rootvg, but you should have a second LUN/disk available to take an alt_disk_copy before any maintenance.
If you want to use FBO (file backed optical) add a third disk in its own volume group so that rootvg doesn’t get too big. You should be keeping rootvg as small as possible.
11. Use NTPI highly recommend implementing NTP on your VIO servers. It’s not recommended to allow them to get out of sync with each other. If you ever have to do a security investigation you will find that having the time on all the LPARs and VIO servers will make your life much easier. If you don’t have NTP servers in your own datacenter you can always point directly to a public pool. You can find the list here.
12. Update Your I/O FirmwareFirmware should be kept up to date and I/O firmware is no different. You download the I/O firmware from Fix Central. In order to do that, you need to know the feature codes for your adapters. Below is an example of updating my 32Gb fibre adapter. During the update the HBA may be taken offline and your LPARs may lose half their paths (assumes your zoning and mapping are correct).
Above is where I uploaded the firmware. The rpm below expands the firmware and putsitin /etc/microcode then the diag command installs it.
rpm -ihv --ignoreos df1000e314101506.00012000040025700030.aix.rpm
diag -T download -d fcs0
The above did both fcs0 and fcs1 as it was a multiport card.
Some of the newer firmware updates require minimum levels of firmware on your adapters or they will not boot so it is important to keep these up to date. You can check the current levels using “lsmcode -A.”
Keep VIO Up to DateMaintaining a VIO server isn’t that different to maintaining any production LPAR. It’s important to keep the VIO up to date and backed up as it’s a critical part of your infrastructure. If you take the tips in this article and modify them to fit your environment you should have a robust, healthy and recoverable VIO environment.
- Virtual I/O Server release notes: include USB Memory/Flash key install
- VIOSUPGRADE command
- VIOS viosupgrade command in VIOS 184.108.40.206
- NIM viosupgrade command on the NIM AIX 7.2 TL3 + sp
- PowerVM 220.127.116.11 Service Pack readme
- Fix Central
- Entitled Software (ESS)
- Virtual I/O Server release notes—includes USB Memory/Flash key install
- Upgrading the VIO server (non-SSP)
- VIOS 3.1 Upgrade presentation handout (Raleigh User Group April 2019)
- VIOS Top 10 Tips
- Jaqui’s Presentations and handouts online—includes links to VIO presentations
- FLRTVC (vulnerability checker)
- FLRT (Fix level reporting tool)
- Know how to get a perfpmr
- Know how to run snap (here or here)
- Uploading snap to Testcase
- Opening a support ticket
- HMC Scanner (here or here)
Jaqui Lynch has over 38 years of experience working with a projects and OSes across vendor platforms, including IBM Z, UNIX systems and more.
See more by Jaqui Lynch