How to Upgrade to PowerVM v4.1.0.10
In this article I will go over my recent experience upgrading VIO servers from version 3 to version 4. The process is similar to going from version 2 to version 3, but I did run into a few issues that you should be able to avoid. The first step is to download the code and the associated patches. Then I doublecheck the readme and make sure the VIO server meets all the prerequisites and has no current errors. Then I do my documentation and take backups before starting the upgrade.
AIX, PowerVM, Etc. Base Code
The first step is to download the code. This is found at ESS (Entitled Systems Support). You will find the software base code, update access keys, inventory explorer and other useful items here. I have a separate filesystem called /usr/local in rootvg on my VIO servers. This is where I put code that I want to install if I don’t have access to a NIM LPAR or NFS mounts.
Get the v4.1.0.10 ISO image from ESS:
5765-VE4 Virtual_IO_Server_Base_Install_4.1.0.10_Flash_112023_LCD8292400.iso
Upload it to the VIO into mkdir /usr/local/soft/powervm41
Now mount it:
loopmount -i /usr/local/soft/powervm41/Virtual_IO_Server_Base_Install_4.1.0.10_Flash_112023_LCD8292400.iso -o "-V udfs -o ro" -m /cdrom
Extract the mksysb image for later:
cp /cdrom/usr/sys/inst.images/mksysb_image /usr/local/soft/powervm41/mksysb_vio41010_2023 umount /cdrom
You will also need to download the following from IBM:
1. Latest Java8 8.0.0.811 from Fix Central
2. openSSH_fix15 from aix.software.ibm.com
3. Python3, latest openssl and openSSH from IBM Web Download Page
4.PowerVM 3.1.4.31 updates if you are not at 3.1.4.31
I also have the latest FLRTVC and HMCScanner installed, and I downloaded the VIOS 3.1.4.31 updates from Fix Central—I selected include all prerequisites when I downloaded it.
VIOS 3.1.4.31 Prerequisites
IBM recommends you upgrade to v3.1.4.30 prior to upgrading to v4. V3.1.4.30 is no longer available so the upgrade would be to v3.1.4.31. If you are using NIM PowerVM 3.1.4.31, it has a prerequisite of AIX 7.2.5.7, and PowerVM 4.1.0.10 requires AIX 7.3.2.1, so I recommend upgrading your NIM server to 7.3.2.1 if it is running on hardware that supports that level (POWER8 or higher).
PowerVM 3.1.4.31 and 4.1.0.10 can run on any server that is POWER8 or higher. If you are still on POWER7 then you need to be on v3.1.2 of the VIO.
The VIOS update 3.1.4.31 can be applied to any VIOS that is at 3.1.0.00 or higher, but prior to the upgrade make sure there Is at least 4GB of free space in rootvg.
The VIOS to NIM mapping page has not been updated so you will need to refer to the Readme files, which is what I did.
You should also check that your current firmware and HMC levels are supported by the levels you are going to. You can find this at the flrtlite site or by running FLRT. There are some prerequisites in the README for certain adapters so make sure your I/O adapter firmware code is up to date prior to upgrading.
Make sure any media images are unloaded:
$lsvopt
If it shows something loaded then unload it as follows replacing ?? with the correct number.
$ unloadopt -vtd vtopt??
Always start by running errpt to check for errors. You do not want to try to update a failing system or one that has errors. Additionally, if you are mirroring rootvg you will need to unmirror it or have two spare disks for the upgrade. I also check all my client LPARs before and after each VIO update to make sure they get all their paths back before I go on to the next update.
PowerVM 3.1.4.31 Process
Before I start, I always document everything on the VIO servers and take copies of important files such as:
/etc/ssh/sshd_config
/etc/inetd.conf
/etc/inittab
/etc/rc.tcpip
/etc/resolv.conf
/home/padmin/config/ntp.conf
/etc/hosts
/etc/motd
I find the following commands useful for documentation and I save the output to an external file (usually a .txt file on my PC).
As padmin:
ioslevel lsnports lspv -size lspv -free lspv lsrep lsvopt lsmap -all -npiv | grep vfc lsmap -all -npiv | grep fcs lsmap -all | grep vhost
As root (oem_setup_env)iIfconfig -a
lspv lspv | grep root bootinfo -b bootlist -m normal -o ifconfig -a
Make a note of any interfaces with IP addresses and document those plus the gateway and subnet masks for them.
oslevel -s instfix -i | grep ML lppchk -v lppchk -vm3 lsdev -C | grep fcs lscfg -vpl fcs* | grep Network lscfg -vpl fcs* | grep fcs lsdev -C | grep ent lsdev -C | grep Shared
Note the ent number it returns and use it in the next command instead of ent6.
lsattr -El ent6 | grep ent
lsmcode -A
In my case the VIO was already at 3.1.4.21, so the steps I followed for the update to v3.1.4.31 were:
Download the patches to /usr/local/soft/powervm31431
Take a clone:
#lspv | grep root
Note which disk is altinst_rootvg and replace ?? below
exportvg altinst_rootvg
#alt_disk_copy -V -B -d hdisk??
Also take a mksysb type backup
Get an HMCScanner report
Run “#emgr -P” to look for any efixes
Remove the efixes using emgr
#emgr -r -L patch????
i.e. “#emgr -r -L 38408m9c”
Use updateios to do updates:
$updateios -commit $updateios -accept -install -dev /usr/local/soft/powervm31431
In my case there were 1 (bos.rte.install) plus197 updates to go on.
Before rebooting, run bosboot to rewrite the boot image and use bootlist to rewrite the bootlist – assuming hdisk3 is rootvg, then:
#updtvpkg
You should run updtvpkg any time you update the operating system, or SSL or rpm:
#bosboot -a -d hdisk3 #bootlist -m normal -o #bootlist -m normal hdisk3 #bootlist -m normal -o
Then it is time to reboot and run for a couple of days (this means you can restore to clone if you need to).
If you don’t wait, then viosupgrade will overwrite your clone and you can’t go back to 3.1.4.21 so any recovery would be from your mksysb image.
PowerVM 4.1.0.10 Process
If you make use of the repository (look for /var/vio/VMLibrary) then you will need to take a copy to an external system if it is in rootvg. The v4 upgrade will wipe it out. You are now ready to do the actual upgrade. Feel free to take another mksysb beforehand if you want to.
#lspv | grep root hdisk2 00c47b30939456b9 altinst_rootvg hdisk3 00c47b3050756642 rootvg active
Above you can see that altinst_rootvg is on hdisk2. altinst_rootvg cannot exist during the upgrade or the upgrade will fail, so we need to remove and clear it:
#exportvg altinst_rootvg #chpv -c hdisk2 #chpv -C hdisk2 #lspv hdisk2 00c47b30939456b9 None hdisk3 00c47b3050756642 rootvg acti
Now as padmin check the disk is free:
$lspv -size NAME PVID SIZE(megabytes) hdisk2 00c47b30939456b9 270648 hdisk3 00c47b3050756642 270648 $lspv -free NAME PVID SIZE(megabytes) hdisk2 00c47b30939456b9 270648
If the disk does not show as free, then your upgrade will fail.
I create /home/padmin/filestosave.txt with the following contents. That way it will back these files up during the upgrade into a directory in /home/padmin. This will allow you to compare files later.
/etc/environment
/etc/group
/etc/hosts
/etc/inetd.conf
/etc/inittab
/etc/motd
/etc/netsvc.conf
/etc/passwd
/etc/profile
/etc/resolv.conf
/etc/syslog.conf
/etc/security/limits
/etc/security/login.cfg
/etc/security/passwd
/etc/tunables/nextboot
/etc/tunables/rc.tunevio.sh
/etc/tunables/rc.tunebufs.sh
/usr/local/bin/runnmon.sh
/etc/ssh/sshd_config
/home/padmin/.profile
/home/padmin/filestosave.txt
/etc/ntp.conf
/etc/rc.tcpip
/home/padmin/config/ntp.conf
I am running viosupgrade on the VIO server—you can also do this from NIM but the syntax is different. If you have any problems, you can look at:
/var/adm/ras/ioslogs/viosupg_global.log
Earlier, I copied the mksysb into:
/usr/local/soft/powervm41/mksysb_vio41010_2023.
Check again that it is there.
Make sure there are no NFS mounted filesystems and disable the automatic viosbr backup if you use it:
$viosbr -nobackup
If you don’t unmount the NFS filesystems it will try to copy them during the upgrade. If you don’t disable the viosbr backup, then you may see the following error:
Unmatched (in regex; marked by <– HERE in m/( <– HERE /usr/ios/cli/ioscli/ at /usr/ios/sbin/viosupg.pl line 10846.
If everything looks good, then it is time to run the upgrade:
#viosupgrade -l -i /usr/local/soft/mksysb_vio41010_2023 -a hdisk2 -g /home/padmin/filestosave.txt
The above tells it to run the upgrade to hdisk2 (which we cleared earlier) using the mksysb we copied and using our filestosave.txt file to figure out which extra files we want backed up in /home/padmin.
You will now see a bunch of messages similar to:
Welcome to viosupgrade tool.
Operation triggered for given node(s).
Broadcast message from root (vty0) at 16:38:01 …
WARNING!!! VIOS Upgrade operation is in progress. Kindly Refrain from making any configuration changes…
Please wait for completion..
Upgrading from ioslevel ‘3.1.4.31’ to ‘4.1.0.10’.
Verifying whether the MPIO software(s) is installed on the VIOS.
Verification of the MPIO software(s) is successful.
Initiating VIOS configuration backup..
VIOS configuration backup successful.
Migration upgrade request initiated.
Warning: User LV(s) ‘lv_dumplv1
lv_dumplv2’ created on rootvg will not be preserved.
Enter ‘Y’ to continue or ‘N’ to abort the viosupgrade.
If no option is entered within 60 seconds, the operation will continue..
You should enter y
y
Initiating installation on alternate disk(s)..
Migration of contents to new rootvg initiated.
Installation on alternate disk(s) successful.
Copying files to altinst_rootvg.
Waking up altinst_rootvg successful.
Lots more messages then:
Note : Backup of migrated files of old vg and new vg will be in ‘/home/padmin/viosupg_backup’ with extensions ‘_src.viosupg’ and ‘_dest.viosupg’ respectively.
VIOS will be rebooted after ’60’ seconds to boot from the newly installed disk.
Press Ctrl + c to terminate.
If you hit ctrl-c here it stops. Otherwise it will continue as follows:
VIOS metadata restore (viosbr -restore) will be automatically resumed after the reboot.
VIOS may be rebooted once during this restore process. Refrain from making any changes to the VIOS virtual configurations during the restore process.
You can verify the restore status using ‘viosupgrade -l -q’ command and resume your operation after the completion of the restore process.
A message ‘Upgrade process is completed successfully’ or ‘Upgrade process is completed with failures’ will be displayed on the console on newvg to indicate the status of viosupgrade.
Run ‘viosupgrade -l -q’ to know more on the status.
After about 20 minutes it should be done, and it then reboots. Wait at least 15 more minutes as it will reboot at least 3 times to restore the virtual settings, etc. When you log in as padmin it may or may not prompt you to set the password. In my case it preserved the password, which was nice. You will then need to accept the license:
Indicate by selecting the appropriate response below whether you accept or decline the software maintenance terms and conditions:
Accept (a) | Decline (d) | View Terms (v) > a
I typed in “a” above and then also did the following:
$license -accept
ioslevel should now show 4.1.0.10
Now run:
viosupgrade -l -q
This checks the status of the upgrade and may or may not cause it to reboot. The last message you should see is:
Upgrade process is completed successfully
At this point you should check all your virtual settings as follows:
As padmin $lsnports $lsrep $lsmap -all -npiv | grep vfc $lsmap -all -npiv | grep fcs $lsmap -all | grep vhost $lsvopt As root #lsdev -C | grep ent #lsdev -C | grep Shared #lspv #ifconfig -a
They should match what you had before.
#oslevel -s
This should show as 7300-02-01-2346
Run lspv to check your disks, as they may have been renumbered.
#lspv hdisk2 00c47b30939456b9 rootvg active hdisk3 00c47b3050756642 old_rootvg
Also check df -g. I found that my /usr/local and /usr/local/logs that I had in rootvg had been brought across, which was nice, as I did not have to copy all my scripts and log file back over.
Customizing
Any customizations that you had done will have been lost. This includes changes to /etc/syslog.conf, paging, etc. I also had to increase some of the rootvg filesystems. You can run “df -g” and figure out if you need to do that.
For some reason it once again created two page files in rootvg. I usually have one 8GB page file there, so I do the following:
#lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum paging00 hdisk2 rootvg 1024MB 0 no no lv 0 hd6 hdisk2 rootvg 512MB 2 yes yes lv 0
#lsvg rootvg | grep PP showed me the PPsize was 512MB for rootvg
So I increased hd6 to 8GB as follows:
#chps -s 15 hd6 and removed paging00 #rmps paging00 #lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum hd6 hdisk2 rootvg 8192MB 0 yes yes lv 0
I normally set fsize to -1 and nofiles to 8000 in the default section of /etc/security/limits and I change the herald in the default section of /etc/security/login.cfg to be:
herald = “Unauthorised access prohibited nlogin: ”
I change the herald and /etc/motd to something unfriendly so no hackers can ever say I invited them into the system.
The next step is to make sure the system dump logical volumes are big enough (which they are not):
#sysdumpdev -e 0453-041 Estimated dump size in bytes: 2456121507
I have only lg_dumplv and it is too small:
(#lsvg -l rootvg | grep dump)
So I create two new logical volumes:
mklv -t sysdump -y lv_dumplv1 rootvg 20 hdisk2 mklv -t sysdump -y lv_dumplv2 rootvg 20 hdisk2 sysdumpdev -K
I then set those as the logical volumes to be used and remove the old one:
sysdumpdev -P -p /dev/lv_dumplv1 sysdumpdev -P -s /dev/lv_dumplv2 rmlv lg_dumplv
/etc/syslog.conf also got reverted even though it was in my filestosave list, so I edit it and append the following to the end of it:
mail.debug /usr/local/logs/mailog rotate size 2m files 10 compress *.emerg /usr/local/logs/syslog rotate size 2m files 10 compress *.alert /usr/local/logs/syslog rotate size 2m files 10 compress *.crit /usr/local/logs/syslog rotate size 2m files 10 compress *.err /usr/local/logs/syslog rotate size 2m files 10 compress auth.notice /usr/local/logs/infolog rotate size 2m files 10 compress *.info /usr/local/logs/messages rotate size 2m files 10 compress cd /usr/local/logs touch mailog syslog infolog messages
Then I stop and start syslogd:
stopsrc -s syslogd startsrc -s syslogd
I work on so many vio servers that I get confused, so I make the following change to /etc/environment:
PS1="vio1$: " EDITOR=vi
Use your VIO name instead of vio1, then log out and back in.
I also clean out /etc/inetd.conf so no dangerous programs are uncommented:
cd /etc cp inetd.conf inetd.conf-jl11262023 vi inetd.conf’
I delete everything except:
#ftp stream tcp6 nowait root /usr/sbin/ftpd ftpd #telnet stream tcp6 nowait root /usr/sbin/telnetd telnetd -a #xmquery dgram udp6 wait root /usr/bin/xmtopas xmtopas -p3 caa_cfg stream tcp6 nowait root /usr/sbin/clusterconf clusterconf >>/var /adm/ras/clusterconf.log 2>&1 refresh -s inetd
I also configure NTP if there is an ntp server:
vi /home/padmin/config/ntp.conf
Add the following:
Server ipofntpserver
Where ipofntpserver
Is the IP address of your NTP server
i.e. 192.168.2.100
If the date is way off, then run:
ntpdate ipofntpserver grep ntp /etc/rc.tcpip
Make sure the ntp line is uncommented as follows:
start /usr/sbin/xntpd -a '-c /home/padmin/config/ntp.conf' "$src_running"
Now I go and clean up and update Java, SSH amd SSL. You may need to recopy the patch and java, ssh and ssl directories back to this server:
lslpp -l | grep ava lslpp -l | grep ssh lslpp -l | grep ssl
These show as:
Java8 8.0.0.800 SSL 3.0.10.1001 SSH 8.1.112.2000
I want to update these as there are many security holes that need to be fixed. Previously, I downloaded the latest Java, ssh and ssl and untarred them and copied them all into one directory. As padmin I now use that directory to update them:
$ updateios -accept -install -dev /usr/local/soft/javasshssl-v3-vio-nov152023
This may not update all the language sets, so I remove those:
As padmin:
updateios -remove openssh.msg.CA_ES updateios -remove openssh.msg.CS_CZ updateios -remove openssh.msg.DE_DE updateios -remove openssh.msg.ES_ES updateios -remove openssh.msg.FR_FR updateios -remove openssh.msg.HU_HU updateios -remove openssh.msg.IT_IT updateios -remove openssh.msg.JA_JP updateios -remove openssh.msg.Ja_JP updateios -remove openssh.msg.KO_KR updateios -remove openssh.msg.PL_PL updateios -remove openssh.msg.PT_BR updateios -remove openssh.msg.RU_RU updateios -remove openssh.msg.SK_SK updateios -remove openssh.msg.ZH_CN updateios -remove openssh.msg.ZH_TW updateios -remove openssh.msg.Zh_CN updateios -remove openssh.msg.Zh_TW updateios -remove openssh.msg.ca_ES updateios -remove openssh.msg.cs_CZ updateios -remove openssh.msg.de_DE updateios -remove openssh.msg.es_ES updateios -remove openssh.msg.fr_FR updateios -remove openssh.msg.hu_HU updateios -remove openssh.msg.it_IT updateios -remove openssh.msg.ja_JP updateios -remove openssh.msg.ko_KR updateios -remove openssh.msg.pl_PL updateios -remove openssh.msg.pt_BR updateios -remove openssh.msg.ru_RU updateios -remove openssh.msg.sk_SK updateios -remove openssh.msg.zh_CN updateios -remove openssh.msg.zh_TW oem_setup_env lslpp -l | grep ava lslpp -l | grep ssh lslpp -l | grep ssl
You should now see:
SSH
9.2.112.2000
SSL
3.0.10.1001
Java
8.0.0.811
Finally there are just a couple of patches to go on. But first we check some levels (as root):
#lslpp -L | grep rpm.rte rpm.rte 4.18.1.2001 C F RPM Package Manager #lslpp -L invscout.rte invscout.rte 2.2.0.25 C F Inventory Scout Runtime #slpp -L perl.rte perl.rte 5.34.1.4 C F Perl Version 5 Runtime #lslpp -l | grep -i python python3.9.base 3.9.17.1
I copied the Python patches into /usr/local/soft/flrtfixes/vioflrt
Then as padmin:
$updateios -accept -install -dev /usr/local/soft/flrtfixes/vioflrt
There were 2 to go on:
$oem_setup_env #lslpp -l | grep -i python python3.9.base 3.9.18.0 COMMITTED Python 3.9 64-bit binary python3.9.test 3.9.18.0 COMMITTED Python 3.9 64-bit self-test
There is one openssh patch but it would not install with updateios so I had to resort to emgr.
As root:
#cd /usr/local/soft/flrtfixes/openssh_fix15 #emgr -p -e 38408m9c.230811.epkg.Z The above runs it in preview mode #emgr -e 38408m9c.230811.epkg.Z
The above runs it for real.
#emgr -P PACKAGE INSTALLER LABEL ======================================================== =========== ========== openssh.base.client installp 38408m9c openssh.base.server installp 38408m9c
Now I run flrtvc:
cd /usr/local/soft/flrtvc ./flrtvc-086.ksh No additional patches were found.
Finally, I run:
#updtvpkg
I do this because I have updated rpm, the operating system and ssl.
I check that nothing is missing from crontab (I usually have backups and I start nmon in there). So, I normally have something like:
59 23 * * * /usr/local/bin/runnmon.sh >/dev/null 2>&1 30 2 1 * * /usr/local/bin/viobackup.sh >/dev/null 2>&1 30 2 16 * * /usr/local/bin/viobackup2.sh >/dev/null 2>&1
As admin I also restart my automatic viosbr backups.
As padmin run:
viosbr -backup -file vio1-viobkup -frequency daily -numfiles 7
(Replace vio1 above with the real vio name)
I now rewrite the boot image and the bootlist then reboot:
As root:
#lspv | grep root #bootinfo -b #bosboot -a -d hdisk2 #bootlist -m normal -o #bootlist -m normal hdisk2 #bootlist -m normal -o
If you had set any tunables (network, buffers, changes to fiber card settings, etc.) then you should rerun those now.
And finally, we reboot.
Once the system comes back up, I run all my checks including errpt. I also log in to the client LPARs and check that all their paths have come back. Only when I am happy with all of that do I go work on the primary VIO server and go through the same process again.
Finally, if you need to recreate the repository go ahead and do so, and copy the files back into it.
And you should always end by taking a mksysb backup and a fresh hmcscanner report. If you mirror your rootvg then wait at least a week before remirroring.
PowerVM v4.1.0.10 Upgrade Summary
I know this seems like a long and arduous process, but I prefer running it this way so I can run checks at every step. You can automate a lot of this with NIM, and you also don’t have to customize the VIO the way I do. I do not like putting logs into /var as they can fill up /var, which is a bad thing. So, I always have separate /usr/local and /usr/local/logs filesystems where I can store logs, nmon output, local scripts, etc. I am also a big fan of putting on efixes and ifixes so my VIO servers are secure.
If you want more information, I recommend checking out the YouTube video below that Andrey Klyachkin (Power DevOps) just uploaded.
References
viosupgrade command for the VIO
PowerVM 4.1 Upgrade on YouTube by Power DevOps
IBM Web Download Page: Download OpenSSH, openssl and the Python3 patch from here.
ESS: Download the PowerVM ISO image from here.
Fix Central: Download Java8 8.0.0.811 from here (or higher as needs be).
FLRTVC 086 ZIP
FLRTLITE and the FLRT Data Tables