How to Upgrade to PowerVM v184.108.40.206
IBM Champion Jaqui Lynch highlights her experience upgrading PowerVM from v3 to v220.127.116.11
AIX, PowerVM, Etc. Base CodeThe first step is to download the code. This is found at ESS (Entitled Systems Support). You will find the software base code, update access keys, inventory explorer and other useful items here. I have a separate filesystem called /usr/local in rootvg on my VIO servers. This is where I put code that I want to install if I don't have access to a NIM LPAR or NFS mounts.
Get the v18.104.22.168 ISO image from ESS:
5765-VE4 Virtual_IO_Server_Base_Install_22.214.171.124_Flash_112023_LCD8292400.isoUpload it to the VIO into mkdir /usr/local/soft/powervm41
Now mount it:
loopmount -i /usr/local/soft/powervm41/Virtual_IO_Server_Base_Install_126.96.36.199_Flash_112023_LCD8292400.iso -o "-V udfs -o ro" -m /cdromExtract the mksysb image for later:
cp /cdrom/usr/sys/inst.images/mksysb_image /usr/local/soft/powervm41/mksysb_vio41010_2023 umount /cdromYou will also need to download the following from IBM:
1. Latest Java8 188.8.131.521 from Fix Central
2. openSSH_fix15 from aix.software.ibm.com
3. Python3, latest openssl and openSSH from IBM Web Download Page
4.PowerVM 184.108.40.206 updates if you are not at 220.127.116.11
I also have the latest FLRTVC and HMCScanner installed, and I downloaded the VIOS 18.104.22.168 updates from Fix Central—I selected include all prerequisites when I downloaded it.
VIOS 22.214.171.124 PrerequisitesIBM recommends you upgrade to v126.96.36.199 prior to upgrading to v4. V188.8.131.52 is no longer available so the upgrade would be to v184.108.40.206. If you are using NIM PowerVM 220.127.116.11, it has a prerequisite of AIX 18.104.22.168, and PowerVM 22.214.171.124 requires AIX 126.96.36.199, so I recommend upgrading your NIM server to 188.8.131.52 if it is running on hardware that supports that level (POWER8 or higher).
PowerVM 184.108.40.206 and 220.127.116.11 can run on any server that is POWER8 or higher. If you are still on POWER7 then you need to be on v3.1.2 of the VIO.
The VIOS update 18.104.22.168 can be applied to any VIOS that is at 3.1.0.00 or higher, but prior to the upgrade make sure there Is at least 4GB of free space in rootvg.
The VIOS to NIM mapping page has not been updated so you will need to refer to the Readme files, which is what I did.
You should also check that your current firmware and HMC levels are supported by the levels you are going to. You can find this at the flrtlite site or by running FLRT. There are some prerequisites in the README for certain adapters so make sure your I/O adapter firmware code is up to date prior to upgrading.
Make sure any media images are unloaded:
$lsvoptIf it shows something loaded then unload it as follows replacing ?? with the correct number.
$ unloadopt -vtd vtopt??Always start by running errpt to check for errors. You do not want to try to update a failing system or one that has errors. Additionally, if you are mirroring rootvg you will need to unmirror it or have two spare disks for the upgrade. I also check all my client LPARs before and after each VIO update to make sure they get all their paths back before I go on to the next update.
PowerVM 22.214.171.124 ProcessBefore I start, I always document everything on the VIO servers and take copies of important files such as:
I find the following commands useful for documentation and I save the output to an external file (usually a .txt file on my PC).
ioslevel lsnports lspv -size lspv -free lspv lsrep lsvopt lsmap -all -npiv | grep vfc lsmap -all -npiv | grep fcs lsmap -all | grep vhostAs root (oem_setup_env)iIfconfig -a
lspv lspv | grep root bootinfo -b bootlist -m normal -o ifconfig -aMake a note of any interfaces with IP addresses and document those plus the gateway and subnet masks for them.
oslevel -s instfix -i | grep ML lppchk -v lppchk -vm3 lsdev -C | grep fcs lscfg -vpl fcs* | grep Network lscfg -vpl fcs* | grep fcs lsdev -C | grep ent lsdev -C | grep SharedNote the ent number it returns and use it in the next command instead of ent6.
lsattr -El ent6 | grep ent lsmcode -AIn my case the VIO was already at 126.96.36.199, so the steps I followed for the update to v188.8.131.52 were:
Download the patches to /usr/local/soft/powervm31431
Take a clone:
#lspv | grep rootNote which disk is altinst_rootvg and replace ?? below
#alt_disk_copy -V -B -d hdisk??Also take a mksysb type backup
Get an HMCScanner report
Run “#emgr -P” to look for any efixes
Remove the efixes using emgr
#emgr -r -L patch????i.e. “#emgr -r -L 38408m9c”
Use updateios to do updates:
$updateios -commit $updateios -accept -install -dev /usr/local/soft/powervm31431In my case there were 1 (bos.rte.install) plus197 updates to go on.
Before rebooting, run bosboot to rewrite the boot image and use bootlist to rewrite the bootlist – assuming hdisk3 is rootvg, then:
#updtvpkgYou should run updtvpkg any time you update the operating system, or SSL or rpm:
#bosboot -a -d hdisk3 #bootlist -m normal -o #bootlist -m normal hdisk3 #bootlist -m normal -oThen it is time to reboot and run for a couple of days (this means you can restore to clone if you need to).
If you don't wait, then viosupgrade will overwrite your clone and you can't go back to 184.108.40.206 so any recovery would be from your mksysb image.
PowerVM 220.127.116.11 ProcessIf you make use of the repository (look for /var/vio/VMLibrary) then you will need to take a copy to an external system if it is in rootvg. The v4 upgrade will wipe it out. You are now ready to do the actual upgrade. Feel free to take another mksysb beforehand if you want to.
#lspv | grep root hdisk2 00c47b30939456b9 altinst_rootvg hdisk3 00c47b3050756642 rootvg activeAbove you can see that altinst_rootvg is on hdisk2. altinst_rootvg cannot exist during the upgrade or the upgrade will fail, so we need to remove and clear it:
#exportvg altinst_rootvg #chpv -c hdisk2 #chpv -C hdisk2 #lspv hdisk2 00c47b30939456b9 None hdisk3 00c47b3050756642 rootvg actiNow as padmin check the disk is free:
$lspv -size NAME PVID SIZE(megabytes) hdisk2 00c47b30939456b9 270648 hdisk3 00c47b3050756642 270648 $lspv -free NAME PVID SIZE(megabytes) hdisk2 00c47b30939456b9 270648If the disk does not show as free, then your upgrade will fail.
I create /home/padmin/filestosave.txt with the following contents. That way it will back these files up during the upgrade into a directory in /home/padmin. This will allow you to compare files later.
I am running viosupgrade on the VIO server—you can also do this from NIM but the syntax is different. If you have any problems, you can look at:
Earlier, I copied the mksysb into:
Check again that it is there.
Make sure there are no NFS mounted filesystems and disable the automatic viosbr backup if you use it:
$viosbr -nobackupIf you don’t unmount the NFS filesystems it will try to copy them during the upgrade. If you don’t disable the viosbr backup, then you may see the following error:
Unmatched (in regex; marked by <-- HERE in m/( <-- HERE /usr/ios/cli/ioscli/ at /usr/ios/sbin/viosupg.pl line 10846.
If everything looks good, then it is time to run the upgrade:
#viosupgrade -l -i /usr/local/soft/mksysb_vio41010_2023 -a hdisk2 -g /home/padmin/filestosave.txtThe above tells it to run the upgrade to hdisk2 (which we cleared earlier) using the mksysb we copied and using our filestosave.txt file to figure out which extra files we want backed up in /home/padmin.
You will now see a bunch of messages similar to:
Welcome to viosupgrade tool.
Operation triggered for given node(s).
Broadcast message from root (vty0) at 16:38:01 ...
WARNING!!! VIOS Upgrade operation is in progress. Kindly Refrain from making any configuration changes...
Please wait for completion..
Upgrading from ioslevel '18.104.22.168' to '22.214.171.124'.
Verifying whether the MPIO software(s) is installed on the VIOS.
Verification of the MPIO software(s) is successful.
Initiating VIOS configuration backup..
VIOS configuration backup successful.
Migration upgrade request initiated.
Warning: User LV(s) 'lv_dumplv1
lv_dumplv2' created on rootvg will not be preserved.
Enter 'Y' to continue or 'N' to abort the viosupgrade.
If no option is entered within 60 seconds, the operation will continue..
You should enter y
Initiating installation on alternate disk(s)..
Migration of contents to new rootvg initiated.
Installation on alternate disk(s) successful.
Copying files to altinst_rootvg.
Waking up altinst_rootvg successful.
Lots more messages then:
Note : Backup of migrated files of old vg and new vg will be in '/home/padmin/viosupg_backup' with extensions '_src.viosupg' and '_dest.viosupg' respectively.
VIOS will be rebooted after '60' seconds to boot from the newly installed disk.
Press Ctrl + c to terminate.
If you hit ctrl-c here it stops. Otherwise it will continue as follows:
VIOS metadata restore (viosbr -restore) will be automatically resumed after the reboot.
VIOS may be rebooted once during this restore process. Refrain from making any changes to the VIOS virtual configurations during the restore process.
You can verify the restore status using 'viosupgrade -l -q' command and resume your operation after the completion of the restore process.
A message 'Upgrade process is completed successfully' or 'Upgrade process is completed with failures' will be displayed on the console on newvg to indicate the status of viosupgrade.
Run 'viosupgrade -l -q' to know more on the status.
After about 20 minutes it should be done, and it then reboots. Wait at least 15 more minutes as it will reboot at least 3 times to restore the virtual settings, etc. When you log in as padmin it may or may not prompt you to set the password. In my case it preserved the password, which was nice. You will then need to accept the license:
Indicate by selecting the appropriate response below whether you accept or decline the software maintenance terms and conditions:
Accept (a) | Decline (d) | View Terms (v) > a
I typed in "a" above and then also did the following:
$license -acceptioslevel should now show 126.96.36.199
viosupgrade -l -qThis checks the status of the upgrade and may or may not cause it to reboot. The last message you should see is:
Upgrade process is completed successfully
At this point you should check all your virtual settings as follows:
As padmin $lsnports $lsrep $lsmap -all -npiv | grep vfc $lsmap -all -npiv | grep fcs $lsmap -all | grep vhost $lsvopt As root #lsdev -C | grep ent #lsdev -C | grep Shared #lspv #ifconfig -aThey should match what you had before.
#oslevel -sThis should show as 7300-02-01-2346
Run lspv to check your disks, as they may have been renumbered.
#lspv hdisk2 00c47b30939456b9 rootvg active hdisk3 00c47b3050756642 old_rootvgAlso check df -g. I found that my /usr/local and /usr/local/logs that I had in rootvg had been brought across, which was nice, as I did not have to copy all my scripts and log file back over.
CustomizingAny customizations that you had done will have been lost. This includes changes to /etc/syslog.conf, paging, etc. I also had to increase some of the rootvg filesystems. You can run “df -g” and figure out if you need to do that.
For some reason it once again created two page files in rootvg. I usually have one 8GB page file there, so I do the following:
#lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum paging00 hdisk2 rootvg 1024MB 0 no no lv 0 hd6 hdisk2 rootvg 512MB 2 yes yes lv 0#lsvg rootvg | grep PP showed me the PPsize was 512MB for rootvg
So I increased hd6 to 8GB as follows:
#chps -s 15 hd6 and removed paging00 #rmps paging00 #lsps -a Page Space Physical Volume Volume Group Size %Used Active Auto Type Chksum hd6 hdisk2 rootvg 8192MB 0 yes yes lv 0I normally set fsize to -1 and nofiles to 8000 in the default section of /etc/security/limits and I change the herald in the default section of /etc/security/login.cfg to be:
herald = "Unauthorised access prohibited \nlogin: "
I change the herald and /etc/motd to something unfriendly so no hackers can ever say I invited them into the system.
The next step is to make sure the system dump logical volumes are big enough (which they are not):
#sysdumpdev -e 0453-041 Estimated dump size in bytes: 2456121507I have only lg_dumplv and it is too small:
(#lsvg -l rootvg | grep dump)So I create two new logical volumes:
mklv -t sysdump -y lv_dumplv1 rootvg 20 hdisk2 mklv -t sysdump -y lv_dumplv2 rootvg 20 hdisk2 sysdumpdev -KI then set those as the logical volumes to be used and remove the old one:
sysdumpdev -P -p /dev/lv_dumplv1 sysdumpdev -P -s /dev/lv_dumplv2 rmlv lg_dumplv/etc/syslog.conf also got reverted even though it was in my filestosave list, so I edit it and append the following to the end of it:
mail.debug /usr/local/logs/mailog rotate size 2m files 10 compress *.emerg /usr/local/logs/syslog rotate size 2m files 10 compress *.alert /usr/local/logs/syslog rotate size 2m files 10 compress *.crit /usr/local/logs/syslog rotate size 2m files 10 compress *.err /usr/local/logs/syslog rotate size 2m files 10 compress auth.notice /usr/local/logs/infolog rotate size 2m files 10 compress *.info /usr/local/logs/messages rotate size 2m files 10 compress cd /usr/local/logs touch mailog syslog infolog messagesThen I stop and start syslogd:
stopsrc -s syslogd startsrc -s syslogdI work on so many vio servers that I get confused, so I make the following change to /etc/environment:
PS1="vio1$: " EDITOR=viUse your VIO name instead of vio1, then log out and back in.
I also clean out /etc/inetd.conf so no dangerous programs are uncommented:
cd /etc cp inetd.conf inetd.conf-jl11262023 vi inetd.conf’I delete everything except:
#ftp stream tcp6 nowait root /usr/sbin/ftpd ftpd #telnet stream tcp6 nowait root /usr/sbin/telnetd telnetd -a #xmquery dgram udp6 wait root /usr/bin/xmtopas xmtopas -p3 caa_cfg stream tcp6 nowait root /usr/sbin/clusterconf clusterconf >>/var /adm/ras/clusterconf.log 2>&1 refresh -s inetdI also configure NTP if there is an ntp server:
vi /home/padmin/config/ntp.confAdd the following:
Is the IP address of your NTP server
If the date is way off, then run:
ntpdate ipofntpserver grep ntp /etc/rc.tcpipMake sure the ntp line is uncommented as follows:
start /usr/sbin/xntpd -a '-c /home/padmin/config/ntp.conf' "$src_running"Now I go and clean up and update Java, SSH amd SSL. You may need to recopy the patch and java, ssh and ssl directories back to this server:
lslpp -l | grep ava lslpp -l | grep ssh lslpp -l | grep sslThese show as:
Java8 188.8.131.520 SSL 184.108.40.2061 SSH 220.127.116.110I want to update these as there are many security holes that need to be fixed. Previously, I downloaded the latest Java, ssh and ssl and untarred them and copied them all into one directory. As padmin I now use that directory to update them:
$ updateios -accept -install -dev /usr/local/soft/javasshssl-v3-vio-nov152023This may not update all the language sets, so I remove those:
updateios -remove openssh.msg.CA_ES updateios -remove openssh.msg.CS_CZ updateios -remove openssh.msg.DE_DE updateios -remove openssh.msg.ES_ES updateios -remove openssh.msg.FR_FR updateios -remove openssh.msg.HU_HU updateios -remove openssh.msg.IT_IT updateios -remove openssh.msg.JA_JP updateios -remove openssh.msg.Ja_JP updateios -remove openssh.msg.KO_KR updateios -remove openssh.msg.PL_PL updateios -remove openssh.msg.PT_BR updateios -remove openssh.msg.RU_RU updateios -remove openssh.msg.SK_SK updateios -remove openssh.msg.ZH_CN updateios -remove openssh.msg.ZH_TW updateios -remove openssh.msg.Zh_CN updateios -remove openssh.msg.Zh_TW updateios -remove openssh.msg.ca_ES updateios -remove openssh.msg.cs_CZ updateios -remove openssh.msg.de_DE updateios -remove openssh.msg.es_ES updateios -remove openssh.msg.fr_FR updateios -remove openssh.msg.hu_HU updateios -remove openssh.msg.it_IT updateios -remove openssh.msg.ja_JP updateios -remove openssh.msg.ko_KR updateios -remove openssh.msg.pl_PL updateios -remove openssh.msg.pt_BR updateios -remove openssh.msg.ru_RU updateios -remove openssh.msg.sk_SK updateios -remove openssh.msg.zh_CN updateios -remove openssh.msg.zh_TW oem_setup_env lslpp -l | grep ava lslpp -l | grep ssh lslpp -l | grep sslYou should now see:
SSH 18.104.22.1680 SSL 22.214.171.1241 Java 126.96.36.1991Finally there are just a couple of patches to go on. But first we check some levels (as root):
#lslpp -L | grep rpm.rte rpm.rte 188.8.131.521 C F RPM Package Manager #lslpp -L invscout.rte invscout.rte 184.108.40.206 C F Inventory Scout Runtime #slpp -L perl.rte perl.rte 220.127.116.11 C F Perl Version 5 Runtime #lslpp -l | grep -i python python3.9.base 18.104.22.168I copied the Python patches into /usr/local/soft/flrtfixes/vioflrt
Then as padmin:
$updateios -accept -install -dev /usr/local/soft/flrtfixes/vioflrtThere were 2 to go on:
$oem_setup_env #lslpp -l | grep -i python python3.9.base 22.214.171.124 COMMITTED Python 3.9 64-bit binary python3.9.test 126.96.36.199 COMMITTED Python 3.9 64-bit self-testThere is one openssh patch but it would not install with updateios so I had to resort to emgr.
#cd /usr/local/soft/flrtfixes/openssh_fix15 #emgr -p -e 38408m9c.230811.epkg.Z The above runs it in preview mode #emgr -e 38408m9c.230811.epkg.ZThe above runs it for real.
#emgr -P PACKAGE INSTALLER LABEL ======================================================== =========== ========== openssh.base.client installp 38408m9c openssh.base.server installp 38408m9cNow I run flrtvc:
cd /usr/local/soft/flrtvc ./flrtvc-086.ksh No additional patches were found.Finally, I run:
#updtvpkgI do this because I have updated rpm, the operating system and ssl.
I check that nothing is missing from crontab (I usually have backups and I start nmon in there). So, I normally have something like:
59 23 * * * /usr/local/bin/runnmon.sh >/dev/null 2>&1 30 2 1 * * /usr/local/bin/viobackup.sh >/dev/null 2>&1 30 2 16 * * /usr/local/bin/viobackup2.sh >/dev/null 2>&1As admin I also restart my automatic viosbr backups.
As padmin run:
viosbr -backup -file vio1-viobkup -frequency daily -numfiles 7(Replace vio1 above with the real vio name)
I now rewrite the boot image and the bootlist then reboot:
#lspv | grep root #bootinfo -b #bosboot -a -d hdisk2 #bootlist -m normal -o #bootlist -m normal hdisk2 #bootlist -m normal -oIf you had set any tunables (network, buffers, changes to fiber card settings, etc.) then you should rerun those now.
And finally, we reboot.
Once the system comes back up, I run all my checks including errpt. I also log in to the client LPARs and check that all their paths have come back. Only when I am happy with all of that do I go work on the primary VIO server and go through the same process again.
Finally, if you need to recreate the repository go ahead and do so, and copy the files back into it.
And you should always end by taking a mksysb backup and a fresh hmcscanner report. If you mirror your rootvg then wait at least a week before remirroring.
PowerVM v188.8.131.52 Upgrade SummaryI know this seems like a long and arduous process, but I prefer running it this way so I can run checks at every step. You can automate a lot of this with NIM, and you also don’t have to customize the VIO the way I do. I do not like putting logs into /var as they can fill up /var, which is a bad thing. So, I always have separate /usr/local and /usr/local/logs filesystems where I can store logs, nmon output, local scripts, etc. I am also a big fan of putting on efixes and ifixes so my VIO servers are secure.
If you want more information, I recommend checking out the YouTube video below that Andrey Klyachkin (Power DevOps) just uploaded.
viosupgrade command for the VIO
PowerVM 4.1 Upgrade on YouTube by Power DevOps
Readme for PowerVM 184.108.40.206
Readme for PowerVM 220.127.116.11
IBM Web Download Page: Download OpenSSH, openssl and the Python3 patch from here.
ESS: Download the PowerVM ISO image from here.
Fix Central: Download Java8 18.104.22.1681 from here (or higher as needs be).
openSSH_fix15 tar file
FLRTVC 086 ZIP
FLRTLITE and the FLRT Data Tables
About the author
Jaqui Lynch has over 38 years of experience working with a projects and OSes across vendor platforms, including IBM Z, UNIX systems and more.
See more by Jaqui Lynch