How to Upgrade to PowerVM v4.1.0.10

Jaqui Lynch December 8, 2023

In this article I will go over my recent experience upgrading VIO servers from version 3 to version 4. The process is similar to going from version 2 to version 3, but I did run into a few issues that you should be able to avoid. The first step is to download the code and the associated patches. Then I doublecheck the readme and make sure the VIO server meets all the prerequisites and has no current errors. Then I do my documentation and take backups before starting the upgrade.

AIX, PowerVM, Etc. Base Code

The first step is to download the code. This is found at ESS (Entitled Systems Support). You will find the software base code, update access keys, inventory explorer and other useful items here. I have a separate filesystem called /usr/local in rootvg on my VIO servers. This is where I put code that I want to install if I don’t have access to a NIM LPAR or NFS mounts.

Get the v4.1.0.10 ISO image from ESS:

5765-VE4  Virtual_IO_Server_Base_Install_4.1.0.10_Flash_112023_LCD8292400.iso

Upload it to the VIO into mkdir /usr/local/soft/powervm41

Now mount it:

loopmount -i /usr/local/soft/powervm41/Virtual_IO_Server_Base_Install_4.1.0.10_Flash_112023_LCD8292400.iso -o "-V udfs -o ro" -m /cdrom

Extract the mksysb image for later:

cp /cdrom/usr/sys/inst.images/mksysb_image  /usr/local/soft/powervm41/mksysb_vio41010_2023
umount /cdrom

You will also need to download the following from IBM:

1. Latest Java8 8.0.0.811 from Fix Central
2. openSSH_fix15 from aix.software.ibm.com
3. Python3, latest openssl and openSSH from IBM Web Download Page
4.PowerVM 3.1.4.31 updates if you are not at 3.1.4.31

I also have the latest FLRTVC and HMCScanner installed, and I downloaded the VIOS 3.1.4.31 updates from Fix Central—I selected include all prerequisites when I downloaded it.

VIOS 3.1.4.31 Prerequisites

IBM recommends you upgrade to v3.1.4.30 prior to upgrading to v4. V3.1.4.30 is no longer available so the upgrade would be to v3.1.4.31. If you are using NIM PowerVM 3.1.4.31, it has a prerequisite of AIX 7.2.5.7, and PowerVM 4.1.0.10 requires AIX 7.3.2.1, so I recommend upgrading your NIM server to 7.3.2.1 if it is running on hardware that supports that level (POWER8 or higher).

PowerVM 3.1.4.31 and 4.1.0.10 can run on any server that is POWER8 or higher. If you are still on POWER7 then you need to be on v3.1.2 of the VIO.

The VIOS update 3.1.4.31 can be applied to any VIOS that is at 3.1.0.00 or higher, but prior to the upgrade make sure there Is at least 4GB of free space in rootvg.

The VIOS to NIM mapping page has not been updated so you will need to refer to the Readme files, which is what I did.

You should also check that your current firmware and HMC levels are supported by the levels you are going to. You can find this at the flrtlite site or by running FLRT. There are some prerequisites in the README for certain adapters so make sure your I/O adapter firmware code is up to date prior to upgrading.

Make sure any media images are unloaded:

$lsvopt

If it shows something loaded then unload it as follows replacing ?? with the correct number.

$ unloadopt -vtd vtopt??

Always start by running errpt to check for errors. You do not want to try to update a failing system or one that has errors. Additionally, if you are mirroring rootvg you will need to unmirror it or have two spare disks for the upgrade. I also check all my client LPARs before and after each VIO update to make sure they get all their paths back before I go on to the next update.

PowerVM 3.1.4.31 Process

Before I start, I always document everything on the VIO servers and take copies of important files such as:

/etc/ssh/sshd_config
/etc/inetd.conf
/etc/inittab
/etc/rc.tcpip
/etc/resolv.conf
/home/padmin/config/ntp.conf
/etc/hosts
/etc/motd

I find the following commands useful for documentation and I save the output to an external file (usually a .txt file on my PC).
As padmin:

ioslevel
lsnports
lspv -size
lspv -free
lspv
lsrep
lsvopt
lsmap -all -npiv | grep vfc
lsmap -all -npiv | grep fcs
lsmap -all | grep vhost

As root (oem_setup_env)iIfconfig -a

 lspv
lspv | grep root
bootinfo -b
bootlist -m normal -o
ifconfig -a

Make a note of any interfaces with IP addresses and document those plus the gateway and subnet masks for them.

oslevel -s
instfix -i | grep ML
lppchk -v
lppchk -vm3
lsdev -C | grep fcs
lscfg -vpl fcs* | grep Network
lscfg -vpl fcs* | grep fcs
lsdev -C | grep ent
lsdev -C | grep Shared

Note the ent number it returns and use it in the next command instead of ent6.

lsattr -El ent6 | grep ent
           
lsmcode -A

In my case the VIO was already at 3.1.4.21, so the steps I followed for the update to v3.1.4.31 were:

Download the patches to /usr/local/soft/powervm31431
Take a clone:

#lspv | grep root

Note which disk is altinst_rootvg and replace ?? below
exportvg altinst_rootvg

#alt_disk_copy -V -B -d hdisk??

Also take a mksysb type backup
Get an HMCScanner report
Run “#emgr -P” to look for any efixes
Remove the efixes using emgr

#emgr -r -L patch????

i.e., “#emgr -r -L 38408m9c”

Use updateios to do updates:

$updateios -commit
$updateios -accept -install -dev /usr/local/soft/powervm31431

In my case there were 1 (bos.rte.install) plus197 updates to go on.

Before rebooting, run bosboot to rewrite the boot image and use bootlist to rewrite the bootlist – assuming hdisk3 is rootvg, then:

#updtvpkg

You should run updtvpkg any time you update the operating system, or SSL or rpm:

#bosboot -a -d hdisk3
#bootlist -m normal -o
#bootlist -m normal hdisk3
#bootlist -m normal -o

Then it is time to reboot and run for a couple of days (this means you can restore to clone if you need to).

If you don’t wait, then viosupgrade will overwrite your clone and you can’t go back to 3.1.4.21 so any recovery would be from your mksysb image.

PowerVM 4.1.0.10 Process

If you make use of the repository (look for /var/vio/VMLibrary) then you will need to take a copy to an external system if it is in rootvg. The v4 upgrade will wipe it out. You are now ready to do the actual upgrade. Feel free to take another mksysb beforehand if you want to.

#lspv | grep root
hdisk2          00c47b30939456b9                    altinst_rootvg
hdisk3          00c47b3050756642                    rootvg          active

Above you can see that altinst_rootvg is on hdisk2. altinst_rootvg cannot exist during the upgrade or the upgrade will fail, so we need to remove and clear it:

#exportvg altinst_rootvg
#chpv -c hdisk2
#chpv -C hdisk2
#lspv
hdisk2          00c47b30939456b9                    None
hdisk3          00c47b3050756642                    rootvg          acti

Now as padmin check the disk is free:

$lspv -size
NAME            PVID                                SIZE(megabytes)
hdisk2          00c47b30939456b9                    270648
hdisk3          00c47b3050756642                    270648
 
$lspv -free
NAME            PVID                                SIZE(megabytes)
hdisk2          00c47b30939456b9                    270648

If the disk does not show as free, then your upgrade will fail.

I create /home/padmin/filestosave.txt with the following contents. That way it will back these files up during the upgrade into a directory in /home/padmin. This will allow you to compare files later.

/etc/environment
/etc/group
/etc/hosts
/etc/inetd.conf
/etc/inittab
/etc/motd
/etc/netsvc.conf
/etc/passwd
/etc/profile
/etc/resolv.conf
/etc/syslog.conf
/etc/security/limits
/etc/security/login.cfg
/etc/security/passwd
/etc/tunables/nextboot
/etc/tunables/rc.tunevio.sh
/etc/tunables/rc.tunebufs.sh
/usr/local/bin/runnmon.sh
/etc/ssh/sshd_config
/home/padmin/.profile
/home/padmin/filestosave.txt
/etc/ntp.conf
/etc/rc.tcpip
/home/padmin/config/ntp.conf

I am running viosupgrade on the VIO server—you can also do this from NIM but the syntax is different. If you have any problems, you can look at:

/var/adm/ras/ioslogs/viosupg_global.log

Earlier, I copied the mksysb into:

/usr/local/soft/powervm41/mksysb_vio41010_2023.

Check again that it is there.

Make sure there are no NFS mounted filesystems and disable the automatic viosbr backup if you use it:

$viosbr -nobackup

If you don’t unmount the NFS filesystems it will try to copy them during the upgrade. If you don’t disable the viosbr backup, then you may see the following error:

Unmatched (in regex; marked by <– HERE in m/( <– HERE /usr/ios/cli/ioscli/ at /usr/ios/sbin/viosupg.pl line 10846.

If everything looks good, then it is time to run the upgrade:

#viosupgrade -l -i /usr/local/soft/mksysb_vio41010_2023 -a hdisk2 -g /home/padmin/filestosave.txt

The above tells it to run the upgrade to hdisk2 (which we cleared earlier) using the mksysb we copied and using our filestosave.txt file to figure out which extra files we want backed up in /home/padmin.

You will now see a bunch of messages similar to:

Welcome to viosupgrade tool.
Operation triggered for given node(s).
Broadcast message from root (vty0) at 16:38:01 ...
WARNING!!! VIOS Upgrade operation is in progress. Kindly Refrain from making any configuration changes...
Please wait for completion..
Upgrading from ioslevel '3.1.4.31' to '4.1.0.10'.
Verifying whether the MPIO software(s) is installed on the VIOS.
Verification of the MPIO software(s) is successful.
Initiating VIOS configuration backup..
VIOS configuration backup successful.
Migration upgrade request initiated.
Warning: User LV(s) 'lv_dumplv1
lv_dumplv2' created on rootvg will not be preserved.
Enter 'Y' to continue or 'N' to abort the viosupgrade.
If no option is entered within 60 seconds, the operation will continue..

You should enter y

y
Initiating installation on alternate disk(s)..
Migration of contents to new rootvg initiated.
Installation on alternate disk(s) successful.
Copying files to altinst_rootvg.
Waking up altinst_rootvg successful.

Lots more messages then:
Note: Backup of migrated files of old vg and new vg will be in ‘/home/padmin/viosupg_backup’ with extensions ‘_src.viosupg’ and ‘_dest.viosupg’ respectively.
VIOS will be rebooted after ’60’ seconds to boot from the newly installed disk.

Press Ctrl + c to terminate.

If you hit ctrl-c here it stops. Otherwise it will continue as follows:

VIOS metadata restore (viosbr -restore) will be automatically resumed after the reboot.

VIOS may be rebooted once during this restore process. Refrain from making any changes to the VIOS virtual configurations during the restore process.

You can verify the restore status using ‘viosupgrade -l -q’ command and resume your operation after the completion of the restore process.

A message ‘Upgrade process is completed successfully’ or ‘Upgrade process is completed with failures’ will be displayed on the console on newvg to indicate the status of viosupgrade.

Run ‘viosupgrade -l -q’ to know more on the status.

After about 20 minutes it should be done, and it then reboots. Wait at least 15 more minutes as it will reboot at least 3 times to restore the virtual settings, etc. When you log in as padmin it may or may not prompt you to set the password. In my case it preserved the password, which was nice. You will then need to accept the license:

Indicate by selecting the appropriate response below whether you accept or decline the software maintenance terms and conditions:

Accept (a) | Decline (d) | View Terms (v) > a

I typed in “a” above and then also did the following:

$license -accept

ioslevel should now show 4.1.0.10

Now run:

viosupgrade -l -q

This checks the status of the upgrade and may or may not cause it to reboot. The last message you should see is:
Upgrade process is completed successfully

At this point you should check all your virtual settings as follows:

As padmin
$lsnports
$lsrep
$lsmap -all -npiv | grep vfc
$lsmap -all -npiv | grep fcs
$lsmap -all | grep vhost
$lsvopt
 
As root
#lsdev -C | grep ent
#lsdev -C | grep Shared
#lspv
#ifconfig -a

They should match what you had before.

#oslevel -s

This should show as 7300-02-01-2346

Run lspv to check your disks, as they may have been renumbered.

#lspv
hdisk2          00c47b30939456b9                    rootvg          active
hdisk3          00c47b3050756642                    old_rootvg

Also check df -g. I found that my /usr/local and /usr/local/logs that I had in rootvg had been brought across, which was nice, as I did not have to copy all my scripts and log file back over.

Customizing

Any customizations that you had done will have been lost. This includes changes to /etc/syslog.conf, paging, etc. I also had to increase some of the rootvg filesystems. You can run “df -g” and figure out if you need to do that.

For some reason it once again created two page files in rootvg. I usually have one 8GB page file there, so I do the following:

#lsps -a
Page Space      Physical Volume   Volume Group    Size %Used   Active    Auto    Type   Chksum
paging00        hdisk2            rootvg        1024MB     0      no      no      lv       0
hd6             hdisk2            rootvg         512MB     2     yes     yes      lv       0

#lsvg rootvg | grep PP showed me the PPsize was 512MB for rootvg
So I increased hd6 to 8GB as follows:

#chps -s 15 hd6

and removed paging00

#rmps paging00
#lsps -a
Page Space      Physical Volume   Volume Group    Size %Used   Active    Auto    Type   Chksum
hd6             hdisk2            rootvg        8192MB     0     yes     yes      lv       0

I normally set fsize to -1 and nofiles to 8000 in the default section of /etc/security/limits and I change the herald in the default section of /etc/security/login.cfg to be:

herald = “Unauthorised access prohibited nlogin: “

I change the herald and /etc/motd to something unfriendly so no hackers can ever say I invited them into the system.

The next step is to make sure the system dump logical volumes are big enough (which they are not):

#sysdumpdev -e
0453-041 Estimated dump size in bytes: 2456121507

I have only lg_dumplv and it is too small:

(#lsvg -l rootvg | grep dump)

So I create two new logical volumes:

mklv -t sysdump -y lv_dumplv1 rootvg 20 hdisk2
mklv -t sysdump -y lv_dumplv2 rootvg 20 hdisk2
sysdumpdev -K

I then set those as the logical volumes to be used and remove the old one:

sysdumpdev -P -p /dev/lv_dumplv1
sysdumpdev -P -s /dev/lv_dumplv2
rmlv lg_dumplv

/etc/syslog.conf also got reverted even though it was in my filestosave list, so I edit it and append the following to the end of it:

mail.debug      /usr/local/logs/mailog rotate size 2m files 10 compress
*.emerg         /usr/local/logs/syslog rotate size 2m files 10 compress
*.alert         /usr/local/logs/syslog rotate size 2m files 10 compress
*.crit          /usr/local/logs/syslog rotate size 2m files 10 compress
*.err           /usr/local/logs/syslog rotate size 2m files 10 compress
auth.notice     /usr/local/logs/infolog rotate size 2m files 10 compress
*.info          /usr/local/logs/messages rotate size 2m files 10 compress
 
cd /usr/local/logs
touch mailog syslog infolog messages

Then I stop and start syslogd:

stopsrc -s syslogd
startsrc -s syslogd

I work on so many vio servers that I get confused, so I make the following change to /etc/environment:

PS1="vio1$: "
EDITOR=vi

Use your VIO name instead of vio1, then log out and back in.

I also clean out /etc/inetd.conf so no dangerous programs are uncommented:

cd /etc
cp inetd.conf inetd.conf-jl11262023
vi inetd.conf’

I delete everything except:

#ftp     stream  tcp6    nowait  root    /usr/sbin/ftpd         ftpd
#telnet  stream  tcp6    nowait  root    /usr/sbin/telnetd      telnetd -a
#xmquery        dgram   udp6    wait    root    /usr/bin/xmtopas xmtopas -p3
caa_cfg stream  tcp6    nowait  root    /usr/sbin/clusterconf clusterconf >>/var
/adm/ras/clusterconf.log 2>&1
 
refresh -s inetd

I also configure NTP if there is an ntp server:

vi /home/padmin/config/ntp.conf

Add the following:

Server ipofntpserver
Where ipofntpserver
Is the IP address of your NTP server
i.e. 192.168.2.100

If the date is way off, then run:

ntpdate ipofntpserver
 
grep ntp /etc/rc.tcpip

Make sure the ntp line is uncommented as follows:

start /usr/sbin/xntpd -a '-c /home/padmin/config/ntp.conf' "$src_running"

Now I go and clean up and update Java, SSH amd SSL. You may need to recopy the patch and java, ssh and ssl directories back to this server:

lslpp -l | grep ava
lslpp -l | grep ssh
lslpp -l | grep ssl

These show as:

Java8             8.0.0.800
SSL                3.0.10.1001
SSH               8.1.112.2000

I want to update these as there are many security holes that need to be fixed. Previously, I downloaded the latest Java, ssh and ssl and untarred them and copied them all into one directory. As padmin I now use that directory to update them:

$ updateios -accept -install -dev /usr/local/soft/javasshssl-v3-vio-nov152023

This may not update all the language sets, so I remove those:

As padmin:

updateios -remove openssh.msg.CA_ES
updateios -remove openssh.msg.CS_CZ
updateios -remove openssh.msg.DE_DE
updateios -remove openssh.msg.ES_ES
updateios -remove openssh.msg.FR_FR
updateios -remove openssh.msg.HU_HU
updateios -remove openssh.msg.IT_IT
updateios -remove openssh.msg.JA_JP
updateios -remove openssh.msg.Ja_JP
updateios -remove openssh.msg.KO_KR
updateios -remove openssh.msg.PL_PL
updateios -remove openssh.msg.PT_BR
updateios -remove openssh.msg.RU_RU
updateios -remove openssh.msg.SK_SK
updateios -remove openssh.msg.ZH_CN
updateios -remove openssh.msg.ZH_TW
updateios -remove openssh.msg.Zh_CN
updateios -remove openssh.msg.Zh_TW
updateios -remove openssh.msg.ca_ES
updateios -remove openssh.msg.cs_CZ
updateios -remove openssh.msg.de_DE
updateios -remove openssh.msg.es_ES
updateios -remove openssh.msg.fr_FR
updateios -remove openssh.msg.hu_HU
updateios -remove openssh.msg.it_IT
updateios -remove openssh.msg.ja_JP
updateios -remove openssh.msg.ko_KR
updateios -remove openssh.msg.pl_PL
updateios -remove openssh.msg.pt_BR
updateios -remove openssh.msg.ru_RU
updateios -remove openssh.msg.sk_SK
updateios -remove openssh.msg.zh_CN
updateios -remove openssh.msg.zh_TW
 
oem_setup_env
lslpp -l | grep  ava
lslpp -l | grep ssh
lslpp -l | grep ssl

You should now see:

SSH
            9.2.112.2000
SSL
            3.0.10.1001
Java
            8.0.0.811

Finally there are just a couple of patches to go on. But first we check some levels (as root):

#lslpp -L | grep rpm.rte
 rpm.rte                 4.18.1.2001    C     F    RPM Package Manager
 
#lslpp -L invscout.rte
  invscout.rte              2.2.0.25    C     F    Inventory Scout Runtime
 
#slpp -L perl.rte
 perl.rte                  5.34.1.4    C     F    Perl Version 5 Runtime
 
#lslpp -l | grep -i python
python3.9.base            3.9.17.1

I copied the Python patches into /usr/local/soft/flrtfixes/vioflrt
Then as padmin:

$updateios -accept -install -dev /usr/local/soft/flrtfixes/vioflrt

There were 2 to go on:

$oem_setup_env
#lslpp -l | grep -i python
  python3.9.base            3.9.18.0  COMMITTED  Python 3.9 64-bit binary
  python3.9.test            3.9.18.0  COMMITTED  Python 3.9 64-bit self-test

There is one openssh patch but it would not install with updateios so I had to resort to emgr.

As root:

#cd /usr/local/soft/flrtfixes/openssh_fix15
#emgr -p -e  38408m9c.230811.epkg.Z
The above runs it in preview mode
#emgr -e  38408m9c.230811.epkg.Z

The above runs it for real.

#emgr -P
PACKAGE                                                  INSTALLER   LABEL
======================================================== =========== ==========
openssh.base.client                                      installp    38408m9c
openssh.base.server                                      installp    38408m9c

Now I run flrtvc:

cd /usr/local/soft/flrtvc
./flrtvc-086.ksh
No additional patches were found.

Finally, I run:

#updtvpkg

I do this because I have updated rpm, the operating system and ssl.

I check that nothing is missing from crontab (I usually have backups and I start nmon in there). So, I normally have something like:

59 23  * * * /usr/local/bin/runnmon.sh >/dev/null 2>&1
30 2  1 * * /usr/local/bin/viobackup.sh >/dev/null 2>&1
30 2  16 * * /usr/local/bin/viobackup2.sh >/dev/null 2>&1

As admin I also restart my automatic viosbr backups.

As padmin run:

viosbr -backup -file vio1-viobkup -frequency daily -numfiles 7

(Replace vio1 above with the real vio name)

I now rewrite the boot image and the bootlist then reboot:

As root:

#lspv | grep root
#bootinfo -b
#bosboot -a -d hdisk2
#bootlist -m normal -o
#bootlist -m normal hdisk2
#bootlist -m normal -o

If you had set any tunables (network, buffers, changes to fiber card settings, etc.) then you should rerun those now.

And finally, we reboot.

Once the system comes back up, I run all my checks including errpt. I also log in to the client LPARs and check that all their paths have come back. Only when I am happy with all of that do I go work on the primary VIO server and go through the same process again.

Finally, if you need to recreate the repository go ahead and do so, and copy the files back into it.

And you should always end by taking a mksysb backup and a fresh hmcscanner report. If you mirror your rootvg then wait at least a week before remirroring.

PowerVM v4.1.0.10 Upgrade Summary

I know this seems like a long and arduous process, but I prefer running it this way so I can run checks at every step. You can automate a lot of this with NIM, and you also don’t have to customize the VIO the way I do. I do not like putting logs into /var as they can fill up /var, which is a bad thing. So, I always have separate /usr/local and /usr/local/logs filesystems where I can store logs, nmon output, local scripts, etc. I am also a big fan of putting on efixes and ifixes so my VIO servers are secure.

If you want more information, I recommend checking out the YouTube video below that Andrey Klyachkin (Power DevOps) just uploaded.