Upgrading to PowerVM v4.1.0.21

IBM Champion Jaqui Lynch highlights her experience upgrading PowerVM from v4.1.0.10 to v4.1.0.21

Recently I decided to upgrade my VIO servers from 4.1.0.10 and wanted to share that experience with you. If you are on v3.1, then you would use the viosupgrade tool to upgrade as I discussed in my article on PowerVM v4 experiences. The only difference is that you would use the 4.1.0.21 flash image that you download from ESS (Entitled Systems Support). In this case I downloaded the fixpack for 4.1.0.21 from Fix Central. I also downloaded the flash image from ESS so I could build the NIM resources for VIO server restores. As a note, 4.1.0.21 requires that your NIM server be at 7.3.2.2, which mine was. 4.1.0.21 is supported only on POWER8 servers and above.

The first step after downloading the fixpack is to read the readme which can be found here. After that, I built the NIM resources for the new version.

Building the NIM Resources

On my NIM server I keep the initial images in /software/powervm41, so I upload the iso image to there.

cd /software/powervm41

The image is called:

Virtual_IO_Server_Base_Install_4.1.0.21_Flash_062024_LCD8292401.iso

I then use loopmount to mount the image so I can copy the code for NIM to use

mkdir /software/powervm41/powervm41021-base
loopmount -i /software/powervm41/Virtual_IO_Server_Base_Install_4.1.0.21_Flash_062024_LCD8292401.iso -o "-V udfs -o ro" -m /cdrom

Here I copy the mksysb image so NIM can use it to build the SPOT

cp /cdrom/usr/sys/inst.images/mksysb_image  /software/powervm41/mksysb_vio41021_2024
cp -R /cdrom/*  /software/powervm41/powervm41021-base
umount /cdrom
cp  /software/powervm41/mksysb_vio41021_2024  /nim/images

Now I build the resources:

1. Create a mksysb nim resource called mksysb_vio41021_2024 from /nim/images/mksysb_vio41021_2024.

• Now create the SPOT from the mksysb:

nim -o define -t spot -a server=master -a source=mksysb_vio41021_2024 -a location=/nim/spot spotvios41021

• Then check the resources:

nim -o check spotvios41021
lsnim -l spotvios41021

Now that the NIM resources are created you can go ahead with the update.

Performing the Fixpack Update

The first step is to download the code. Since I am at 4.1.0.10 I use Fix Central to download the updates and I save them in /software/powervm41/powervm41021-updates.

You will also need to download the following from IBM:

Latest Java8 8.0.0.826 from Fix Central

Patches from https://public.dhe.ibm.com/aix/efixes/security/

openSSH_fix17.tar

curl_fix5.tar

invscout_fix26.tar

Python3, latest openssl and openSSH from IBM Web Download Page

Python3         3.9.19.0

SSH                9.2.112.2400

SSL                3.0.13.1000

You should also download the latest perl-5.34.1.6 and rpm 4.18.1.2003 from the IBM web download site.

Make sure you have the latest flrtvc (0810) and HMCScanner (0.11.54) downloaded:

flrtvc

HMCScanner

When I downloaded the VIOS 4.1.0.21 updates from Fix Central, I selected include all prerequisites.

You should also check that your current firmware and HMC levels are supported by the levels you are going to. You can find this at the flrtlite site or by running FLRT. There are some prerequisites in the README for certain adapters so make sure your I/O adapter firmware code is up to date prior to upgrading.

Make sure any media images are unloaded:

$lsvopt

If it shows something loaded, then unload it as follows, replacing ?? with the correct number.

$ unloadopt -vtd vtopt??

Always start by running errpt to check for errors. You do not want to try to update a failing system or one that has errors. Additionally, if you are mirroring rootvg you will need to unmirror it or have two spare disks for the upgrade. I also check all my client LPARs before and after each VIO update to make sure they get all their paths back before I go on to the next update.

PowerVM 4.1.0.21 Process

Before I start, I always document everything on the VIO servers and take copies of important files such as:

/etc/ssh/sshd_config

/etc/ssh/ssh_config

/etc/inetd.conf

/etc/inittab

/etc/rc.tcpip

/etc/resolv.conf

/home/padmin/config/ntp.conf

/etc/hosts

/etc/motd

I find the following commands useful for documentation and I save the output to an external file (usually a .txt file on my PC).

As padmin
ioslevel
lsnports
lspv -size
lspv -free
lspv
lsrep
lsvopt
lsmap -all -npiv | grep vfc
lsmap -all -npiv | grep fcs
lsmap -all | grep vhost

As root (oem_setup_env)
ifconfig -a
lspv
lspv | grep root
bootinfo -b
bootlist -m normal -o
ifconfig -a

Make a note of any interfaces with IP addresses and document those plus the gateway and subnet masks for them.

oslevel -s
instfix -i | grep ML
lppchk -v
lppchk -vm3
lsdev -C | grep fcs
lscfg -vpl fcs* | grep Network
lscfg -vpl fcs* | grep fcs
lsdev -C | grep ent
lsdev -C | grep Shared

Note the ent number it returns and use it in the next command instead of ent6.

lsattr -El ent6 | grep ent
	
lsmcode -A

In my case the VIO was already at 4.1.0.10, so the steps I followed for the update to 4.1.0.21 were:

Download the patches to /soft/powervm41.

Download the security updates to /software/flrtfixes and untar them.

Combine the ssh, java and ssl updates into one directory - /software/javasshssl-v3-vio-jun122024.

Combine the ssh, invscout and curl patches into /software/flrtfixes/vioflrt.

Take a clone:           

#lspv | grep root

Note which disk is altinst_rootvg and replace ?? below:

exportvg altinst_rootvg
#alt_disk_copy -V -B -d hdisk??

Also take a mksysb type backup.

Get an HMCScanner report.

Run “#emgr -P” to look for any efixes.

Remove the efixes using emgr.

#emgr -r -L patch????

I.e., “#emgr -r -L 38408m9c”

Use updateios to do updates:

$updateios -commit

$updateios -accept -install -dev /software/javasshssl-v3-vio-jun122024
	4 to go on
$updateios -accept -install -dev /software/powervm41/ powervm41021-updates
	2 to go on
	Then 123 to go on
$updateios -commit

Now, apply the patches. You can do this one of two ways:

1. $updateios -accept -install -dev /software/ flrtfixes/vioflrt
2. Use emgr is any of the above patches fail.

lslpp -L invscout.rte
invscout.rte              2.2.0.26    C     F    Inventory Scout Runtime
cd /software/flrtfixes/invscout_fix6
emgr -p -e is22026s1a.240514.epkg.Z
emgr -e is22026s1a.240514.epkg.Z

lslpp -L perl.rte
If not at 5.34.1.6:
cd /software/flrtfixes/perl-534
installp -apYd . perl.rte
installp -aXYd . perl.rte

lslpp -l | grep -i python
python3.9.base            3.9.18.1  COMMITTED  Python 3.9 64-bit binary
cd /software/flrtfixes/python-3.9.19.0
installp -apYd . python*
installp -aXYd . python*
lslpp -l | grep -i python
  python3.9.base            3.9.19.0  COMMITTED  Python 3.9 64-bit binary

cd /software/flrtfixes/curl_fix5
emgr -p -e 853sa.240503.epkg.Z 
emgr  -e 853sa.240503.epkg.Z 


cd /software/flrtvc
./flrtvc-0810-nodl.ksh

No problems should be found.

Now, run emgr -P and you should see:

PACKAGE                                                  INSTALLER   LABEL
======================================================== =========== ==========
invscout.rte                                             installp    is22026s1a
openssl.base                                             installp    301002sa
oss.lib.libcurl                                          installp    853sa
openssh.base.client                                      installp    92112ma
openssh.base.server                                      installp    

#updtvpkg

You should run updtvpkg any time you update the operating system, or SSL or rpm.

Run Your Checks Then Reboot

oslevel -s
7300-02-02-2420

instfix -i | grep ML
    All filesets for 7.3.0.0_AIX_ML were found.
    All filesets for 7300-00_AIX_ML were found.
    All filesets for 7300-01_AIX_ML were found.
    All filesets for 7300-02_AIX_ML were found.
oslevel -s -l 7300-02-02-2420
lppchk -v
lppchk -vm3
instfix -icqk  7300-00_AIX_ML | grep :-:
instfix -icqk  7300-01_AIX_ML | grep :-:
instfix -icqk  7300-02_AIX_ML | grep :-:

lslpp -l | grep ssh
lslpp -l | grep ssl
lslpp -l | grep Java

These should show:

SSH                9.2.112.2400

Java                8.0.0.826

SSL                3.0.13.1000

The SSH update may not update all the language sets, so I remove those:

As padmin:
updateios -remove openssh.msg.CA_ES
updateios -remove openssh.msg.CS_CZ
updateios -remove openssh.msg.DE_DE
updateios -remove openssh.msg.ES_ES
updateios -remove openssh.msg.FR_FR
updateios -remove openssh.msg.HU_HU
updateios -remove openssh.msg.IT_IT
updateios -remove openssh.msg.JA_JP
updateios -remove openssh.msg.Ja_JP
updateios -remove openssh.msg.KO_KR
updateios -remove openssh.msg.PL_PL
updateios -remove openssh.msg.PT_BR
updateios -remove openssh.msg.RU_RU
updateios -remove openssh.msg.SK_SK
updateios -remove openssh.msg.ZH_CN
updateios -remove openssh.msg.ZH_TW
updateios -remove openssh.msg.Zh_CN
updateios -remove openssh.msg.Zh_TW
updateios -remove openssh.msg.ca_ES
updateios -remove openssh.msg.cs_CZ
updateios -remove openssh.msg.de_DE
updateios -remove openssh.msg.es_ES
updateios -remove openssh.msg.fr_FR
updateios -remove openssh.msg.hu_HU
updateios -remove openssh.msg.it_IT
updateios -remove openssh.msg.ja_JP
updateios -remove openssh.msg.ko_KR
updateios -remove openssh.msg.pl_PL
updateios -remove openssh.msg.pt_BR
updateios -remove openssh.msg.ru_RU
updateios -remove openssh.msg.sk_SK
updateios -remove openssh.msg.zh_CN
updateios -remove openssh.msg.zh_TW

Before rebooting run bosboot to rewrite the boot image and use bootlist to rewrite the bootlist – assuming hdisk3 is rootvg then:

#bosboot -a -d hdisk3
#bootlist -m normal -o
#bootlist -m normal hdisk3
#bootlist -m normal -o

Once the system comes back up, I run all my checks including errpt. I also log in to the client LPARs and check that all their paths have come back. Only when I am happy with all of that do I go work on the primary VIO server and go through the same process again.

And you should always end by taking a mksysb backup and a fresh hmcscanner report. If you mirror your rootvg then wait at least a week before remirroring.

Gotchas

On these VIO servers I had gone through the issues associated with using viosupgrade to upgrade to 4.1.0.10 so I had put on the patches to avoid the padmin login issues, and I had updated /etc/security/user to ensure padmin was set as follows:

padmin:
        admin = false
        default_roles = PAdmin,CacheAdm
        core_path = on
        core_pathname = /home/ios/logs
        maxage = 0
        maxexpired = -1
        histsize = 0
        histexpire = 0

I have a backup padmin account of jlynch so I had run as padmin:

chuser -attr maxage=0 padmin
chuser -attr maxexpired=-1 padmin
chuser -attr histsize=0 padmin
chuser -attr histexpire=0 padmin

chuser -attr maxage=0 jlynch
chuser -attr maxexpired=-1 jlynch
chuser -attr histsize=0 jlynch
chuser -attr histexpire=0 jlynch

Much to my surprise, after I had put on the patches and before the reboot I could still login to jlynch but for padmin I got an error that my password had expired, and it forced a password change. So, I changed the password then changed it back and I could then login fine. The first time I found this out after the reboot, so from then on, I tested the login before rebooting. It was very strange that they expired the padmin account but not the jlynch one.

This article shows you how to do the update from VIO 4.1.0.10 to 4.1.0.21 manually. A lot of this can be done using NIM if you prefer that. I am a big fan of fully patching (efixes, etc.) so that my VIO servers are as secure as possible. Happy patching!

References

IBM Web Download Page: Download OpenSSH, openssl and the Python3 patch from here

ESS: Download the PowerVM ISO image from here

Fix Central: Download Java8 8.0.0.826 from here (or higher as needs be)

FLRTLITE and the FLRT Data Tables

FLRT

---