Adventures with NIMADM: Upgrading from AIX V7.2 to V7.3

IBM Champion Jaqui Lynch highlights her recent experience using NIMADM to upgrade AIX from V7.2 to V7.3

Jaqui Lynch August 28, 2024

Recently I had to upgrade several LPARs from AIX V7.2 to V7.3. Normally I do this by booting from an ISO image, a CD or NIM and doing a preservation or migration upgrade. However, these systems were on a NetApp. The NetApp MPIO drivers are not included in the ISO, which means that you get a copy of every disk for every path and you have to figure out which one is the real one and a number of other issues.

I decided to draw on my NIM experience and do what I should have been doing all along–using nimadm. This article covers what I did and the issues that came up. I have always been a fan of NIM and will now be using nimadm a lot more.

The first step is ensuring the NIM server is at the latest 7.3.2.2 with all the patches applied. Then I had to build the necessary NIM resources.

Building the NIM Resources

I keep the initial images in /software/aix73 on my NIM server, so I upload the iso image to there.

cd /software/aix73

mkdir aix73tl2sp2-2420-base

I then use loopmount to mount the image so I can copy the code for NIM to use.

loopmount -i /software/aix73/AIX_v7.3_Install_7300-02-02-2420_flash_062024_LCD8265305.iso  -o "-V cdrfs -o ro" -m /cdrom

I then create the base directory to use later.

smitty bffcreate from /cdrom into /software/aix73/aix73tl2sp2-2420-base

umount /cdrom

Then I build the resources:

Create a mksysb nim resource called lpp_73tl2sp2-2420 from /software/aix73/aix73tl2sp2-2420-base.

nim -o define -t lpp_source -a server=master -a location=/nim/lpp_source/lpp_73tl2sp2-2420 -a source=/software/aix73/aix73tl2sp2-2420-base   lpp_73tl2sp2-2420

Add java, ssh and ssl updated levels to the LPP

nim -o update -a packages=all -a source=/software/flrtfixes/javasshssl-jul112024  lpp_73tl2sp2-2420

Install the NetApp HAK (MPIO drivers) into the LPP

nim -o update -a packages=all -a source=/software/netapphak61 lpp_73tl2sp2-2420

Put latest perl fix into the LPP

nim -o update -a packages=all -a source=/software/flrtfixes/perl-534   lpp_73tl2sp2-2420

Put latest invscout fix into the LPP

nim -o update -a packages=all -a source=/software/flrtfixes/invscout_fix6   lpp_73tl2sp2-2420

Put latest python fix into the LPP

nim -o update -a packages=all -a source=/software/flrtfixes/python-3.9.19.2 lpp_73tl2sp2-2420

Put RPM patch into the LPP

nim -o update -a packages=all -a source=/software/flrtfixes/rpm-73/732andvio41 lpp_73tl2sp2-2420

Create two new directories for opensll and curl fixes – just copy in the 7.3 efixes

Copy into openssl_fix42a - 853sa.240503.epkg.Z

Copy into curl_fix5a - 3013sa.240722.epkg.Z

nim -o update -a packages=all -a source=/software/flrtfixes/openssl_fix42a lpp_73tl2sp2-2420

nim -o update -a packages=all -a source=/software/flrtfixes/curl_fix5a lpp_73tl2sp2-2420

Put latest SSH fix into the LPP

nim -o update -a packages=all -a source=/software/flrtfixes/openssh_fix17 lpp_73tl2sp2-2420

Now check the LPP resources:

nim -o check lpp_73tl2sp2-2420

lsnim -l lpp_73tl2sp2-2420

Now create the SPOT from the LPP:

nim -o define -t spot -a server=master -a location=/nim/spot -a source=lpp_73tl2sp2-2420 spot_73tl2sp2-2420

Then check the resources:

nim -o check spot_73tl2sp2-2420

lsnim -l spot_73tl2sp2-2420

I then realized I had forgotten to ensure that bos.alt.disk.install* was installed so I went back and did the following:

I copied the bos.alt.disk images from my base directory into 73altdisk and then added them to the LPP:

nim -o update -a packages=all -a source=/software/aix73/73altdisk lpp_73tl2sp2-2420

Then I customized the spot

nim -o cust -a filesets=bos.alt_disk_install.boot_images -a lpp_source=lpp_73tl2sp2-2420 spot_73tl2sp2-2420

nim -o cust -a filesets=bos.alt_disk_install -a lpp_source=lpp_73tl2sp2-2420 spot_73tl2sp2-2420

nim -o showres spot_73tl2sp2-2420 | grep -i bos.alt

nim -o showres lpp_73tl2sp2-2420 | grep -i bos.alt

You should now see something like:

#nim -o showres spot_73tl2sp2-2420 | grep -i bos.alt

  bos.alt_disk_install.boot_images

  bos.alt_disk_install.rte   7.3.2.1    C     F    Alternate Disk Installation

#nim -o showres lpp_73tl2sp2-2420 | grep -i bos.alt

  bos.alt_disk_install.boot_images 7.3.2.3                    I  N usr

  bos.alt_disk_install.rte    7.3.2.1                    I  N usr,root

Now you need to make sure the client is configured correctly for NIM.

Check the LPAR is defined:

So to test lpar lpar001

lsnim -l | grep lpar001

If it is not defined, then go ahead and define the machine.

Use ping first to make sure you have name resolution and can reach the LPAR

Once you have the machine defined and can ping it, then check the /etc/niminfo file is correct. I had migrated our NIM LPAR to a new name and server previously and a couple of the clients had the wrong /etc/niminfo file. After fixing this, I also checked that nimclient was running on the LPAR:

ps -ef | grep nim

If you don’t see nimclient then start ir:

startsrc -g nimclient

Finally, the next step to is to ensure the NIM server can talk to the client.

nim -o lslpp lpar01 | grep bos.mp

If this fails check /var/adm/ras/nimsh.log on the client LPAR (lpar001)

cat /var/adm/ras/nimsh.log

If the last line is similar to:

error: local value passed, '', does not match environment value '00CE48D04B00'

This means the CPUID NIM has for the LPAR is not correct. I ran into this as we had moved several LPARs to new servers and forgot to update their CPUID on NIM.

To fix this:

On the client LPAR run “uname -a” to get the cpuid.

In this case it showed:

00CE48D04B00

Now on the NIM LPAR reset the cpuid for that client:

nim -o change -a cpuid=00CE48D04B00 lpar001

lsnim -l lpar001 | grep -i cpuid

Then test it again:

nim -o lslpp lpar001 | grep bos.mp

This should work now

Now that the NIM resources are created you can go ahead with the update.

Final Preparations for the Version Upgrade

Make sure you have the latest flrtvc (0810) and HMCScanner (0.11.54) downloaded as you will need these later:

flrtvc

HMCScanner

Always start by running errpt to check for errors. You do not want to try to update a failing system or one that has errors. Additionally, if you are mirroring rootvg, you will need to unmirror it or have two spare disks for the upgrade. nimadm takes a clone of rootvg and does the update to the clone. This makes for a fast failback as you basically set the bootlist back to the original rootvg hdisk and reboot to recover.

#lspv | grep root

In this case I see:

hdisk1          00c47b30e52f4912                    altinst_rootvg

hdisk0          00c47b30e1c4c959                    rootvg          active

Note which disk is altinst_rootvg and write it down:

exportvg altinst_rootvg

Take a mksysb type backup.

Get an HMCScanner report for the HMC the LPAR is running on.

From my desktop I change into c:\hmcscanner54

Assuming HMC is called hmc001 and my id is jlynch

hmcscanner.bat hmc001 jlynch

It will prompt for your password and will then create the files.

Performing the Version Upgrade

On the client LPAR – lpar001

oslevel -s

7200-05-07-2346

Make sure there are no issues with missing filesets.

instfix -i | grep ML

oslevel -s -l 7200-05-07-2346

lppchk -v

lppchk -vm3

instfix -icqk  7200-01_AIX_ML | grep :-:

instfix -icqk  7200-02_AIX_ML | grep :-:

instfix -icqk  7200-03_AIX_ML | grep :-:

instfix -icqk  7200-04_AIX_ML | grep :-:

instfix -icqk  7200-05_AIX_ML | grep :-:

lslpp -l | grep ssh

lslpp -l | grep ssl

lslpp -l | grep Java

df -g

Check for missing paths:

lspath | grep iss

lspath | grep ail

lspath | grep efin

errpt

If there are any errors, fix them before continuing.

#lspv | grep root

hdisk0          00c47b30e1c4c959                    rootvg          active

Remember hdisk1 was altinst_rootvg and this is where we will install to.

Before rebooting, run bosboot to rewrite the boot image and use bootlist to rewrite the bootlist – assuming hdisk0 is rootvg then:

#bosboot -a -d hdisk0

#bootlist -m normal -o

#bootlist -m normal hdisk0

#bootlist -m normal -o

If the bosboot fails, stop here and fix that.

I also check the disk setups now:

#: lsmpio -ql hdisk0

Device:  hdisk0

          Vendor Id:  NETAPP

         Product Id:  LUN C-Mode

           Revision:  9800

           Capacity:  150.00GiB

      Volume Serial:  5D4B5A337352680000A0983830427244  (Page 83 EUI)

#: lsmpio -ql hdisk1

Device:  hdisk1

          Vendor Id:  NETAPP

         Product Id:  LUN C-Mode

           Revision:  9800

           Capacity:  150.00GiB

      Volume Serial:  5D4B5A337352690000A0983830427244  (Page 83 EUI)

lscfg -vpl hdisk0

lscfg -vpl hdisk1

Hopefully, I will not need this information, but it is useful to have just in case.

Check how many disks you have and compare later after reboot. Do the same with mounted filesystems;

lspv | wc -l

df -g | wc -l

Now on the NIM LPAR:

nimadm -c lpar001 -l lpp_73tl2sp2-2420 -s spot_73tl2sp2-2420 -d hdisk1 -Y -V

This should take about 1.5-1.75 hours,

When it is done, hdisk1 will contain the upgraded version and the bootlist will have been set to hdisk1.

You can check this as follows:

bootinfo -b

bootlist -m normal -o

#bootinfo -b

hdisk0

#bootlist -m normal -o

hdisk1 blv=hd5 pathid=0

hdisk1 blv=hd5 pathid=1

hdisk1 blv=hd5 pathid=2

hdisk1 blv=hd5 pathid=3

hdisk1 blv=hd5 pathid=4

If you don’t plan to reboot the new version immediately, then you should set the bootlist back to hdisk0 to avoid accidents:

#bootlist -m normal hdisk0

If you are going to boot to the new version now, then don’t reset the bootlist. Shutdown any applications, databases, etc. then reboot:

shutdown -r now

After the reboot, check your have all the disks and mounts.

mount /software

lspv | wc -l

df -g | wc -l

Make sure there are no errors after reboot.

errpt

I needed to uninstall some apps as oslevel -s did not show the correct level.

On client LPAR:

installp -u devices.tmiscsw.rte

installp -u perl.man.en_US

installp -u NetApp.FCP_Host_Utilities_Kit.LUN.msg.en_US

installp -u   NetApp.FCP_Host_Utilities_Kit.ODM_stanzas

installp -u   NetApp.MPIO_Host_Utilities_Kit.iscsi

Some LPARs were also missing a few filesets that I had to install from my base (/software/aix73/aix73tl2sp2-2420-base). These were:

I:X11.Dt.ToolTalk              7.3.2.1

    I:X11.Dt.rte                   7.3.2.0

    I:printers.hpJetDirect.attach  7.3.2.0

Run the following checks:

oslevel -s

oslevel -s -l 7300-02-02-2420

lppchk -v

lppchk -vm3

instfix -i | grep ML

instfix -icqk  7300-00_AIX_ML | grep :-:

instfix -icqk  7300-01_AIX_ML | grep :-:

instfix -icqk  7300-02_AIX_ML | grep :-:

oslevel -s and instfix should show:

lpar001# oslevel -s

7300-02-02-2420

lpar001# instfix -i | grep ML

    All filesets for 7.3.0.0_AIX_ML were found.

    All filesets for 7300-00_AIX_ML were found.

    All filesets for 7300-01_AIX_ML were found.

    All filesets for 7300-02_AIX_ML were found.

The others should return nothing. If they identify missing or problem filesets, then either remove or correct them.

Check /etc/inetd.conf – the upgrade tends to replace it with the default.

cat /etc/inetd.conf

I normally replace it as follows:

cp /etc/inetd.conf /etc/inetd.conf-jl08022024

Then I edit /etc/inetd.conf so it only contains:

#ftp    stream  tcp6    nowait  root    /usr/sbin/ftpd  /usr/sbin/ftpd ftpd

#telnet stream  tcp6    nowait  root    /usr/sbin/telnetd       telnetd -a

dtspcd  stream  tcp     nowait  root    /usr/dt/bin/dtspcd /usr/dt/bin/dtspcd

xmquery dgram   udp     wait    root    /usr/bin/xmtopas xmtopas -p3

Your contents may vary.

Now refresh inetd:

refresh -s inetd

Run all your normal checks. You may need to replace /etc/motd as I found it overwrote it a couple of times.

My checks now show:

lslpp -L rpm.rte

  rpm.rte                 4.18.1.2003    C     F    RPM Package Manager

lslpp -L perl.rte

  perl.rte                  5.34.1.6    C     F    Perl Version 5 Runtime

lslpp -l | grep ssh

lslpp -l | grep ava

lslpp -l | grep ssl

SSH                9.2.112.2400

Java                8.0.0.826

SSL                3.0.13.1000

Even though I had loaded the efixes into the LPP, they did not end up on the system so I had to put them on manually as follows:

emgr -P

lpar001# emgr -P

There is no efix data on this system.

cd /software/flrtfixes/openssh_fix17

emgr -p -e 9211224a.240708.epkg.Z

emgr  -e 9211224a.240708.epkg.Z

cd /software/flrtfixes/invscout_fix6

emgr -p -e is22026s1a.240514.epkg.Z

emgr -e is22026s1a.240514.epkg.Z

cd /software/flrtfixes/curl_fix5

emgr -p -e 853sa.240503.epkg.Z

emgr -e 853sa.240503.epkg.Z

lslpp -l | grep -i python

  python3.9.base            3.9.19.0  COMMITTED  Python 3.9 64-bit binary

cd /software/flrtfixes/python-3.9.19.2

installp -apYd . python*

installp -aXYd . python*

lslpp -l | grep -i python

  python3.9.base            3.9.19.2  COMMITTED  Python 3.9 64-bit binary

cd /software/flrtfixes/openssl_fix42

emgr -p -e 3013sa.240722.epkg.Z

emgr -e 3013sa.240722.epkg.Z

lslpp -l | grep bind

lslpp -L bind.rte

lpar001# lslpp -l | grep bind

  bind.rte              7.3.916.2601  COMMITTED  BIND Domain Name System

  bos.rte.bind_cmds          7.3.2.1  COMMITTED  Binder and Loader Commands

  bind.rte              7.3.916.2601  COMMITTED  BIND Domain Name System

  bos.rte.bind_cmds          7.3.2.1  COMMITTED  Binder and Loader Commands

lpar001# lslpp -L bind.rte

  Fileset                      Level  State  Type  Description (Uninstaller)

  ----------------------------------------------------------------------------

  bind.rte              7.3.916.2601    C     F    BIND Domain Name System

If bind.rte is installed, then:

cd /software/flrtfixes/bind_fix26/73bind916

installp -apYd . bind.rte

installp -aXYd . bind.rte

lslpp -l | grep bind

lslpp -L bind.rte

lpar001# lslpp -l | grep bind

  bind.rte              7.3.916.4800  COMMITTED  BIND Domain Name System

  bos.rte.bind_cmds          7.3.2.1  COMMITTED  Binder and Loader Commands

lslpp -L bind.rte

  bind.rte              7.3.916.4800  COMMITTED  BIND Domain Name System

  bos.rte.bind_cmds          7.3.2.1  COMMITTED  Binder and Loader Commands

lpar001# lslpp -L bind.rte

  Fileset                      Level  State  Type  Description (Uninstaller)

  ----------------------------------------------------------------------------

  bind.rte              7.3.916.4800    C     F    BIND Domain Name System

Now check using flrtvc to see if you are missing any patches.

cd /software/flrtvc

./flrtvc-0810.ksh

lpar001# ./flrtvc-0810.ksh

Fileset|Current Version|Type|EFix Installed|Abstract|Unsafe Versions|APARs|Bulletin URL|Download URL|CVSS Base Score|Reboot Required|Last Update|Fixed In

All vulnerabilities fixed.

emgr -P should now show:

PACKAGE                                                  INSTALLER   LABEL

======================================================== =========== ==========

openssh.base.client                                      installp    9211224a

openssh.base.server                                      installp    9211224a

invscout.rte                                             installp    is22026s1a

oss.lib.libcurl                                          installp    853sa

openssl.base                                             installp    3013sa

emgr -l should show:

ID  STATE LABEL      INSTALL TIME      UPDATED BY ABSTRACT

=== ===== ========== ================= ========== ======================================

1    S    9211224a   08/05/24 19:37:05            ifix for CVE-2024-6387

2    S    is22026s1a 08/05/24 19:37:51            invscout fix for CVE-2024-27260

3    S    853sa      08/05/24 19:38:07            ifix for libcurl CVE-2024-0853

4    S    3013sa     08/05/24 19:38:41            ifix for openssl july CVEs

There may be other patches required, but these were all the levels and patches as of Aug. 5, 2024.

I use dnf on my systems, so the next step was to update my dnf applications.

dnf check

I got a weird error, which is a known issue with the 7.3 upgrade. The fix is to reinstall:

cd /software/dnf

 ./dnf_aixtoolbox.sh -d

dnf installed successfully.

dnf update

Upgrade  36 Packages

dnf clean all

dnf check

gcc10-10.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0

gcc10-cpp-10.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0

libgcc-1:10-2.ppc has missing requires of AIX-rpm < 7.3.0.0

libgcc10-10.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0

libgo10-10.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0

libgo10-devel-10.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0

libgomp-1:10-2.ppc has missing requires of AIX-rpm < 7.3.0.0

libgomp10-10.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0

libstdc++-1:10-2.ppc has missing requires of AIX-rpm < 7.3.0.0

libstdc++-devel-1:10-2.ppc has missing requires of AIX-rpm < 7.3.0.0

libstdc++10-10.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0

libstdc++10-devel-10.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0

Error: Check discovered 12 problem(s)

There are known issues with GCC when upgrading to AIX 7.3, so it was necessary to reinstall it:

dnf reinstall gcc10-10.3.0-6 gcc10-cpp-10.3.0-6  libgcc-1:10-2 libgcc10-10.3.0-6 libgomp-1:10-2 libgomp10-10.3.0-6 libstdc++-1:10-2 libstdc++10-10.3.0-6 libgo10-10.3.0-6 libstdc++-devel-1:10-2 libstdc++10-devel-10.3.0-6

There may be additional files listed on the dnf check above, so make sure the reinstall list matches the error list in dnf check.

dnf check

updtvpkg

dnf update

You should see:

Last metadata expiration check: 0:05:44 ago on Fri Aug  2 13:38:33 2024.

Dependencies resolved.

Nothing to do.

Complete!

Check for any missing patches:

cd /software/flrtvc

./flrtvc-0810-nodl.ksh

You should see:

Fileset|Current Version|Type|EFix Installed|Abstract|Unsafe Versions|APARs|Bulletin URL|Download URL|CVSS Base Score|Reboot Required|Last Update|Fixed In

All vulnerabilities fixed.

lspv | grep root

lpar001: lspv | grep root

hdisk1          00c47b30e52f4912                    rootvg          active

hdisk0          00c47b30e1c4c959                    old_rootvg

bootinfo -b

bootlist -m normal -o

bosboot -a -d hdisk1

bootlist -m normal hdisk1

bootlist -m normal -o

Check errpt

Now for the final reboot:

date

shutdown -r now

Run all your final checks and bring up applications and databases, etc. and test.

Take a mksysb, so you have a backup. Wait about two weeks and then reclone, so both disks are the same level.

exportvg old_rootvg

alt_disk_copy -V -B -d hdisk0

Reverting to Old Copy

If you have to revert due to any issues, then reset the bootlist and reboot:

bootinfo -b

Should show hdisk1

bootlist -m normal -o

bootlist -m normal hdisk0

bootlist -m normal -o

shutdown -r now

Updating From AIX V7.2 to AIX V7.3

This article shows you how to do the update from AIX V7.2 to AIX V7.3 using nimadm. You can use the same technique to perform the upgrade from V7.1 to V7.2 or V7.3.