Skip to main content

AIX 7.3 and HMC v10 Upgrades

In late 2021, IBM released the final versions of HMC v10 and AIX v7.3. Below is a summary of my experiences upgrading my HMC to v10 and a couple of LPARs to AIX v7.3

HMC v10 Upgrades on Your POWERHMC

The first step for me is to login to the BMC interface on the HMC and bring up the Java console so I can monitor the HMC throughout. This is done by clicking on configuration, then console redirection and finally launch console. It then prompts you to save a jnlp file—after you save it you should then execute it and it will bring up a java interface into the HMC. This will let you monitor the HMC during the reboots.

Step 1 of any HMC upgrade for me is to download the code from IBM to my NIM server. This accomplishes two things: 1) It removes any dependency on the IBM site being up during my upgrade and 2) The loading of the upgrade files is much faster when I do the upgrades.

Step 2 is always to run backups of profile data, then saveupgdata and bkconsdata.

SSH to the HMC

Backup of profile data should be run for every server on the HMC.

bkprofdata -m servername -f backup-servername-date
saveupgdata to disk and then to my NIM server
saveupgdata -r disk
saveupgdata -r disksftp -h NIMIP -u jlynch -d /software/hmcsavev9
Take a remote backup NIM lpar
bkconsdata -r sftp -h NIMIP -u jlynch -d /software/hmcbkup
or you can backup to USB
bkconsdata -r usb

The backups can take anywhere from 15 minutes to an hour but they can be done the day before. I also get an HMCScanner report.

The upgrades I was doing were from v9.2m951 to v10.0.1011 plus patch MF69263 (which addresses the log4j security issue). Additionally my BMC and PNOR had already been updated to v3.26/v3.11.

The upgrade requires three reboots. First check that the time on both your BMC and HMC are correct. If they’re not, then correct them and reboot the HMC. Assuming your fixes are on your NIM server the first step is the v10 R1 SP 1010 upgrade which has to be done using getupgfiles—it can’t be done from the GUI.
From the HMC SSH login:

getupgfiles -r sftp -h NIMIP -u jlynch -d /software/hmcv10/network

Or you can get them from IBM:

getupgfiles -h -u anonymous --passwd anonymous -d /software/server/hmc/network/v1010/ppc

Once this completes you tell the HMC to upgrade from the alternate disk and then reboot it:

chhmc -c altdiskboot -s enable --mode upgrade
hmcshutdown -r -t now

The upgrade will take between 30-50 minutes. You can monitor it from the Java console you opened earlier.

Once the HMC is back lshmc -V should show the system is at v10r1 SP 1010.
Now you install the update to v10.1 SP 1011—you can do this from the HMC SSH login or using the GUI.

From SSH login:

updhmc -t sftp -h NIMIP  -u jlynch -f /software/hmcv10/HMC_Update_V10R1M1011_ppc.iso

Or you can get it from the IBM site here.

Then reboot:

hmcshutdown -r -t now

lshmc -V should show the following after the reboot:

lshmc -V
"version= Version: 10
 Release: 1
 Service Pack: 1011
HMC Build level 2111180904
MF69181 - HMC V10R1 M1011

Finally, there is a patch ofix the log4j security problem—MF69263_ppc.iso. Again you can use the GUI or the line command on the HMC:

updhmc -t sftp -h NIMIP  -u jlynch -f  /software/hmcv10/MF69263_ppc.iso

Or from IBM here.

Then reboot:

hmcshutdown -r -t now

One thing to note is that throughout the updates the HMC opened three or more problems which then became cases at IBM—the error code was 9100 SRC E35F0010. We had to close those at the HMC and then close the cases that were opened at the IBM Support site. IBM is aware of the issue and has sent it on to development. If you don’t close the case at the HMC and at the IBM website you will get multiple calls and emails from IBM asking if you just did an update.

Finally, don’t forget to take another set of backups of your HMC and get another HMCScanner report.

AX v7.3 Upgrade

All of the systems I was looking to upgrade were at AIX and were using DNF instead of YUM. One key thing to note is that AIX v7.3 uses the 64-bit version of DNF, not the 32-bit one previously used so you will need to reinstall DNFwhich I will cover below. Make sure you download the latest version of the DNF install script from here.

The upgrade to AIX 7.3 from 7.2.5 is very straightforward. I loaded the ISO image to my VIO and made it available to the LPAR using loadopt. I made sure DNF was up to date and that updtvpkg had been run. I also took a clone using alt_disk_copy to my spare disk.

An issue with bos.rte.streams may occur. If you run “lslpp -f bos.rte.streams” you will have the problem is you see:

Path: /etc/objrepos

The solution for this is to remove the ODM definitions but you should send a snap to IBM first so they can be sure the lpp_id is correct. The fix IBM gave me is below but the lpp_id in your case may be quite different:

To fix the streams issue first backup the ODM:

 # /usr/lib/instl/saveodms odmbak

Remove the bos.rte.streams 7.2 fileset from /etc/objrepos ODM.

 # export ODMDIR=/etc/objrepos
 # odmget -q "lpp_name=bos.rte.streams and ver=7 and rel=2" product
 # odmdelete -q "lpp_name=bos.rte.streams and ver=7 and rel=2" -o product
0518-307 odmdelete: 3 objects deleted.
 # odmget -q "name=bos.rte.streams and ver=7 and rel=2" lpp
 # odmdelete -q "name=bos.rte.streams and ver=7 and rel=2" -o lpp
 0518-307 odmdelete: 1 objects deleted.

This next step is where you need to have IBM find the correct lpp_id. So far it has been 423 on all of my systems:

 # odmget -q "lpp_id=423" history
 # odmdelete -q "lpp_id=423" -o history
 (I saw the lpp_id of bos.rte.streams 7.2 is 423 from the ODM in snap data)..
 0518-307 odmdelete: 1 objects deleted.

Check the filesets with “lppchk -v.” It should come back clean.

I started the update (after creating the clone) by running the following:

dnf check
dnf update
            This updated libxml2, python3 and wget
dnf check

I already had YUM on as well but this caused problems in some of the updates so I removed it as follows (DNF still remains):

rpm -qa | grep yum
dnf remove yum yum-metadata-parser
            Removed 2 packages
rpm -qa | grep yum
dnf check
gcc8 is replaced with gcc-10 on AIX 7.3 so I removed the old gcc
dnf remove gcc
  gcc-1:8-1.ppc    gcc-cpp-1:8-1.ppc   gcc8-8.3.0-6.ppc  gcc8-cpp-8.3.0-6.ppc
  gmp-6.1.2-1.ppc  libmpc-1.1.0-1.ppc  mpfr-4.0.2-2.ppc  sed-4.8-1.ppc
rpm -qa | grep gcc

You can’t remove the above two yet as DNF depends on them.

Now I run the pre_migration script which is in the ISO image. You can copy it across or just loopmount the ISO image which is what I do.

loopmount -i  /software/aix73/AIX_v7.3_Install_7300-00-01-2148_flash_122021_LCD8265300.iso  -m /cdrom  -o "-V cdrfs -o ro"
cd /cdrom/usr/lpp/bos

All saved information can be found in: /home/pre_migration.220107133646
umount /cdrom

lspv | grep root
hdisk3          00c47b307f5f4996                    rootvg          active
hdisk0          00c47b30cba20b09                    altinst_rootvg

Now shut down the LPAR.

From the HMC you will now activate the LPAR in SMS mode and select boot from CD—when prompted choose migration install.

Choose 1 to choose this as the console:
1 for english
2 to check settings

Make sure it is migration:
1 – system settings
3 -migration install
then 77 so can see disk PVIDs

In my case it showed disks as:
>>>  1 7.2    hdisk0    00c47b307f5f4996
2 7.2    hdisk1    00c47b30cba20b09

I matched this to the lspv | grep root above and selected 1 (which is rootvg)
Hit return twice so the upgrade starts
When prompted tell it to continue (option 0 – continue migrating)

There were 639 fixes to go on and the upgrade took about 50 minutes
When prompted for display enter vt100
Accept the license agreements
Go back and select Tasks completed – Exit to login

On the VIO unload the ISO you had loaded using unloadopt

Login to the LPAR and run the post_migration script

Checking for software that was unchanged during the migration.
Comparing saved configuration files.
Running lppchk commands. This may take awhile.
All saved information can be found in:/home/post_migration.220104131322

Run your checks:

oslevel -s
oslevel -s -l 7300-00-01-2148
lppchk -v
lppchk -vm3
instfix -i | grep ML
    All filesets for were found.
    All filesets for 7300-00_AIX_ML were found.
instfix -icqk  7300-00_AIX_ML | grep :-:

If everything looks good you can now update Java, SSH and SSL to the latest patch levels.

Then run “updtvpkg”—you should run this whenever you update RPM or SSL. I run it multiple times during updates.

Inetd.conf may have been completely replaced so you should fix it now. My copy of /etc/inetd.conf normally looks as follows (except on NIM which has tftp and bootp as well). If you run PowerHA yours may also have caa uncommented.

#ftp    stream  tcp6    nowait  root    /usr/sbin/ftpd  ftpd
#telnet stream  tcp6    nowait  root    /usr/sbin/telnetd       telnetd -a
#caa_cfg        stream  tcp6    nowait  root    /usr/sbin/clusterconf clustercon
f >>/var/adm/ras/clusterconf.log 2>&1
dtspcd  stream  tcp     nowait  root    /usr/dt/bin/dtspcd /usr/dt/bin/dtspcd
#xmquery        dgram   udp6    wait    root    /usr/bin/xmtopas xmtopas -p3

At this point you need to reinstall DNF.

For some reason AIX 7.3 does not put the 7.3 toolbox into the dnf.conf file during the migration. It does put it there during a fresh install of AIX though.

cd /opt/freeware/etc/dnf
cp dnf.conf dnf.conf-jl01042022
vi dnf.conf and add following lines to the end:
name=AIX 7.3 specific repository

Change into the directory where you put the script:

./ -d
            It installed 36 packages

Please note, RPM packages are downloaded in DNF cache /var/cache/dnf.

Now run:

dnf update
It installed 8 packages and updated 6 packages
dnf check
libgcc8-8.3.0-6.ppc has missing requires of AIX-rpm <
libgomp8-8.3.0-6.ppc has missing requires of AIX-rpm <
libstdc++8-8.3.0-6.ppc has missing requires of AIX-rpm <
Error: Check discovered 3 problem(s

You can now remove those old gcc8 libraries:

dnf remove libgcc8-8.3.0-6.ppc
            Removes the above 3 error filesets

DNF check is now clean.

Run bosboot, bootlist and reboot and your upgrade should be complete unless you have other software that you may need to customize or update.

General Update Advice 

The HMC v10 and AIX v7.3 updates are very straightforward although there are a couple of gotchas to watch out for. For the HMC it is the errors that it opens with IBM. This has been happening with HMC updates going back to HMC8. For AIX 7.3 the two I have found have been with bos.streams and DNF. After I opened a case regarding the DNF issues, IBM issued a technote 6538884 that explains it. Happy new year and have fun upgrading!