AIX 7.3 and HMC v10 Upgrades
In late 2021, IBM released the final versions of HMC v10 and AIX v7.3. Below is a summary of my experiences upgrading my HMC to v10 and a couple of LPARs to AIX v7.3
HMC v10 Upgrades on Your POWERHMC
The first step for me is to login to the BMC interface on the HMC and bring up the Java console so I can monitor the HMC throughout. This is done by clicking on configuration, then console redirection and finally launch console. It then prompts you to save a jnlp file—after you save it you should then execute it and it will bring up a java interface into the HMC. This will let you monitor the HMC during the reboots.
Step 1 of any HMC upgrade for me is to download the code from IBM to my NIM server. This accomplishes two things: 1) It removes any dependency on the IBM site being up during my upgrade and 2) The loading of the upgrade files is much faster when I do the upgrades.
Step 2 is always to run backups of profile data, then saveupgdata and bkconsdata.
SSH to the HMC
Backup of profile data should be run for every server on the HMC.
bkprofdata -m servername -f backup-servername-date saveupgdata to disk and then to my NIM server saveupgdata -r disk saveupgdata -r disksftp -h NIMIP -u jlynch -d /software/hmcsavev9 Take a remote backup NIM lpar bkconsdata -r sftp -h NIMIP -u jlynch -d /software/hmcbkup or you can backup to USB bkconsdata -r usb
The backups can take anywhere from 15 minutes to an hour but they can be done the day before. I also get an HMCScanner report.
The upgrades I was doing were from v9.2m951 to v10.0.1011 plus patch MF69263 (which addresses the log4j security issue). Additionally my BMC and PNOR had already been updated to v3.26/v3.11.
The upgrade requires three reboots. First check that the time on both your BMC and HMC are correct. If they’re not, then correct them and reboot the HMC. Assuming your fixes are on your NIM server the first step is the v10 R1 SP 1010 upgrade which has to be done using getupgfiles—it can’t be done from the GUI.
From the HMC SSH login:
getupgfiles -r sftp -h NIMIP -u jlynch -d /software/hmcv10/network
Or you can get them from IBM:
getupgfiles -h public.dhe.ibm.com -u anonymous --passwd anonymous -d /software/server/hmc/network/v1010/ppc
Once this completes you tell the HMC to upgrade from the alternate disk and then reboot it:
chhmc -c altdiskboot -s enable --mode upgrade hmcshutdown -r -t now
The upgrade will take between 30-50 minutes. You can monitor it from the Java console you opened earlier.
Once the HMC is back lshmc -V should show the system is at v10r1 SP 1010.
Now you install the update to v10.1 SP 1011—you can do this from the HMC SSH login or using the GUI.
From SSH login:
updhmc -t sftp -h NIMIP -u jlynch -f /software/hmcv10/HMC_Update_V10R1M1011_ppc.iso
Or you can get it from the IBM site here.
Then reboot:
hmcshutdown -r -t now
lshmc -V should show the following after the reboot:
lshmc -V "version= Version: 10 Release: 1 Service Pack: 1011 HMC Build level 2111180904 MF69181 - HMC V10R1 M1011 ","base_version=V10R1 "
Finally, there is a patch ofix the log4j security problem—MF69263_ppc.iso. Again you can use the GUI or the line command on the HMC:
updhmc -t sftp -h NIMIP -u jlynch -f /software/hmcv10/MF69263_ppc.iso
Or from IBM here.
Then reboot:
hmcshutdown -r -t now
One thing to note is that throughout the updates the HMC opened three or more problems which then became cases at IBM—the error code was 9100 SRC E35F0010. We had to close those at the HMC and then close the cases that were opened at the IBM Support site. IBM is aware of the issue and has sent it on to development. If you don’t close the case at the HMC and at the IBM website you will get multiple calls and emails from IBM asking if you just did an update.
Finally, don’t forget to take another set of backups of your HMC and get another HMCScanner report.
AX v7.3 Upgrade
All of the systems I was looking to upgrade were at AIX 7.2.5.3 and were using DNF instead of YUM. One key thing to note is that AIX v7.3 uses the 64-bit version of DNF, not the 32-bit one previously used so you will need to reinstall DNFwhich I will cover below. Make sure you download the latest version of the DNF install script from here.
The upgrade to AIX 7.3 from 7.2.5 is very straightforward. I loaded the ISO image to my VIO and made it available to the LPAR using loadopt. I made sure DNF was up to date and that updtvpkg had been run. I also took a clone using alt_disk_copy to my spare disk.
An issue with bos.rte.streams may occur. If you run “lslpp -f bos.rte.streams” you will have the problem is you see:
Path: /etc/objrepos
bos.rte.streams 7.2.0.0
NONE
The solution for this is to remove the ODM definitions but you should send a snap to IBM first so they can be sure the lpp_id is correct. The fix IBM gave me is below but the lpp_id in your case may be quite different:
To fix the streams issue first backup the ODM:
# /usr/lib/instl/saveodms odmbak
Remove the bos.rte.streams 7.2 fileset from /etc/objrepos ODM.
# export ODMDIR=/etc/objrepos # odmget -q "lpp_name=bos.rte.streams and ver=7 and rel=2" product # odmdelete -q "lpp_name=bos.rte.streams and ver=7 and rel=2" -o product 0518-307 odmdelete: 3 objects deleted. # odmget -q "name=bos.rte.streams and ver=7 and rel=2" lpp # odmdelete -q "name=bos.rte.streams and ver=7 and rel=2" -o lpp 0518-307 odmdelete: 1 objects deleted.
This next step is where you need to have IBM find the correct lpp_id. So far it has been 423 on all of my systems:
# odmget -q "lpp_id=423" history # odmdelete -q "lpp_id=423" -o history (I saw the lpp_id of bos.rte.streams 7.2 is 423 from the ODM in snap data).. 0518-307 odmdelete: 1 objects deleted.
Check the filesets with “lppchk -v.” It should come back clean.
I started the update (after creating the clone) by running the following:
updtvpkg
dnf check
dnf update
This updated libxml2, python3 and wget
dnf check
I already had YUM on as well but this caused problems in some of the updates so I removed it as follows (DNF still remains):
rpm -qa | grep yum
yum-4.2.17-32_1.noarch
yum-metadata-parser-1.1.4-2.ppc
dnf remove yum yum-metadata-parser
Removed 2 packages
rpm -qa | grep yum
dnf check
gcc8 is replaced with gcc-10 on AIX 7.3 so I removed the old gcc
dnf remove gcc
Removed:
gcc-1:8-1.ppc gcc-cpp-1:8-1.ppc gcc8-8.3.0-6.ppc gcc8-cpp-8.3.0-6.ppc
gmp-6.1.2-1.ppc libmpc-1.1.0-1.ppc mpfr-4.0.2-2.ppc sed-4.8-1.ppc
rpm -qa | grep gcc
libgcc-8-1.ppc
libgcc8-8.3.0-6.ppc
You can’t remove the above two yet as DNF depends on them.
Now I run the pre_migration script which is in the ISO image. You can copy it across or just loopmount the ISO image which is what I do.
loopmount -i /software/aix73/AIX_v7.3_Install_7300-00-01-2148_flash_122021_LCD8265300.iso -m /cdrom -o "-V cdrfs -o ro" cd /cdrom/usr/lpp/bos ./pre_migration
All saved information can be found in: /home/pre_migration.220107133646
umount /cdrom
lspv | grep root hdisk3 00c47b307f5f4996 rootvg active hdisk0 00c47b30cba20b09 altinst_rootvg
Now shut down the LPAR.
From the HMC you will now activate the LPAR in SMS mode and select boot from CD—when prompted choose migration install.
Choose 1 to choose this as the console:
1 for english
2 to check settings
Make sure it is migration:
1 – system settings
3 -migration install
then 77 so can see disk PVIDs
In my case it showed disks as:
>>> 1 7.2 hdisk0 00c47b307f5f4996
2 7.2 hdisk1 00c47b30cba20b09
I matched this to the lspv | grep root above and selected 1 (which is rootvg)
Hit return twice so the upgrade starts
When prompted tell it to continue (option 0 – continue migrating)
There were 639 fixes to go on and the upgrade took about 50 minutes
When prompted for display enter vt100
Accept the license agreements
Go back and select Tasks completed – Exit to login
On the VIO unload the ISO you had loaded using unloadopt
Login to the LPAR and run the post_migration script
/usr/lpp/bos/post_migration
Checking for software that was unchanged during the migration.
Comparing saved configuration files.
Running lppchk commands. This may take awhile.
All saved information can be found in:/home/post_migration.220104131322
Run your checks:
oslevel -s
7300-00-01-2148
oslevel -s -l 7300-00-01-2148
lppchk -v
lppchk -vm3
instfix -i | grep ML
All filesets for 7.3.0.0_AIX_ML were found.
All filesets for 7300-00_AIX_ML were found.
instfix -icqk 7300-00_AIX_ML | grep :-:
If everything looks good you can now update Java, SSH and SSL to the latest patch levels.
Then run “updtvpkg”—you should run this whenever you update RPM or SSL. I run it multiple times during updates.
Inetd.conf may have been completely replaced so you should fix it now. My copy of /etc/inetd.conf normally looks as follows (except on NIM which has tftp and bootp as well). If you run PowerHA yours may also have caa uncommented.
/etc/inetd.conf #ftp stream tcp6 nowait root /usr/sbin/ftpd ftpd #telnet stream tcp6 nowait root /usr/sbin/telnetd telnetd -a #caa_cfg stream tcp6 nowait root /usr/sbin/clusterconf clustercon f >>/var/adm/ras/clusterconf.log 2>&1 dtspcd stream tcp nowait root /usr/dt/bin/dtspcd /usr/dt/bin/dtspcd #xmquery dgram udp6 wait root /usr/bin/xmtopas xmtopas -p3
At this point you need to reinstall DNF.
For some reason AIX 7.3 does not put the 7.3 toolbox into the dnf.conf file during the migration. It does put it there during a fresh install of AIX though.
cd /opt/freeware/etc/dnf cp dnf.conf dnf.conf-jl01042022 vi dnf.conf and add following lines to the end: [AIX_Toolbox_73] name=AIX 7.3 specific repository baseurl=https://anonymous:anonymous@public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc-7.3/ enabled=1 gpgcheck=0
Change into the directory where you put the dnf_aixtoolbox.sh script:
./dnf_aixtoolbox.sh -d
It installed 36 packages
Please note, RPM packages are downloaded in DNF cache /var/cache/dnf.
Now run:
dnf update It installed 8 packages and updated 6 packages
dnf check libgcc8-8.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0 libgomp8-8.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0 libstdc++8-8.3.0-6.ppc has missing requires of AIX-rpm < 7.3.0.0 Error: Check discovered 3 problem(s
You can now remove those old gcc8 libraries:
dnf remove libgcc8-8.3.0-6.ppc
Removes the above 3 error filesets
DNF check is now clean.
Run bosboot, bootlist and reboot and your upgrade should be complete unless you have other software that you may need to customize or update.
General Update Advice
The HMC v10 and AIX v7.3 updates are very straightforward although there are a couple of gotchas to watch out for. For the HMC it is the errors that it opens with IBM. This has been happening with HMC updates going back to HMC8. For AIX 7.3 the two I have found have been with bos.streams and DNF. After I opened a case regarding the DNF issues, IBM issued a technote 6538884 that explains it. Happy new year and have fun upgrading!
References
- Technote on DNF
- Fix Central
- AIX Linux Toolbox
- IBM Web Download Page
- A quick guide to DNF for YUM users
- HMC v1010 Update
- HMC Patch MF69263
- DNF Install Script