Maintaining Your AIX Systems
Jaqui Lynch on the different ways to perform AIX maintenance, updates and patching
AIX Maintenance MethodsIf you are staying within a release level, then patching can easily be done using a number of methods such as NIM, AIX Live Update, alternate disk install, or in-place updates. The simplest is in-place updates which I will discuss below.
AIX v7.3 UpgradeIf you are changing release levels (AIX 7.2 to 7.3, as an example) then the update is a little more complex. You can take a mksysb and use NIM to update it to an alternate disk, or you can boot from the DVD or ISO image (ISO from the VIO) and do a migration upgrade.
For third-party products I now use DNF instead of YUM or RPM to install them. This ensures that prerequisites and corequisites are taken care of. When updating to AIX 220.127.116.11 to AIX 7.3 you will need to reinstall the new version of DNF as AIX v7.3 uses the 64-bit version of DNF, not the 32-bit one previously used. Make sure you download the latest version of the DNF install script from here:
Details on upgrading to AIX 7.3 and HMC v10 are covered in my January 2022 article, so I will include the link to them here:
Putting Maintenance on AIX at the Same Release LevelI always start with my NIM LPAR as it is the one system that can download patches from IBM. The first step is to run flrtvc on the NIM LPAR so I download the latest apar.csv file and it tells me what patches I need for the NIM LPAR at the level it is currently running. Below is the process I followed to do an in-place update of my NIM LPAR. The next step is to get an inventory of where the system is at before starting.
Using lslpp I find:
SSL is at 18.104.22.1680 SSH is at 22.214.171.1245 Java7 is 126.96.36.1995 Java8 is 188.8.131.525 oslevel -s 7300-00-01-2148I also check for efixes already installed:
emgr -P PACKAGE INSTALLER LABEL ======================================================== =========== ========== bos.cluster.rte installp IJ36596s1a bos.rte.security installp IJ36816s1a bos.mp64 installp IJ37001m1a bos.net.tcp.bind_utils installp IJ37226s1b bos.net.tcp.bind installp IJ37226s1b bos.perf.tune installp IJ38518s1b bos.pfcdd.rte installp IJ37403s1a bos.pmapi.pmsvcs installp IJ37411s1a bos.rte.security installp IJ38121m1a bos.sysmgt.nim.client installp IJ36593m1a
Continuing Maintenance on AIX After UpgradesAt this point, I want to upgrade AIX from 184.108.40.206 to 220.127.116.11 and I also need to patch Python3, SSL, SSH and Java. I will also want to update any RPMs using DNF.
Java and the AIX ML are downloaded from Fix Central.
The AIX level I am downloading is 7300-00-02-2220, which became available on 5/20/2022.
For Java I download the 32-bit and 64-bit Java runtimes
For Java8 this is 18.104.22.1681 – 6/28/2022
For Java7 this is 22.214.171.1245 – 8/11/2022
SSH and SSL are downloaded from the IBM Web Download page. You can also download Python3 for AIX 7.3 here—there is a security issue with the default Python3 so you should download this.
OpenSSL 126.96.36.1991 with no weak ciphers support
If you are not at these levels on AIX 7.3 then you need to download them and update them.
The IBM Web Download page is also where you can download lsof, Xgzip, SMB client, NTPv4, the LDAP client code, KRB5 client code and several other applications.
To download the full ISO image of AIX you will need to go to Entitled Software Support (ESS) at: https://www.ibm.com/servers/eserver/ess/index.wss
Preparing for UpdatesNote that IBM has updated many of their download pages now to require both signing in with your IBM account and using MFA with IBM Verify. Make sure you have this set up on your device prior to starting your downloads. The IBM web download page is one that requires IBM verification before you can even check levels.
Once all the software and patches are downloaded it is time to start the updates. I use NFS to share out patches from /software on the NIM LPAR to the client LPARs. I untar all the python3, java, ssh and ssl updates into one directory (in this case javasshssl-aug2022). I untar the AIX service pack into aixtl3sp02-2220-updates.
Before doing any maintenance it is important to run errpt to make sure there are no errors that could affect the upgrade. Any disk or other errors should be fixed prior to continuing. You should also check the readmes for the updates.
I usually run a mksysb to tape and then I take a clone so my recovery can be a fast reboot.
lspv | grep root hdisk1 00f95d3a425513d5 rootvg active hdisk2 00f95d3a421072a7 altinst_rootvgIn the section above, you can see hdisk1 is our boot image and hdisk2 is the alternate. If you have them mirrored (both showing rootvg) then you will need to break the mirror to free up one of the disks. Even though they are not mirrored I still do the following:
uptime bootinfo -b bootlist -m normal -o bosboot -a -d hdisk1 bootlist -m normal hdisk1 bootlist -m normal -oThe first two commands show me how long the system has been up and which disk it booted from. In a mirrored rootvg this is more important.
# uptime 11:54AM up 101 days, 18:54, 1 user, load average: 1.54, 1.20, 1.51 # bootinfo -b hdisk1Before I do maintenance I check and rewrite the bootlist, especially if it has not been rebooted for a while. In this case, it was 101 days, but I probably would rewrite it anyway as I have been caught out by damaged boot images and bootlists before.
Now I take my clone:
exportvg altinst_rootvg alt_disk_copy -V -B -d hdisk2In the section above, I free up hdisk2 so I can use it for the clone. I then tell it to take a copy to hdisk2 with all the details (V=verbose). The -B tells it not to change the bootlist.
When that is complete we can start the update. The first step is to remove any currently installed efixes/ifixes. emgr -P provides that list:
emgr -P PACKAGE INSTALLER LABEL ======================================================== =========== ========== bos.cluster.rte installp IJ36596s1a bos.rte.security installp IJ36816s1a bos.mp64 installp IJ37001m1a bos.net.tcp.bind_utils installp IJ37226s1b bos.net.tcp.bind installp IJ37226s1b bos.perf.tune installp IJ38518s1b bos.pfcdd.rte installp IJ37403s1a bos.pmapi.pmsvcs installp IJ37411s1a bos.rte.security installp IJ38121m1a bos.sysmgt.nim.client installp IJ36593m1aThey are removed as follows:
emgr -r -L IJ36596s1a emgr -r -L IJ37001m1a emgr -r -L IJ36816s1a emgr -r -L IJ37226s1b emgr -r -L IJ38518s1b emgr -r -L IJ37403s1a emgr -r -L IJ37411s1a emgr -r -L IJ38121m1a emgr -r -L IJ36593m1a emgr -PIt should now say, “There is no efix data on this system.”
At this point, I do some basic checks:
oslevel -s oslevel -s -l 7300-00-01-2148 lppchk -v lppchk -vm3 instfix -i | grep ML instfix -icqk 7300-00_AIX_ML | grep :-: dnf check updtvpkgThese checks are to ensure there are no missing filesets or known problems before starting. Assuming all is well we then start the updates.
I save a couple of files that maintenance tends to replace:
cp /etc/inetd.conf /etc/inetd.conf-premaint cp /etc/inittab /etc/inittab-premaint cp /etc/motd /etc/motd-premaint cp /etc/ssh/sshd_config /etc/ssh/sshd_config-premaint cd /software/javasshssl-aug2022 smitty update_all
The above will patch ssh, ssl and java as this is where those patches were placed.
cd /software/aixtl3sp02-2220-updates smitty install Select bos.rte.install and install it smitty update_all Install the updatesWhen these are complete check the SSH, SSL and Java levels using lslpp to make sure they are at the new levels. I noticed that after the most recent patching some of the openssh.msg files remained backlevel at 2105—I removed them as follows:
installp -u openssh.msg.CA_ES installp -u openssh.msg.CS_CZ installp -u openssh.msg.DE_DE installp -u openssh.msg.EN_US installp -u openssh.msg.ES_ES installp -u openssh.msg.FR_FR installp -u openssh.msg.HU_HU installp -u openssh.msg.IT_IT installp -u openssh.msg.JA_JP installp -u openssh.msg.Ja_JP installp -u openssh.msg.KO_KR installp -u openssh.msg.PL_PL installp -u openssh.msg.PT_BR installp -u openssh.msg.RU_RU installp -u openssh.msg.SK_SK installp -u openssh.msg.ZH_CN installp -u openssh.msg.ZH_TW installp -u openssh.msg.Zh_CN installp -u openssh.msg.Zh_TW lslpp -l | grep ssh | grep 2105 lslpp -l | grep sshNow use diff to see if any of the files saved earlier got changed:
diff /etc/inetd.conf /etc/inetd.conf-premaint diff /etc/inittab /etc/inittab-premaint diff /etc/motd /etc/motd-premaint diff /etc/ssh/sshd_config /etc/ssh/sshd_config-premaintIf changes were made and need to be reverted then save the changed file and copy the one ending premaint over the top.
refresh -s inetd stopsrc -s sshd startsrc -s sshdAfter updating SSL you should always run updtvpkg
I then run “dnf check” to make sure there are no DNF issues.
Finally, I run the following:
oslevel -s 7200-05-04-2220 oslevel -s -l 7200-05-04-2220 lppchk -v lppchk -vm3 instfix -i | grep MLThere should be no missing filesets. If everything is fine then it is time to run flrtvc to find any needed efixes/ifixes. The default script is flrtvc-085.ksh. Since most of my LPARs cannot download from IBM, I create a copy called flrtvc-085-nodl.ksh and edit it to change SKIP_DOWNLOAD=1 to tell it to use the current apar.csv file and not to download it from IBM. I always use the NIM LPAR prior to any maintenance to update this file.
cd /software/flrtvc ./flrtvc-085-nodl.kshThis will provide you with a list of efixes needed for your system along with links to get them. For 188.8.131.52 (8/23/2022) these consist of patches for python3 (we fixed this above), bind_fix21 and openssl_fix36. Depending on when it is run additional fixes may be needed. These fixes are normally installed with emgr. Below is an example of installing the bind_fix21 efix.
cd bind_fix21 emgr -p -e IJ40614m2b.220718.epkg.ZThe above does a verification—if it is successful then the actual installation is done as follows:
emgr -e IJ40614m2b.220718.epkg.ZMy final step is to run ”dnf update” to update any third-party RPMs from the toolbox. When that is done I run “dnf check” and “updtvpkg”.
If everything looks good it is time to reboot:
lspv | grep root hdisk1 00f95d3a425513d5 rootvg active hdisk2 00f95d3a421072a7 altinst_rootvg bosboot -a -d hdisk1 bootlist -m normal hdisk1 bootlist -m normal -o shutdown -r nowOnce the system is back up run all the checks you would normally run.
If there are issues set the bootlist back to the cloned copy and reboot.
bootlist -m normal hdisk2 bootlist -m normal -o shutdown -r now
General Update AdviceIt is important to keep AIX patched to avoid known problems and security issues. Everything above can be done using NIM bundles or alternate disk install and some can be done using AIX live update. There are also systems like Bigfix to automate much of this as well. The purpose of this article is to show a process for updating that is tried and true and includes ensuring that the system is fully patched. In addition to the above, there are potential updates to I/O firmware for the LPAR if it has dedicated resources. The approach for the VIO servers is very similar except you should use updateios for both the operating system updates as well as the efixes/ifixes.
I pre-document all my installs and updates so they go smoothly. Part of that is documenting network, etc. settings in case I have to recover anything. Hopefully, this will help you streamline your patching process.
About the author
Jaqui Lynch has over 38 years of experience working with a projects and OSes across vendor platforms, including IBM Z, UNIX systems and more.
See more by Jaqui Lynch