Maintaining Your AIX Systems
Unfortunately, there are a number of people out there testing your security so it is critical that your systems are regularly updated and patched to avoid problems. Normally maintenance consists of updating the HMC (both software and the BMC/PNOR), server firmware, I/O adapter firmware, VIO servers and your AIX system. In all cases, there may be prerequisites and a mandated order for the updates. In this article, we will discuss maintaining and patching AIX LPARs.
AIX Maintenance Methods
If you are staying within a release level, then patching can easily be done using a number of methods such as NIM, AIX Live Update, alternate disk install, or in-place updates. The simplest is in-place updates which I will discuss below.
AIX v7.3 Upgrade
If you are changing release levels (AIX 7.2 to 7.3, as an example) then the update is a little more complex. You can take a mksysb and use NIM to update it to an alternate disk, or you can boot from the DVD or ISO image (ISO from the VIO) and do a migration upgrade.
For third-party products I now use DNF instead of YUM or RPM to install them. This ensures that prerequisites and corequisites are taken care of. When updating to AIX 7.2.5.3 to AIX 7.3 you will need to reinstall the new version of DNF as AIX v7.3 uses the 64-bit version of DNF, not the 32-bit one previously used. Make sure you download the latest version of the DNF install script from here:
https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/dnf_aixtoolbox.sh
Details on upgrading to AIX 7.3 and HMC v10 are covered in my January 2022 article, so I will include the link to them here:
https://techchannel.com/SMB/01/2022/aix-7-3-hmc-v10-upgrades
Putting Maintenance on AIX at the Same Release Level
I always start with my NIM LPAR as it is the one system that can download patches from IBM. The first step is to run flrtvc on the NIM LPAR so I download the latest apar.csv file and it tells me what patches I need for the NIM LPAR at the level it is currently running. Below is the process I followed to do an in-place update of my NIM LPAR. The next step is to get an inventory of where the system is at before starting.
Using lslpp I find:
SSL is at 1.1.1.1200
SSH is at 8.1.102.2105
Java7 is 7.0.0.705
Java8 is 8.0.0.705
oslevel -s
7300-00-01-2148
I also check for efixes already installed:
emgr -P
PACKAGE INSTALLER LABEL
======================================================== =========== ==========
bos.cluster.rte installp IJ36596s1a
bos.rte.security installp IJ36816s1a
bos.mp64 installp IJ37001m1a
bos.net.tcp.bind_utils installp IJ37226s1b
bos.net.tcp.bind installp IJ37226s1b
bos.perf.tune installp IJ38518s1b
bos.pfcdd.rte installp IJ37403s1a
bos.pmapi.pmsvcs installp IJ37411s1a
bos.rte.security installp IJ38121m1a
bos.sysmgt.nim.client installp IJ36593m1a
Continuing Maintenance on AIX After Upgrades
At this point, I want to upgrade AIX from 7.3.0.1 to 7.3.0.2 and I also need to patch Python3, SSL, SSH and Java. I will also want to update any RPMs using DNF.
Java and the AIX ML are downloaded from Fix Central.
The AIX level I am downloading is 7300-00-02-2220, which became available on 5/20/2022.
For Java I download the 32-bit and 64-bit Java runtimes
For Java8 this is 8.0.0.711 – 6/28/2022
For Java7 this is 7.0.0.715 – 8/11/2022
SSH and SSL are downloaded from the IBM Web Download page. You can also download Python3 for AIX 7.3 here—there is a security issue with the default Python3 so you should download this.
Python3 3.9.12.0
OpenSSL 1.1.2.1201 with no weak ciphers support
OpenSSH 8.1.112.1201
If you are not at these levels on AIX 7.3 then you need to download them and update them.
The IBM Web Download page is also where you can download lsof, Xgzip, SMB client, NTPv4, the LDAP client code, KRB5 client code and several other applications.
To download the full ISO image of AIX you will need to go to Entitled Software Support (ESS) at: https://www.ibm.com/servers/eserver/ess/index.wss
Preparing for Updates
Note that IBM has updated many of their download pages now to require both signing in with your IBM account and using MFA with IBM Verify. Make sure you have this set up on your device prior to starting your downloads. The IBM web download page is one that requires IBM verification before you can even check levels.
Once all the software and patches are downloaded it is time to start the updates. I use NFS to share out patches from /software on the NIM LPAR to the client LPARs. I untar all the python3, java, ssh and ssl updates into one directory (in this case javasshssl-aug2022). I untar the AIX service pack into aixtl3sp02-2220-updates.
Before doing any maintenance it is important to run errpt to make sure there are no errors that could affect the upgrade. Any disk or other errors should be fixed prior to continuing. You should also check the readmes for the updates.
I usually run a mksysb to tape and then I take a clone so my recovery can be a fast reboot.
lspv | grep root hdisk1 00f95d3a425513d5 rootvg active hdisk2 00f95d3a421072a7 altinst_rootvg
In the section above, you can see hdisk1 is our boot image and hdisk2 is the alternate. If you have them mirrored (both showing rootvg) then you will need to break the mirror to free up one of the disks. Even though they are not mirrored I still do the following:
uptime bootinfo -b bootlist -m normal -o bosboot -a -d hdisk1 bootlist -m normal hdisk1 bootlist -m normal -o
The first two commands show me how long the system has been up and which disk it booted from. In a mirrored rootvg this is more important.
# uptime 11:54AM up 101 days, 18:54, 1 user, load average: 1.54, 1.20, 1.51 # bootinfo -b hdisk1
Before I do maintenance I check and rewrite the bootlist, especially if it has not been rebooted for a while. In this case, it was 101 days, but I probably would rewrite it anyway as I have been caught out by damaged boot images and bootlists before.
Now I take my clone:
exportvg altinst_rootvg alt_disk_copy -V -B -d hdisk2
In the section above, I free up hdisk2 so I can use it for the clone. I then tell it to take a copy to hdisk2 with all the details (V=verbose). The -B tells it not to change the bootlist.
When that is complete we can start the update. The first step is to remove any currently installed efixes/ifixes. emgr -P provides that list:
emgr -P PACKAGE INSTALLER LABEL ======================================================== =========== ========== bos.cluster.rte installp IJ36596s1a bos.rte.security installp IJ36816s1a bos.mp64 installp IJ37001m1a bos.net.tcp.bind_utils installp IJ37226s1b bos.net.tcp.bind installp IJ37226s1b bos.perf.tune installp IJ38518s1b bos.pfcdd.rte installp IJ37403s1a bos.pmapi.pmsvcs installp IJ37411s1a bos.rte.security installp IJ38121m1a bos.sysmgt.nim.client installp IJ36593m1a
They are removed as follows:
emgr -r -L IJ36596s1a emgr -r -L IJ37001m1a emgr -r -L IJ36816s1a emgr -r -L IJ37226s1b emgr -r -L IJ38518s1b emgr -r -L IJ37403s1a emgr -r -L IJ37411s1a emgr -r -L IJ38121m1a emgr -r -L IJ36593m1a emgr -P
It should now say, “There is no efix data on this system.”
At this point, I do some basic checks:
oslevel -s oslevel -s -l 7300-00-01-2148 lppchk -v lppchk -vm3 instfix -i | grep ML instfix -icqk 7300-00_AIX_ML | grep :-: dnf check updtvpkg
These checks are to ensure there are no missing filesets or known problems before starting. Assuming all is well we then start the updates.
I save a couple of files that maintenance tends to replace:
cp /etc/inetd.conf /etc/inetd.conf-premaint cp /etc/inittab /etc/inittab-premaint cp /etc/motd /etc/motd-premaint cp /etc/ssh/sshd_config /etc/ssh/sshd_config-premaint cd /software/javasshssl-aug2022 smitty update_all
The above will patch ssh, ssl and java as this is where those patches were placed.
cd /software/aixtl3sp02-2220-updates
smitty install
Select bos.rte.install and install it
smitty update_all
Install the updates
When these are complete check the SSH, SSL and Java levels using lslpp to make sure they are at the new levels. I noticed that after the most recent patching some of the openssh.msg files remained backlevel at 2105—I removed them as follows:
installp -u openssh.msg.CA_ES installp -u openssh.msg.CS_CZ installp -u openssh.msg.DE_DE installp -u openssh.msg.EN_US installp -u openssh.msg.ES_ES installp -u openssh.msg.FR_FR installp -u openssh.msg.HU_HU installp -u openssh.msg.IT_IT installp -u openssh.msg.JA_JP installp -u openssh.msg.Ja_JP installp -u openssh.msg.KO_KR installp -u openssh.msg.PL_PL installp -u openssh.msg.PT_BR installp -u openssh.msg.RU_RU installp -u openssh.msg.SK_SK installp -u openssh.msg.ZH_CN installp -u openssh.msg.ZH_TW installp -u openssh.msg.Zh_CN installp -u openssh.msg.Zh_TW lslpp -l | grep ssh | grep 2105 lslpp -l | grep ssh
Now use diff to see if any of the files saved earlier got changed:
diff /etc/inetd.conf /etc/inetd.conf-premaint diff /etc/inittab /etc/inittab-premaint diff /etc/motd /etc/motd-premaint diff /etc/ssh/sshd_config /etc/ssh/sshd_config-premaint
If changes were made and need to be reverted then save the changed file and copy the one ending premaint over the top.
refresh -s inetd stopsrc -s sshd startsrc -s sshd
After updating SSL you should always run updtvpkg
I then run “dnf check” to make sure there are no DNF issues.
Finally, I run the following:
oslevel -s
7200-05-04-2220
oslevel -s -l 7200-05-04-2220
lppchk -v
lppchk -vm3
instfix -i | grep ML
There should be no missing filesets. If everything is fine then it is time to run flrtvc to find any needed efixes/ifixes. The default script is flrtvc-085.ksh. Since most of my LPARs cannot download from IBM, I create a copy called flrtvc-085-nodl.ksh and edit it to change SKIP_DOWNLOAD=1 to tell it to use the current apar.csv file and not to download it from IBM. I always use the NIM LPAR prior to any maintenance to update this file.
cd /software/flrtvc
./flrtvc-085-nodl.ksh
This will provide you with a list of efixes needed for your system along with links to get them. For 7.3.0.2 (8/23/2022) these consist of patches for python3 (we fixed this above), bind_fix21 and openssl_fix36. Depending on when it is run additional fixes may be needed. These fixes are normally installed with emgr. Below is an example of installing the bind_fix21 efix.
cd bind_fix21 emgr -p -e IJ40614m2b.220718.epkg.Z
The above does a verification—if it is successful then the actual installation is done as follows:
emgr -e IJ40614m2b.220718.epkg.Z
My final step is to run ”dnf update” to update any third-party RPMs from the toolbox. When that is done I run “dnf check” and “updtvpkg”.
If everything looks good it is time to reboot:
lspv | grep root
hdisk1 00f95d3a425513d5 rootvg active
hdisk2 00f95d3a421072a7 altinst_rootvg
bosboot -a -d hdisk1
bootlist -m normal hdisk1
bootlist -m normal -o
shutdown -r now
Once the system is back up run all the checks you would normally run.
If there are issues set the bootlist back to the cloned copy and reboot.
bootlist -m normal hdisk2 bootlist -m normal -o shutdown -r now
General Update Advice
It is important to keep AIX patched to avoid known problems and security issues. Everything above can be done using NIM bundles or alternate disk install and some can be done using AIX live update. There are also systems like Bigfix to automate much of this as well. The purpose of this article is to show a process for updating that is tried and true and includes ensuring that the system is fully patched. In addition to the above, there are potential updates to I/O firmware for the LPAR if it has dedicated resources. The approach for the VIO servers is very similar except you should use updateios for both the operating system updates as well as the efixes/ifixes.
I pre-document all my installs and updates so they go smoothly. Part of that is documenting network, etc. settings in case I have to recover anything. Hopefully, this will help you streamline your patching process.
References
- Fix Central
- Technote on DNF
- Fix Central
- AIX Linux Toolbox
- IBM Web Download Page
- A quick guide to DNF for YUM users
- DNF Install Script
- January 2022 Article on upgrading to HMC v10 and AIX 7.3
- FLRTVC 085