Skip to main content

Maintaining Your AIX Systems

Unfortunately, there are a number of people out there testing your security so it is critical that your systems are regularly updated and patched to avoid problems.  Normally maintenance consists of updating the HMC (both software and the BMC/PNOR), server firmware, I/O adapter firmware, VIO servers and your AIX system. In all cases, there may be prerequisites and a mandated order for the updates. In this article, we will discuss maintaining and patching AIX LPARs.

AIX Maintenance Methods

If you are staying within a release level, then patching can easily be done using a number of methods such as NIM, AIX Live Update, alternate disk install, or in-place updates. The simplest is in-place updates which I will discuss below.

AIX v7.3 Upgrade

If you are changing release levels (AIX 7.2 to 7.3, as an example) then the update is a little more complex. You can take a mksysb and use NIM to update it to an alternate disk, or you can boot from the DVD or ISO image (ISO from the VIO) and do a migration upgrade.
 
For third-party products I now use DNF instead of YUM or RPM to install them. This ensures that prerequisites and corequisites are taken care of. When updating to AIX 7.2.5.3  to AIX 7.3 you will need to reinstall the new version of DNF as AIX v7.3 uses the 64-bit version of DNF, not the 32-bit one previously used. Make sure you download the latest version of the DNF install script from here:
https://public.dhe.ibm.com/aix/freeSoftware/aixtoolbox/ezinstall/ppc/dnf_aixtoolbox.sh
 
Details on upgrading to AIX 7.3 and HMC v10 are covered in my January 2022 article, so I will include the link to them here:
https://techchannel.com/SMB/01/2022/aix-7-3-hmc-v10-upgrades

Putting Maintenance on AIX at the Same Release Level

I always start with my NIM LPAR as it is the one system that can download patches from IBM. The first step is to run flrtvc on the NIM LPAR so I download the latest apar.csv file and it tells me what patches I need for the NIM LPAR at the level it is currently running. Below is the process I followed to do an in-place update of my NIM LPAR. The next step is to get an inventory of where the system is at before starting.
 
Using lslpp I find:

SSL is at 1.1.1.1200

SSH is at 8.1.102.2105
Java7 is 7.0.0.705
Java8 is 8.0.0.705

oslevel -s
7300-00-01-2148 

I also check for efixes already installed:

emgr -P
PACKAGE                                                  INSTALLER   LABEL
======================================================== =========== ==========
bos.cluster.rte                                          installp    IJ36596s1a
bos.rte.security                                         installp    IJ36816s1a
bos.mp64                                                 installp    IJ37001m1a
bos.net.tcp.bind_utils                                   installp    IJ37226s1b
bos.net.tcp.bind                                         installp    IJ37226s1b
bos.perf.tune                                            installp    IJ38518s1b
bos.pfcdd.rte                                            installp    IJ37403s1a
bos.pmapi.pmsvcs                                         installp    IJ37411s1a
bos.rte.security                                         installp    IJ38121m1a
bos.sysmgt.nim.client                                    installp    IJ36593m1a

Continuing Maintenance on AIX After Upgrades

At this point, I want to upgrade AIX from 7.3.0.1 to 7.3.0.2 and I also need to patch Python3, SSL, SSH and Java. I will also want to update any RPMs using DNF.
 
Java and the AIX ML are downloaded from Fix Central.
The AIX level I am downloading is 7300-00-02-2220, which became available on 5/20/2022.
 
For Java I download the 32-bit and 64-bit Java runtimes
For Java8 this is 8.0.0.711 – 6/28/2022
For Java7 this is 7.0.0.715 – 8/11/2022
 
SSH and SSL are downloaded from the IBM Web Download page. You can also download Python3 for AIX 7.3 here—there is a security issue with the default Python3 so you should download this.
 
Python3         3.9.12.0        
OpenSSL       1.1.2.1201 with no weak ciphers support
OpenSSH      8.1.112.1201
 
If you are not at these levels on AIX 7.3 then you need to download them and update them.
 
The IBM Web Download page is also where you can download lsof, Xgzip, SMB client, NTPv4, the LDAP client code, KRB5 client code and several other applications.
 
To download the full ISO image of AIX you will need to go to Entitled Software Support (ESS) at: https://www.ibm.com/servers/eserver/ess/index.wss

Preparing for Updates

Note that IBM has updated many of their download pages now to require both signing in with your IBM account and using MFA with IBM Verify. Make sure you have this set up on your device prior to starting your downloads. The IBM web download page is one that requires IBM verification before you can even check levels.
 
Once all the software and patches are downloaded it is time to start the updates.  I use NFS to share out patches from /software on the NIM LPAR to the client LPARs. I untar all the python3, java, ssh and ssl updates into one directory (in this case javasshssl-aug2022). I untar the AIX service pack into aixtl3sp02-2220-updates.
 
Before doing any maintenance it is important to run errpt to make sure there are no errors that could affect the upgrade. Any disk or other errors should be fixed prior to continuing. You should also check the readmes for the updates.
 
I usually run a mksysb to tape and then I take a clone so my recovery can be a fast reboot.

lspv | grep root
hdisk1          00f95d3a425513d5                    rootvg          active
hdisk2          00f95d3a421072a7                    altinst_rootvg

In the section above, you can see hdisk1 is our boot image and hdisk2 is the alternate. If you have them mirrored (both showing rootvg) then you will need to break the mirror to free up one of the disks. Even though they are not mirrored I still do the following: 

uptime
bootinfo -b
bootlist -m normal -o
bosboot -a -d hdisk1
bootlist -m normal hdisk1
bootlist -m normal -o

The first two commands show me how long the system has been up and which disk it booted from. In a mirrored rootvg this is more important. 

# uptime
  11:54AM   up 101 days,  18:54,  1 user,  load average: 1.54, 1.20, 1.51
# bootinfo -b
hdisk1

Before I do maintenance I check and rewrite the bootlist, especially if it has not been rebooted for a while. In this case, it was 101 days, but I probably would rewrite it anyway as I have been caught out by damaged boot images and bootlists before.
 
Now I take my clone:

exportvg altinst_rootvg
alt_disk_copy -V -B -d hdisk2

In the section above, I free up hdisk2 so I can use it for the clone. I then tell it to take a copy to hdisk2 with all the details (V=verbose). The -B tells it not to change the bootlist.
 
When that is complete we can start the update. The first step is to remove any currently installed efixes/ifixes. emgr -P provides that list:

emgr -P
PACKAGE                                                  INSTALLER   LABEL
======================================================== =========== ==========
bos.cluster.rte                                                      installp    IJ36596s1a
bos.rte.security                                        installp    IJ36816s1a
bos.mp64                                                 installp    IJ37001m1a
bos.net.tcp.bind_utils                               installp    IJ37226s1b
bos.net.tcp.bind                                       installp    IJ37226s1b
bos.perf.tune                                            installp    IJ38518s1b
bos.pfcdd.rte                                            installp    IJ37403s1a
bos.pmapi.pmsvcs                                  installp    IJ37411s1a
bos.rte.security                                        installp    IJ38121m1a
bos.sysmgt.nim.client                             installp    IJ36593m1a

They are removed as follows:

emgr -r -L IJ36596s1a
emgr -r -L IJ37001m1a
emgr -r -L  IJ36816s1a
emgr -r -L  IJ37226s1b
emgr -r -L IJ38518s1b
emgr -r -L  IJ37403s1a
emgr -r -L IJ37411s1a
emgr -r -L IJ38121m1a
emgr -r -L IJ36593m1a
 
emgr -P

It should now say, “There is no efix data on this system.”
 
At this point, I do some basic checks:

oslevel -s
oslevel -s -l 7300-00-01-2148
lppchk -v
lppchk -vm3
instfix -i | grep ML
instfix -icqk  7300-00_AIX_ML | grep :-:
dnf check
updtvpkg

These checks are to ensure there are no missing filesets or known problems before starting. Assuming all is well we then start the updates.
 
I save a couple of files that maintenance tends to replace:

cp /etc/inetd.conf /etc/inetd.conf-premaint
cp /etc/inittab /etc/inittab-premaint
cp /etc/motd /etc/motd-premaint
cp /etc/ssh/sshd_config  /etc/ssh/sshd_config-premaint
cd /software/javasshssl-aug2022
smitty update_all

 
The above will patch ssh, ssl and java as this is where those patches were placed.

cd /software/aixtl3sp02-2220-updates
smitty install
            Select bos.rte.install and install it
smitty update_all
            Install the updates

When these are complete check the SSH, SSL and Java levels using lslpp to make sure they are at the new levels. I noticed that after the most recent patching some of the openssh.msg files remained backlevel at 2105—I removed them as follows:

installp -u openssh.msg.CA_ES
installp -u openssh.msg.CS_CZ
installp -u openssh.msg.DE_DE
installp -u openssh.msg.EN_US
installp -u openssh.msg.ES_ES
installp -u openssh.msg.FR_FR
installp -u openssh.msg.HU_HU
installp -u openssh.msg.IT_IT
installp -u openssh.msg.JA_JP
installp -u openssh.msg.Ja_JP
installp -u openssh.msg.KO_KR
installp -u openssh.msg.PL_PL
installp -u openssh.msg.PT_BR
installp -u openssh.msg.RU_RU
installp -u openssh.msg.SK_SK
installp -u openssh.msg.ZH_CN
installp -u openssh.msg.ZH_TW
installp -u openssh.msg.Zh_CN
installp -u openssh.msg.Zh_TW
 
lslpp -l | grep ssh | grep 2105
lslpp -l | grep ssh

Now use diff to see if any of the files saved earlier got changed:

diff /etc/inetd.conf /etc/inetd.conf-premaint
diff /etc/inittab /etc/inittab-premaint
diff /etc/motd /etc/motd-premaint
diff /etc/ssh/sshd_config  /etc/ssh/sshd_config-premaint

If changes were made and need to be reverted then save the changed file and copy the one ending premaint over the top.

refresh -s inetd
stopsrc -s sshd
startsrc -s sshd

After updating SSL you should always run updtvpkg
 
I then run “dnf check” to make sure there are no DNF issues.
 
Finally, I run the following:

oslevel -s
7200-05-04-2220
oslevel -s -l 7200-05-04-2220
lppchk -v
lppchk -vm3
instfix -i | grep ML

There should be no missing filesets. If everything is fine then it is time to run flrtvc to find any needed efixes/ifixes. The default script is flrtvc-085.ksh. Since most of my LPARs cannot download from IBM, I create a copy called flrtvc-085-nodl.ksh and edit it to change SKIP_DOWNLOAD=1 to tell it to use the current apar.csv file and not to download it from IBM. I always use the NIM LPAR prior to any maintenance to update this file.

cd /software/flrtvc
./flrtvc-085-nodl.ksh

This will provide you with a list of efixes needed for your system along with links to get them. For 7.3.0.2 (8/23/2022) these consist of patches for python3 (we fixed this above), bind_fix21 and openssl_fix36. Depending on when it is run additional fixes may be needed. These fixes are normally installed with emgr. Below is an example of installing the bind_fix21 efix.

cd bind_fix21
emgr -p -e IJ40614m2b.220718.epkg.Z

The above does a verification—if it is successful then the actual installation is done as follows:

emgr  -e IJ40614m2b.220718.epkg.Z

My final step is to run ”dnf  update” to update any third-party RPMs from the toolbox. When that is done I run “dnf check” and “updtvpkg”.
 
If everything looks good it is time to reboot:

lspv | grep root
hdisk1          00f95d3a425513d5                    rootvg          active
hdisk2          00f95d3a421072a7                    altinst_rootvg

bosboot -a -d hdisk1
bootlist -m normal hdisk1
bootlist -m normal -o

shutdown -r now

Once the system is back up run all the checks you would normally run.
 
If there are issues set the bootlist back to the cloned copy and reboot.

bootlist -m normal hdisk2
bootlist -m normal -o
shutdown -r now

General Update Advice

It is important to keep AIX patched to avoid known problems and security issues.  Everything above can be done using NIM bundles or alternate disk install and some can be done using AIX live update. There are also systems like Bigfix to automate much of this as well. The purpose of this article is to show a process for updating that is tried and true and includes ensuring that the system is fully patched. In addition to the above, there are potential updates to I/O firmware for the LPAR if it has dedicated resources. The approach for the VIO servers is very similar except you should use updateios for both the operating system updates as well as the efixes/ifixes.
 
I pre-document all my installs and updates so they go smoothly. Part of that is documenting network, etc. settings in case I have to recover anything. Hopefully, this will help you streamline your patching process.

References