FLRT, NIM, DNF, HMC and More: Technical Lessons From 2023
2023 has been incredibly busy, but I have also learned a lot during that time. It was great to attend TechXchange, see everyone in person and share ideas and thoughts with everyone. In this article I have tried to distill some of the things I learned throughout the year:
- Updates to tools
- How to install an ifix into a NIM spot
- Information on a CPU setting
- DNF information
- Dealing with duplicate connections on the HMC
- How to find things at IBM
Important Tool Updates in 2023
FLRT
The FLRT (fix level recommendation tool) pages are being constantly updated. From the home page there are multiple links that allow you to enter your configuration and levels to get recommendations on updates. Additionally, you can look at the data tables to make sure you are installing supported levels and combinations for software, hardware and firmware. You can also check the dates that support ends for various hardware and software.
FLRTVC
FLRTVC (fix level recommendation tool vulnerability checker) is a tool that you run on the AIX LPAR or VIO server to check for known vulnerabilities. It has been updated to use secured FTP, which is now required by IBM. The output provides a list of vulnerabilities and the addresses to download them from at IBM. The latest level is v0.8.6 which includes the changes to use secured FTP.
IBM is changing the public IP addresses for the IBM download servers that are used for many of the support downloads. As part of this, the public FTP URL has been changed from ftp.software.ibm.com to public.dhe.ibm.com. Additionally, some of the IP addresses have changed or will be changing. There are new IP addresses for public.dhe.ibm.com and service.boulder.ibm.com. IP addresses are also changing for the IBM servers that support Call Home and electronic download of fixes for customer systems’ software, hardware and operating system. These new IP addresses will need to be added to your firewall, but don’t remove the old ones as they are still used for other delivery operations.
You can find more about these changes at:
Changes to IBM Software Download servers
Preparing customer firewalls and proxies for the upcoming infrastructure changes: make sure to add Call Home and Electronic Fix Distribution to your firewalls.
For the new secured FTP, the FTP server will respond with a port number in the range 65024-65535, so please make sure that the ports in the range 65024-65535 are kept open to receive the response from FTP server.
HMCScanner 50
The HMCScanner is one of my favorite tools to use to document all the servers and partitions that an HMC can see. It has been updated to support Power10 and some issues with SRIOV. If you are not on 0.11.50 you should download it and use the new version. This level of the HMCScanner requires Java 1.8 or later.
NMON Analyser
Although it has not been updated for a while, I want to mention NMON Analyser. This produces a Microsoft Excel spreadsheet that takes an output file from nmon for Linux and/or NMON for AIX and produces some nice graphs to aid in the performance monitoring, analysis and report writing.
PowerDraw
IBM just announced the availability of PowerDraw, a product that provides customers with an interactive graphical representation of their Power Systems.
It collects information about the server, VIO servers and LPARs from the HMC and then creates drawings of the environment. PowerDraw is available from IBM’s Lab Solutions group.
Installing an Interim Fix into a NIM SPOT Resource
I perform many of my installs and updates using NIM, but always end up putting on efixes and ifixes manually after the installation. It turns out you can install an interim fix into a NIM SPOT resource so you don’t have to put it on after the install. After you have built your LPP and SPOT, put the efixes into the efix directory for the LPP and then use nim cust to install it as follows:
i.e for AIX 7.2.5.6 to install ij46487 into the SPOT mkdir /nim/lpp_source/lpp_aix72tl5sp6/emgr/ppc cd /usr/local/software/flrtfixes/ij46487 cp IJ46541s6a.230503.AIX72TL05SP06.epkg.Z /nim/lpp_source/lpp_aix72tl5sp6/emgr/ppc Now customize the SPOT as follows: nim -o cust -a lpp_source=lpp_aix72tl5sp6 -a filesets=IJ46541s6a.230503.AIX72TL05SP06.epkg.Z spot_aix72tl5sp6 You can list the efixes in a SPOT with: nim -o lslpp -a lslpp_flags=e spot_aix72tl5sp6 You can uninstall the fix using: nim -Fo maint -a installp_flags=u -a filesets=IJ46541s6a spot_aixtl5sp6
CPU Setting for ShMode
When you look at the HMCScanner output for a server there is a tab called LPAR_Profiles. For each LPAR you will see a column (usually W) called ShMode. For dedicate processor LPARs it will be set to one of share_idle_procs, keep_idle_procs, share_idle_procs_active or share_idle_procs_always. For shared processor pool LPARs it will be set to uncap, uncap_weight or cap.
Keep_idle_procs – never share processors Share_idle procs – share processors only when the LPAR is inactive Share_idle_procs_active – share processors only when the partition is active Share_idle_procs_always – always share processors
Cap – capped
Uncap – uncapped
There is no description provided for uncap_weight
The only LPARs that grow and shrink dynamically are shared processor pool LPARs, so share_idle_procs and keep_idle_procs basically do nothing if all the LPARs and VIO servers are dedicated. If you have shared processor pool LPARs then they can use CPUs that were dedicated to the dedicated LPARs depending on the dedicated LPAR status.
share_idle_procs_active turns the dedicated LPAR into a dedicated donating LPAR, which impacts several things even if there are no shared LPARs on the system. This is because the LPAR is constantly ceding cores if it is not using them, which gives the hypervisor/dispatcher more to do. You may see VCSWs, which indicates that cores are being ceded. You may also see the physc varying rather than staying at its dedicated allocation, which again means cores are being ceded. If there are no shared LPARs on a server then why cede CPU cycles? You should also note that NMON reports CPU% differently as it knows CPU could vary.
For dedicated cores NMON CPU% is based on %used
For dedicated donating NMON CPU% is based on physcpu/vcpu
I compared 2 NMON reports and saw 24.9% CPU on the dedicated and 72.8% on the dedicated donating but they were both actually using the same amount of CPU (physc) resources.
DNF
DNF is the replacement for YUM that uses the latest version 3 of Python. DNF has been updated to use the correct IBM servers so you may want to redownload the installer.
Below are a few recommendations to avoid issues with DNF. Before doing any of this take a mksysb backup or a clone (alt_disk_copy -V -B -d hdisk?).
DNF can have issues over time with the repository, so I periodically run “dnf clean all” to clean the repository. Then I run “dnf update” to reload the repository.
When running DNF you may experience errors about libssl or other SSL related libraries. This is normally because LIBPATH is set, and it finds a different product’s version instead.
Make sure LIBPATH is not set – if it is then unset it:
env | grep LIBPATH export LIBPATH=
The latest version of DNF requires the latest version of rpm which needs to be downloaded from the Web Downloads page. RPM should be updated prior to updating DNF to avoid errors.
Check the current level:
lslpp -f rpm.rte
lslpp -L rpm.rte
The latest levels are:
AIX 7.3 - 4.15.1.2009
AIX 7.1/7,2 - 4.15.1.1011
After updating rpm run “updtvpkg”
Other useful DNF commands are:
dnf createrepo /opt/freeware/etc/dnf/dnf.conf dnf list dnf update dnf check dnf clean all
To install DNF
First download dnf_aixtoolbox.sh from the AIX toolbox and make it executable
Make sure LIBPATH is not set
export LIBPATH=
If you run the script with -? it provides help information.
# ./dnf_aixtoolbox.sh -?
Usage: dnf_aixtoolbox.sh <-d> <-y> <-n> -?
-d Install and setup DNF if YUM is not installed.
-y Update YUM3 to YUM4(softlink to dnf).
Use this option if want to redirect all YUM commands to DNF.
Existing YUM-3.4.3 will be updated to yum-4.x.x(dnf)
-n Install DNF where both YUM and DNF can coexist if YUM is installed already.
This is not a recommended option.
-y or -n require you to run DNF migrate prior to using any DNF commands.
DNF uses the repository conf file /opt/freeware/etc/dnf/dnf.conf.
If you get errors regarding installed conflict with Python, such as:
python3-dnf-plugin-migrate-4.0.16-32_51.ppc has installed conflict "python3-dnf-plugin-migrate < 4.0.16-64_1": python3-dnf-plugin-migrate-4.0.16-32_51.ppc python3-dnf-plugins-core-4.0.16-32_51.ppc has installed conflict "python3-dnf-plugins-core < 4.0.16-64_1": python3-dnf-plugins-core-4.0.16-32_51.ppc Error: Check discovered 2 problem(s)
The fix is:
dnf remove python3-dnf-plugin-migrate python3-dnf-plugins-core
It should remove 4 Packages
Then:
dnf update dnf check
Run DNF repolist to list the repositories to make sure they are what you expect to see.
If you see duplicate errors when running DNF check or DNF update then you need to remove those using:
dnf remove –duplicates
To ensure RPM is aware of the vital binaries and shared objects installed through installp filesets, a virtual package (AIX-rpm) is created which will contain these shared objects, shells and binaries. This package is created and updated by the ‘/usr/sbin/updtvpkg’ script. updtvpkg is run automatically when rpm.rte is installed or updated but not when other filesets such as openssl are installed or updated. It is recommended that you run updtvpkg whenever you patch systems or update SSL.
If you have problems, check out the latest community post or visit this page.
Duplicate Connections on HMC
I recently had a weird problem where the server was behaving strangely. After opening a case with IBM, we figured out that it had two connections defined to the HMC. We discovered this by using:
lssysconn -r all
This showed the same server twice with one of the connections not valid.
resource_type=sys,type_model_serial_num=8286-41A*21XXXXX,sp_type=fsp,sp=primary,sp_phys_loc=U78C9.001.WZS0234-P1,ipaddr=10.1.255.2,alt_ipaddr=unavailable,state=Connected resource_type=sys,type_model_serial_num=8286-41A*21XXXXX,sp_type=fsp,sp=unavailable,sp_phys_loc=unavailable,ipaddr=10.1.255.3,alt_ipaddr=unavailable,state=No Connection,connection_error_code=Connecting 0000-0000-00000000
The second connection is not valid, so the fix was to remove that extra connection and clean up:
rmsysconn -o remove --ip 10.1.255.3 lssysconn -r all mksysconn -o auto lssysconn -r all
We ran the mksysconn command to enable the HMC to automatically discover all the systems and frames. You may need to do this if the Remove Connection task or the rmsysconn command was previously run to remove a system or frame from the HMC, and you want the HMC to be able to manage a system or frame with the same IP address again. Whenever a managed system or a managed frame is removed from the HMC when using DHCP, the HMC places the IP address(es) of that system or frame in a list of removed IP addresses. Any IP address in that list will not be automatically rediscovered by the HMC. The mksysconn -o auto command removes all IP addresses from that list. To remove a specific IP address from that list, you can run the rmsysconn -o rediscover command instead.
Finding Things at IBM
Over the years I have discovered that as soon as I know where to find something at IBM it magically moves. Below is a list of locations that you need to find some of the tools you may want to access.
AIX TOOLBOX
DNF and most of the open source for AIX is found here or here.
You can access this using ftp -s and anonymous. Just make sure to set your session to bin and passive
AIX, PowerVM, Etc. Base Code
This is found at ESS (Entitled Systems Support). You will find the software base code, update access keys, inventory explorer and other useful items here.
AIX Security Patches
Efixes, ifixes and security fixes are found at this link.
The syntax above shows the list of security patches so that the most recent ones are at the top.
Java and Spectrum Scale
These are downloaded from Fix Central.
To download Java updates, go to Fix Central then:
Find product, type in Java
Select Runtimes for Java Technology
Select the base level i.e. 8.0.0.0 or 7.0.0.0
You will have to select AIX 32 bit and 64 bit separately
As of 10/15/2023 latest 8.0 is 8.0.0.810 and 7.0 is 7.0.0.715
To download Spectrum updates, go to Fix Central then:
Find product, Spectrum Scale
Select Spectrum Scale (software defined storage)
Select the version and platform (I chose 5.1.7 and AIX)
If you are at a level prior to 5.1.7 you need to download 5.1.7.0-ppc64-AIX-install and 5.1.7.1-ppc64-AIX-update
Updates to Various Software Products
These are found at the Web Downloads site.
Updates can be found here for lsof, bind, xgzip, SMB Client for AIX, RPM, Python, Perl, OpenSSL, Openssh, NTP v4, IBM Network Authentication Service for AIX (krb5 1.16.1.6), IBM Security Directory Server V6.4, Db2 and GSKit 6.4.0.4 (idsldap).
Some of the applications as of 10/15/2023 are:
lsof – 4.892 useful tool for seeing who is using a file
bind 9.16.26.02 – patches security holes in bind
xgzip – zlib library-based compression utility for AIX 7.2.4 and above.
smb client for aix (smbc) – 3.0.2 – different files for AIX 7.1 vs 7.2/7.3
python3 for AIX 7.3 – 3.9.17.0 – patches security holes
openssl – 3.0.10.1000 for AIX 7.1 and higher, 1.1.2.2000 with no weak ciphers support for AIX 6.1 and higher
openssh – 9.2.112.2000 – both compiled with openssl 1.1.1
NAS (krb5) latest is 1.16.1.6 – needed for LDAP and kerberos authentication
LDAP – IBM Security Directory Server V6.4, DB2 & GSKit 6.4.0.4
RPM Package manager – latest patched versions for AIX – needed for DNF
AIX 7.3 – 4.15.1.2009
AIX 7.1/7,2 – 4.15.1.1011
HMC Code
HMC code can be downloaded here.
network – network install files
fixes – patches such as MF71107_ppc.iso
recovery_images – Reinstall recovery images
updates – Updates to a new release
As I mentioned, I have learned a lot this year and had to find my way around a few problems. Hopefully, this article will help you avoid spending time solving or hunting for the same things.
References
Getting Started With the AIX Toolbox for Open-Source Software
AIX Open Source Tip: Potential DNF Errors When Migrating to AIX 7.3
You can download dnf_aixtoolbox.sh and the bundles here.
FLRTVC 086 ZIP
Changes to IBM Software Download Servers
Changes to IBM Software Download Servers, Continued