A Look at the Latest AIX 7.3 Enhancements

In December 2022, AIX 7.3 Technology Level (TL) 1 became available. Many new features and functions were delivered with this TL. You can find the official list of announcements and “what’s new” information at the following links:

• IBM AIX 7.3 TL1, IBM AIX 7.2 TL5, IBM AIX 7 Standard Edition, IBM AIX 7 Enterprise Edition 1.9, IBM Private Cloud Edition 1.9, and IBM Private Cloud Edition 1.9 with AIX 7 have been updated and enhanced
• What’s new in AIX 7.3:
• AIX 7.3 Release Notes
• AIX 7.3 Expansion Pack Release Notes

I wanted to take a moment to talk about some of the new enhancements that caught my attention with AIX 7.3 TL1.

AIX Dump Exploitation of Power HW GZIP

The AIX system dump facility has been enhanced to improve performance using hardware-accelerated compression on IBM POWER9 and Power10 systems. The AIX Firmware Assisted Dump (FWAD) can now be compressed using the on-chip (POWER9 and later) Nest Accelerator (NX) GZIP compression accelerator. This is available with AIX 7.3 TL1. This feature is enabled by default when your logical partition (LPAR) is configured to run in POWER9 or Power10 mode (i.e. running in “POWER9” or “Power10” processor compatibility mode). In this mode, the LPAR has access to the NX GZIP accelerator.

By using this hardware capability, the time required to compress an AIX system dump image can be reduced significantly. The overall time require to capture an AIX system dump remains unchanged, but the dump compression time is improved. This type of dump is only supported with FWAD and not traditional AIX dump.

root@aix73 / # oslevel -s
7300-01-01-2246

root@aix73 / # sysdumpdev
primary              /dev/lg_dumplv
secondary            /dev/sysdumpnull
copy directory       /var/adm/ras
forced copy flag     TRUE
always allow dump    FALSE
dump compression     ON
type of dump         fw-assisted
full memory dump     disallow
enable NX GZIP       TRUE

AIX sysdumpdev Command

The AIX sysdumpdev command includes two new options for enabling and disabling NX GZIP dump compression. The -n flag disables the feature, while the -N flag enables NX GZIP compression.

sysdumpdev:       
-n: Disables the Nest Accelerators (NX) GZIP dump compression. The system dump is compressed without using the NX GZIP.

-N: Enables the Nest Accelerators (NX) GZIP accelerated dump compression. NX GZIP dump compression is only applicable to firmware-assisted type system dumps.

You can refer to the sysdumpdev command reference here.

Just a reminder, you can confirm if your LPAR is running in the correct mode using nxstat -S. Use this command to check for the availability of the NX GZIP accelerator. Refer to the nxstat command reference here.

Example Scenarios

In the example below, we observe that the accelerator is available, confirming that our LPAR is running in the correct processor mode.

# nxstat -S
nx_accel_mask = 1

GZIP accelerator available

In the next example, we notice that the accelerator is not available, confirming that our LPAR is not running in the correct processor mode. In this instance, NX GZIP-capable AIX system dumps would not be possible.

# nxstat -S
nx_accel_mask = 0

** No accelerators available **
** Accelerators are not available for partitions in POWER9_base mode

AIX Fast Failover for Fibre Channel Adapters

AIX 7.3 TL1 introduced support for 16Gbps and faster Fibre Channel (FC) adapters, including NPIV Virtual Fibre Channel (VFC) clients, to improve the time for I/O failover. The new feature allows AIX to fail I/O to alternative paths when link events are detected on either the host or target sides. This new capability is disabled by default and needs to be enabled. The good news is you can enable this option concurrently, i.e., without an outage, using the “chdev -U” command.

When this feature is enabled the Fibre Channel SCSI protocol (fscsiX) driver quickly fails over I/O operations to the SCSI device driver when the host or storage link is down. This failover operation helps the SCSI device driver to switch I/O operations to alternative paths.

This tunable parameter is only used for a Multiple Path I/O (MPIO) environment where another path to failover is available. As a best practice, you should have an alternative path through another FC adapter to handle FC adapter hardware failure scenarios. In a non-MPIO environment, it is recommended that administrators do not enable this new feature and that the default settings are left unchanged.

To facilitate the fast failover for FC adapters, the fast_lnk_recov attribute has been introduced to enable/disable this option with the fscsiX AIX device. This is turned on by setting the fast_lnk_recov attribute on the fscsiX devices to “yes”, instead of the default “no.”  In testing, IBM found a typical reduction from ~15 seconds to ~5 seconds for retrying the I/O on a different path.

root@aix73 / # lsattr -Rl fscsi0 -a fast_lnk_recov
yes
no

root@aix73 / # lsattr -El fscsi3 -a fast_lnk_recov
fast_lnk_recov no Quick I/O failure for FC link failure case True+

root@aix73 / # chdev -l fscsi3 -a fast_lnk_recov=yes -U

root@aix73 / # lsattr -El fscsi3 -a fast_lnk_recov
fast_lnk_recov yes Quick I/O failure for FC link failure case True+

You can refer to the disk and disk adapter tunable parameters reference here for more information on fast_lnk_recov.

AIX JFS2 Log Shuffle

This new feature allows AIX administrators to use chfs to convert or swap existing JFS2 file systems to either inline or external (outline) log devices. There is a new option for chfs called logshuffle. Using this option, an administrator can specify which log device a file system should use. This capability is only available with AIX 7.3. You can find more information about this new option in the chfs command reference material here.

# man chfs
...
-a logshuffle={INLINE | logdevicename}

Sets a file system to use the specified log. The specified log device must be in the same volume group as the current log device. If you specify logshuffle=INLINE, the logical volume will be extended to create an inline log device of the default size (0.4% of the file system, size up to 2047 MB) for the file system. Specifying an outline log device does not shrink the logical volume.

When using logshuffle, please note that the file system is quiesced while the log change is taking place. The file system does not need to be unmounted for the change to take effect, but it is quiesced to ensure file system data integrity.

You may be asking, but why would an administrator want to use this feature? One of the main reasons is when an administrator needs to switch from external logging to JFS2 inline logging. When there are multiple filesystems using/sharing the same log device (in a volume group), this can cause a performance bottleneck. It is often more optimal to provide each file system with its own inline JFS2 log device.

The logshuffle feature allows administrators to switch logging devices, on existing file systems, without having to create a new file system with an inline log and migrate the user data over to the new file system. Below is an example of swapping the log device for a file system, from external to inline:

root@aix73 / # oslevel -s
7300-01-01-2246

root@aix73 / # lsfs -q /myfs
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/mylv       --         /myfs                  jfs2  32768   --         yes  no
  (lv size: 32768, fs size: 32768, block size: 4096, sparse files: yes, inline log: no, inline log size: 0, EAformat: v1, Quota: no, DMAPI: no, VIX: yes, EFS: no, ISNAPSHOT: no, MAXEXT: 0, MountGuard: no, LFF: no)

root@aix73 / # mount | grep myfs
/dev/mylv        /myfs            jfs2   Dec 06 13:33 rw,log=/dev/myjfs2loglv

root@aix73 / # chfs -a logshuffle=INLINE /myfs

root@aix73 / # mount | grep myfs
         /dev/mylv        /myfs            jfs2   Dec 06 13:33 rw,log=INLINE

root@aix73 / # lsfs -q /myfs
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/mylv       --         /myfs                  jfs2  33280   --         yes  no
  (lv size: 65536, fs size: 33280, block size: 4096, sparse files: yes, inline log: yes, inline log size: 0.25, EAformat: v1, Quota: no, DMAPI: no, VIX: yes, EFS: no, ISNAPSHOT: no, MAXEXT: 0, MountGuard: no, LFF: no)

Reset Hypervisor Call Statistics

AIX 7.3 introduces a new flag (-G) to the lparstat command. This new option allows an administrator to reset the hypervisor call (HCALL) statistics reported by the lparstat -H command. The -G flag calls the config_perfmon API to reset HCALL statistics.

The lparstat -H command provides statistics for each of the hypervisor calls. It tracks the number of calls, total time spent, max call time and so forth for each of the HCALLs. This new feature “lparstat –G” will reset the hypervisor statistics. This has not been possible with previous versions of AIX. Sometimes during troubleshooting, it is useful to track the time and number of HCALLs during a performance monitoring window. This capability allows administrators to reset the stats as needed when debugging or testing on their AIX 7.3 systems.

In the example below we run the “lparstat -H” command and check for a particular hypervisor call. We then reset the statistics for all HCALLs and run the ‘lparstat -H’ command again to verify the statistics have been reset to zero for all HCALLs.

root@aix73 / # lparstat -H | head -7 ; lparstat -H | grep h_send_sub_crq ; lparstat -G ; lparstat -H | head -7 ; lparstat -H | grep h_send_sub_cr>
System configuration: type=Shared mode=Uncapped smt=8 lcpu=8 mem=4096MB psize=20 ent=0.25
           Detailed information on Hypervisor Calls
Hypervisor                  Number of    %Total Time   %Hypervisor   Avg Call    Max Call
  Call                        Calls         Spent      Time Spent    Time(ns)    Time(ns)
h_send_sub_crq                   2            0.0           0.0        1156         1375
System configuration: type=Shared mode=Uncapped smt=8 lcpu=8 mem=4096MB psize=20 ent=0.25
           Detailed information on Hypervisor Calls
Hypervisor                  Number of    %Total Time   %Hypervisor   Avg Call    Max Call
  Call                        Calls         Spent      Time Spent    Time(ns)    Time(ns)
h_send_sub_crq                   0            0.0           0.0          0           0

Of course, this is just the tip of the iceberg. There are many more new features and functions which are well worth understanding with AIX 7.3 TL1. So, please take the time to read the announcements and “what’s new” information in more detail.

---