Apache Log4j Vulnerability, AIX Tips and Tools, a Look at Time Drift, and More

System security is no laughing matter. Well, occasionally it’s a laughing matter, but I’ll get to that in a moment. On Dec. 15 IBM issued a security bulletin regarding a vulnerability in Apache Log4j that affects the Power HMC. Check the IBM PSIRT blog for the latest updates:

“… Product teams are releasing remediations for Log4j 2.x CVE-2021-44228 as fast as possible, moving to the latest version that’s available when they are developing a fix. Where possible, the dependency on Log4j is removed entirely.

IBM is aware of additional, recently disclosed vulnerabilities in Apache Log4j, tracked under CVE-2021-45105 and CVE-2021-45046. Work continues to mitigate or remediate these vulnerabilities in products and services that already have released a remediation based on Log4j 2.15.

With so much active industry research on Log4j, mitigation and remediation recommendations will evolve. We are actively assessing the latest Log4j developments and will share updates accordingly.”

The blog also lists products that are confirmed not impacted, as well as products that have been remediated. Patch your systems as soon as possible.

Now for a little Log4j levity. In a reprise of the great jif/gif debates, admins are again arguing about pronunciation. Is it “log four jay”? “Log forge”? I’ve seen and heard both. Twitter has also weighed in on this very important matter:

Problem: Apache Log4j
Solution: A patchy Log4j
*crowd booing*

Encrypting AIX Logical Volumes

On Twitter, IBM’s Soumya Menon cites documentation about encrypting logical volumes in AIX:

“Starting with IBM AIX 7.2 with Technology Level 5, the Logical Volume Manager (LVM) supports the data encryption at the logical volume (LV) level. Using this feature, you can encrypt the data at rest to protect data exposure because of lost or stolen hard disk drives or because of inappropriately decommissioned computers. The term data at rest refers to an inactive data that is stored physically in any digital form.

Each LV is encrypted with a unique key. The logical volume data is encrypted before the data is written to the physical volume. This data is decrypted when it is read from the physical volume. By default, data encryption is not enabled in logical volumes. You must enable the data encryption option at the volume group level before you enable the data encryption option at the logical volume level.

The hdcryptmgr command manages the encryption keys, data encryption, and data decryption of the logical volume.”

An Option for Decoding and Summarizing AIX I/O Error Messages

Here’s an interesting tool for your bag of tricks:

NAME
summ
PURPOSE
Diagnostic tool for decoding and summarizing AIX I/O error messages
SYNTAX
summ [Flags] [Filename]
FLAGS
-e Include FC driver error numbers for each error.
-p Paginate the output.
-r Reverse the order of output.
-s Include sequence numbers in each line’s header.
-c I/O retry cmd_history failure time and reason
NOTE: summ –help displays the flag options.

DESCRIPTION
The summ command is an AIX only diagnostic tool used to decode Fibre Channel and SCSI disk AIX error report entries. It is an invaluable tool that can aid in diagnosing storage array or SAN fabric-related problems providing the source of the error.

The script generates single-line error messages enhancing the readability of the AIX error report. The tool is used by IBM Support worldwide, and is considered safe to run in a production environment.

Timeout Issues When Querying VIOS Resources From the HMC

IBM Support has an explanation:

Question: When trying to query virtual resources configuration on a managed system from HMC, it may happen that it takes very long time before completing or it fails with a timeout error message.

Cause: Any time a query is performed from HMC, a call is made to all VIOS on the managed system to get details on the configuration. On VIOS, the vio_daemon will proceed with this request by sending a query to the CMDB and respond to HMC.

There are different possible issue which could lead in timeout or at least long delay for this query, and the most common error message seen is:

-> The system is currently too busy to complete the specified request. Please retry the operation at a later time. If the operation continues to fail, check the error log to see if the filesystem is full.

Answer: The error above let us think that the VIOS is currently suffering some performance issue. Indeed the VIOS has to manage all the resource shared to client lpar (including disk access/IO, network communication…), but it also has to deal with all resource management request from all connected HMC (and in some case Novalink, PowerVC or other management product).

To reduce the risk of experiencing timeout issues, use the part command to monitor VIOS resources (CPU/memory).

Time Drift on POWER8 and POWER9 Servers

Also from IBM Support:

“Some clients noticed that the Power server time drifts seconds per day when compared to other systems, wall clock, or an NTP reference. They might observe it over a period of days, weeks, or months. The client can be asking questions.

Why is the server behaving differently?
Is there something wrong with my server?
Why must I use NTP when I never had to before?
Why is IBM not told me they changed the TOD accuracy of the server?

It does not indicate hardware needing replacement. The immediate suggestion is that clients use Simple Network Time Protocol (SNTP)/NTP as the power system Time of Day (TOD) can be expected to drift seconds per day when NTP synchronization is not used.

The only sure method to eliminate TOD drift is to deploy NTP and it is the IBM-recommended method to synchronize partition and system time and date for several generations of Power servers. Configuring and deploying NTP is outside the scope of the document but is described by OS documentation in IBM Knowledge Center. The reference section has pointers to some of the documentation.”

For more on time drift, see my old post from 2009.

E1080 at a Glance, AIX 7.3 Released, a Handy Tool for Job Hunters, NIM Install Troubleshooting

• Nigel Griffiths posted a pair of images that summarize key features of the IBM Power Systems server model E1080. Download them from IBM Support.
• We had the open beta, now we have the real thing, AIX 7.3 has been released. Keep in mind the requirements: an IBM POWER8, POWER9, POWER10, or later, technology-based server. Also note that POWER8 Nutanix (CS821 and CS822) does not support AIX 7.3.
• According to his Twitter bio, Ron Gould is a systems and network administrator. He’s come up with an interesting tool to help other techies update and customize their resumes. Download the relevant files here.
• If you’ve ever had trouble installing NIM, bookmark this page from IBM Support.

Note: “This document is intended as a reference guide for troubleshooting common NIM LED hangs. It is not intended as a fail-safe resolution guide, however [these] steps represent the most likely causes and resolutions to various NIM hangs.”

---