IBM Brings Server Design Changes to OpenSSH

Rob McNelly explains what he found helpful about the changes and goes on share IBM Support tips and details about his latest hiking adventure

The latest version of OpenSSH is upon us. IBM Support has a summary of the design changes:

“The new sshd design divides the sshd server into two binaries: The listener daemon ‘sshd’ and the ‘sshd-session’ daemon. The former runs on standby, while the latter gets invoked on demand. Also the -D flag is not critical to the sshd server functionality.”

Personally, I find it helpful that the following can now be determined from the process table:

– Which user is using which terminal.

– Which sshd daemon is parent to which ssh sessions.

In addition, when starting OpenSSH 9.9p1 (VRMF 9.9.3015.1000), the sshd daemon shows up in the process table as follows:

sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

There’s more, including screenshots.

Dealing with Volume Group Failures and Errors

This IBM Support doc examines the causes of varyonvg failure and file system mount errors:

“When attempting to vary on a Volume Group (VG), the operation may fail with error 0516-013 varyonvg: The volume group cannot be varied on because there are no good copies of the descriptor area.

“Additionally, you might encounter I/O ERROR DETECTED BY LVM in the error report (errpt) for the Physical Volume (PV) associated with the same VG.

“The error might appear with the label SC_DISK_ERR10 or SC_DISK_ERR4 or SC_DISK_ERR2 or SC_DISK_ERR9, but always check the sense code associated with the error and pay attention to sense code 0118 as it indicates disk reservation conflict

“If no disk errors with sense code 0118 are present but issues persist, please open a case with IBM Support for further assistance.

“Cause: A SCSI reservation conflict on a disk may result in a ‘DISK Busy’ state, causing read or write operations to fail when attempting to varyon a Volume Group.

“Diagnosing the problem: Use lsattr command to verify the current reserve policy.

lsattr -H -El hdisk# -a reserve_policy

“Note: if the disk is shared across multiple hosts or LPARs, verify the disk attribute reserve_policy on all systems and confirm it is set to no_reserve.

“If the reserve_policy is set to no_reserve, use devrsrv command to identify the stale reservation.

devrsrv -c query -l hdisk#”

At the end there’s a link to a related document on recurring errors associated with MPIO hdisk devices:

“Error B0EE9AF5 indicate a reservation conflict on a path to an MPIO hdisk device.

“Reservation conflicts typically occur in environments where multiple initiators (e.g., AIX hosts, Fibre Channel adapter ports, or processes) attempt to access or open a shared disk concurrently.

“If an initiator places a reservation on a disk, it restricts access to other initiators, resulting in operational errors. This scenario is common in clustered environments like HACMP or GPFS, where shared disk access is routine.”

Important Info About AIX Toolbox

Via Chris Gibson, an important update regarding IBM AIX Toolbox for open source software:

“To ensure integrity and authenticity of the AIX Toolbox rpm packages, AIX Toolbox team has started signing these packages. These packages are signed and published by default using OpenPGP key using gunpg2. Existing four recent versions of the different rpm packages present in AIX toolbox ftp site are signed and republished. DNF repository data is also update with signed rpm packages. Please read through the below instructions to see if you need to take some actions.”

Note: IBM registration is required to view the entire post.

An Update on AIX MustGather

In 2022 IBM Support posted a complete guide to MustGather and live partition mobility (LPM) data collection. Recently, Support came out with an AIX-specific doc on the requirements for collecting performance data when opening a case with the AIX System Performance team:

“Abstract: This MustGather document is meant to assist AIX Administrators with collecting AIX performance data needed when opening a support case with AIX System Performance team.

AIX System Performance Support requires the person opening the case to have some insight into the reported performance issue.

“Gathering information before calling IBM support might shorten the time it takes to resolve performance issues. The AIX System Performance Analysis team requires at least two pieces of information when diagnosing a AIX System Performance issue:

“Problem description: The perfpmr data collection

1. A detailed problem description of the performance issue is required….

2. Collect performance data by using the perfpmr.sh script.”

Read the whole thing for details, including links to scripts. And for further context, consult my 2022 piece on lights-out data center issues.

Return Trip

On a personal note, I just completed my latest Grand Canyon adventure. Three years ago I completed a  hike from the North Rim to the South Rim. I wanted to take the same rim to rim route this year, but a water pipeline construction project that was supposed to be completed by now is taking longer than planned. So I switched my route, this time going from the South Rim to the North Rim. (Here’s a good guide that explains what to expect.)

Rim-to-rim hikes are challenging, not just physically but logistically. The North Rim is only open to vehicles from May 15 to October 15 due to the fact that during the winter, there is often significant snow and the primary road goes unplowed. You can hike there during the offseason, but then you also have to hike yourself back out.

Of course, the heat is a significant issue during the spring and summer months. To deal with the conditions, The National Park Service advises hikers to bring ample amounts of water and salty snacks, take frequent breaks in shade and, most important, avoid the hottest hours of the midday (10 a.m. to 4 p.m.). How hot? Heading down into the canyon from the elevations at the rim, temperatures commonly exceed 100 degrees. It hit 102 in Phantom Ranch the day I hiked there, and temps up to 115-120 aren’t uncommon.

I’ll add a tip of my own: Take dips in the creek at regular intervals. Soak your shirt and hat. As that water evaporates, it really cools you off.

For all the planning and effort, I love the end of the hike. When you look back out over the canyon and see how far you came, it feels like you really accomplished something. This view is from the North Rim toward the South Rim. The two trails of lights are hikers and their headlamps. The left trail is called the South Kaibab, the right is the Bright Angel.

---