What to Do if Your VIOS Backup Fails
Rob McNelly aggregates updates and alerts from IBM Support
We have another nice collection of IBM Support links this month, starting with this one on VIOS backups. The document covers the relevant issues and how to resolve them:
“Summary: A VIOS backup fails — either through an HMC-scheduled backup job or a manual ‘viosbr -backup’ command.
“Root Cause: The viosbr backup process generates an XML file describing the VIOS configuration, pulling values from system configuration data (ODM) and device attributes. If any value included in that file contains an invalid or non-printable character — most commonly a stray control character (e.g. a backspace) left over from a configuration value that wasn’t fully or cleanly updated in the past — the resulting XML file becomes invalid, and viosbr cannot complete the backup.
“A typical example: a system attribute such as the hostname was changed at some point, and a stray, invisible character got saved into that value along with the intended text. The value may look correct or nearly correct when viewed casually, but the embedded character is enough to break the backup file.”
Recent APARs
This could affect anyone who uses an NVMe disk for alt disk:
“IJ59424: UPDATEIOS WITH “-ALTDISK” FAILS FOR NVME DISK
APAR status
Closed as program error.
Error description
updateios with '-altdisk' fails for NVME disk with error:
$ updateios -dev /41210 -install -accept -altdisk hdisk0
hdisk0 is either not a valid disk or it is already in use
Local fix
Use SAS disk or SAN disk
Problem summary
updateios with '-altdisk' fails for NVME disk with error:
$ updateios -dev /41210 -install -accept -altdisk hdisk0
hdisk0 is either not a valid disk or it is already in use
Problem conclusion
In updateios script, use 'lspv -unused' instead of 'lspv -free' which will work for NVME disks”
updateios and altdisk problem scenarios
This is problem scenario you could see with updateios and altdisk:
IJ59448: UPDATEIOS WITH “-ALTDISK” FAILS WITH BOSBOOT ERROR
APAR status
Closed as program error.
Error description
$ updateios -dev /up_4121 -install -accept -altdisk
hdisk0
Calling mkszfile to create new /image.data file.
Checking disk sizes.
Creating cloned rootvg volume group and associated
Creating logical volume alt_hd5.
Creating logical volume alt_hd6.
<=======snip=====>
+--------------------------------------------------------
+
BUILDDATE Verification ...
+--------------------------------------------------------
+
Verifying build dates...done
FILESET STATISTICS
406 Selected to be installed, of which:
377 Passed pre-installation verification
16 FAILED pre-installation verification
13 Already installed (directly or via superseding
filesets)
377 Total to be installed
Filesystem size changed to 10485760
0503-409 installp: bosboot verification starting...
0503-497 installp:
An error occurred during bosboot verification processing.
Local fix
Problem summary
On VIOS 4.1.2.00 when trying to do updateios with -altdisk, users will see failure during bosboot verification on alternate disk.
Problem conclusion
bosboot verification is skipped as it is not required for alternate disk updates.
AIX and VIOS Service Packs
IBM Support recently updated support lifecycle information for AIX. This document lists dates of service pack releases, including the latest version and the expected schedule for future releases.
Here’s the same information for VIOS. You can also consult this doc to see if your version of VIOS is still being supported.
Vulnerabilities in IBM AIX and PowerVM VIOS
Updated on Aug. 21: “Updated the summary to highlight the additional installation instructions. IBM is providing security updates for supported AIX and VIOS releases that are under active fix support. Delivered through Service Packs (SPs) and Fix Packs (FPs), these updates remediate vulnerabilities affecting operating system, virtualization, and third-party components as described in the vulnerability details section. To simplify deployment and maintenance, the security fixes have been incorporated into cumulative maintenance packages. IBM strongly recommends that customers remain on supported releases and promptly apply all applicable security updates and fixes. Please review the Remediation/Fixes and Release Notes sections before deployment. Special steps are required while using NIM to apply the updates. Additional steps are also required after applying VIOS 4.1.0/4.1.1 updates.”
On ‘Orderly’ System Updates
This doc provides recommendations on the order in which VIOS, system firmware and AIX should be updated. Turns out it does matter where you start. IBM cites this recent AIX and VIOS bulletin, along with Power Systems firmware bulletins.
Again, from the document, here are the steps involved:
- Install August Firmware SP.
- Install August VIOS FP on VIOS1 and Reboot VIOS 1 (assuming dual VIOS).
- All the clients are running on redundant VIOS 2.
- Install August VIOS FP on VIOS2 and Reboot VIOS 2 (assuming dual VIOS).
- Reboot VIOS 2.
- All clients are now running on VIOS 1.
- Install August AIX SP.
- Reboot AIX client (we want to mention boot the ones with the largest CPU/Mem footprint and go down to the smallest. Otherwise, we might see a perf issue and would have to run DPO).
- Reboot the Power CEC/Frame.
- If reboot of the Power CEC/Frame is done and all LPARS are going down at the same time after SP/FP install, we would just recommend boot the VIOSes and then start with largest CPU/Mem LPAR.
Be sure to read the whole thing.
NIM Security Enhancements
Learn more about NIM security enhancements:
“The NIM Security enhancements for NIM secure (nimsh) communication applies to:
● AIX 7.3 TL4 SP2 / VIOS 4.1.1.30
● AIX 7.3 TL3 SP3 / VIOS 4.1.2.20
● AIX 7.3 TL2 SP5 / VIOS 4.1.0.40
● AIX 7.2 TL5 SP13
“This section describes the following topics:
1 What’s new
2 Upgrading the NIM master and NIM clients
3 Verifying NIM secure communication
4 Troubleshooting
5 High availability considerations
6 nimadm Considerations
7 Recommended upgrade methods
9 Frequently asked questions”
Related: A NIM Q&A:
“What if I update my NIM master before updating all NIM clients?
Any considerations when updating the NIM master?
Is non-SSL encrypted communication impacted by the NIM changes?
How can I boot in to Maintenance Mode in case of boot failures during the transition to the new Service Pack?
What happens to a NIM client after it is updated?
How to update NIM clients via NIM in NIMSH(secure) mode?
Restoring mksysb backups?
Can older clients still work with secure NIM after it is updated to the 2633 build?
Does update order matter?
Other considerations?
Managing SSPs
For those using shared storage pools, this describes how to move existing logical units (LUs) to a new SSP or the same SSP cluster in VIOS environments.
Grand Canyon Flooding
Most every year around this time, I start thinking about taking another rim-to-rim hike in the Grand Canyon. Not this year.
A significant flash-flood event occurred in Bright Angel Canyon and the Phantom Ranch area on August 29. As of the morning of August 30, 62 people were evacuated from Phantom Ranch and the lower North Kaibab Trail. The flash flood resulted in significant impacts to infrastructure throughout Bright Angel Canyon. Nearly all footbridges spanning Bright Angel Creek were destroyed, eliminating hiker access across the creek. As a result of major damage to the water pipeline, area hotels have been shut down due to a lack of water. Grand Canyon National Park staff continue to assess damage to trails, bridges, utilities, facilities and other infrastructure.
The Colorado River is currently closed to river traffic until further notice due to metal structures and debris entering the river. River parties already in the canyon are receiving direction through river operations rangers and the Grand Canyon River Alert system. Visitors with upcoming river trips should contact their commercial outfitter or NPS River Permits for trip-specific information.
In short, the canyon won’t be accessible anytime soon. This episode serves as another reminder to me: There are no guarantees in this world. We can only seize the opportunities before us. On that note, I am in the midst of preparing for my annual triathalon, which takes place in a few weeks. I’ll let you know how that goes.