Interesting Points About the Power HMC
Jaqui Lynch on a real-life experience upgrading a 7063-cr1 HMC from 931 to 951, and new HMC upgrades
In the past few months I have been migrating many clients to the new Power HMC (7063-cr1 and 7063-cr2). During that time I have found some great reasons to migrate from the Intel HMCs, apart from the fact that they are end of life and can’t run the new levels of the HMC code. In this article I want to share with you a recent experience upgrading a 7063-cr1 HMC from 931 to 951 and how the new HMC allowed me to do things to recover that the old Intel ones do not.
Normal Upgrade
The HMC was a 7063-cr1 at v9.1m931 with MH01826 applied. BMC and PNOR were at 3.08/3.07. The plan was to update the HMC to v9.1m951 plus MH01893 and the BMC and PNOR to 3.26/3.11.
Upgrade steps were as follows:
bkprofdata for each server connected to the HMC
saveupgdata -r disk
Then I took a remote saveupgdata to my nim server to the /software/drhmcsave directory:
saveupgdata -r disksftp -h nimserver -u jlynch -d /software/drhmcsave
Since someone was temporarily onsite we took a backup to USB then removed the USB stick:
bkconsdata -r usb
Finally I took a backup to the NIM server as well:
bkconsdata -r ftp -h nimserver -u jlynch -d /software/drhmcbkup
Then we started the normal upgrade steps:
On my NIM server I had preloaded the update files in /software/hmcv9. I do this in case the IBM website is down when I am ready to do this and also because it is faster. Also, many clients do not allow their systems to have outside access. Files in /software/hmcv9 are:
HMC_Update_V9R2M951_ppc.iso MH01893_ppc.iso MH01858_ppc.iso hmc950-network – this is the 950 network directory that contains the network install images
First steps were to check memory and clear out disk space:
monhmc -r mem -n 0
KiB Mem : 33399424 total, 21657920 free, 9087296 used, 2654208 buff/cache
This looks fine so now we check the disk:
monhmc -r disk -n 0 Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda2 16374696 9213504 6322744 60% / devtmpfs 15808384 0 15808384 0% /dev tmpfs 16699712 0 16699712 0% /dev/shm tmpfs 16699712 18624 16681088 1% /run tmpfs 16699712 0 16699712 0% /sys/fs/cgroup /dev/sda3 6061632 494072 5252988 9% /var /dev/mapper/HMCDataVG-HomeLV 10190100 111672 9537756 2% /home /dev/mapper/HMCDataVG-LogLV 20511312 6243080 13203272 33% /var/hsc/log /dev/mapper/HMCDataVG-ExtraLV 20511312 8596712 10849640 45% /extra /dev/mapper/HMCDataVG-DumpLV 361110668 74264 342669940 1% /dump /dev/mapper/HMCDataVG-DataLV 412716456 290916 391437636 1% /data
This looks fine but we clear the tmp files anyway:
chhmcfs -o f -d 0
I also check for any old pedbg files as these can fill up filesystems and cause the getupgfiles to fail:
ls -al /dump
If you see any log files in /dump then you need to login as hscpe and remove them as follows:
pedbg -r
At this point I started the upgrades.
Because the HMC is very backlevel this is a multistep upgrade and it takes a long time as it reboots after every update:
- Put on the mandatory pre-ptf MH01858
updhmc -t sftp -h nimserver -u jlynch -f /software/hmcv9/MH01858_ppc.iso hmcshutdown -r -t now
- Now upgrade to 950
On one ssh session:
while true ; do ls -la /hmcdump ls -la /hmcboot monhmc -r disk -n 0 sleep 60
On a second ssh session:
getupgfiles -h nimserver -u jlynch -d /software/hmcv9/hmc950-network
Once that is done you should see no files in /hmcdump and /hmcboot
Now you tell the HMC to boot from the alternate disk:
chhmc -c altdiskboot -s enable --mode upgrade
And then to reboot:
hmcshutdown -r -t now
- Upgrade to 951
updhmc -t sftp -h nimserver -u jlynch -f /software/hmcv9/HMC_Update_V9R2M951_ppc.iso hmcshutdown -r now
- Put on patch MH01893
updhmc -t sftp -h nimserver -u jlynch -f /software/hmcv9/MH01893_ppc.iso
hmcshutdown -r now
When the upgrade is done the BMC and PNOR should be updated. But my upgrade failed at the 950 stage and the HMC became unbootable—it kept hanging at the petitboot menu. The system is in a data center where there are normally no people so this is where my learning curve increased rapidly as I was going to have to rebuild the HMC. I spent a lot of time on the phone with IBM logging into the BMC and checking disks and adapters, etc. Finally we determined that the disks were fine but somehow the boot images were corrupted and we needed to reinstall the HMC. This is where the BMC became very useful as no one was onsite.
Using the BMC to Recover Your HMC
If you use the web interface to connect to the BMC you will see a small graphical box that is the HMC console but it is unreadable as it is so tiny. If you click on that box it will launch a Java jnlp (this works better with Chrome) and you now have a full size console with lots of options across the top. The other option you have is to login to the BM over the web, click on configuration, console redirection and launch console. Across the top on the console you will see a number of options. One of those options is “virtual media” which allows you to attach an iso image to the HMC as if it was a DVD—similar to FBO (file backed optical) on the VIO server. I attached the 950 recovery ISO and proceeded to use the iso image to reinstall the HMC. This required intercepting the reboot at the petitboot prompt. This process is described in this document even though it is an HMC v10 document.
After the reboot all the systems and LPAR profiles came back after I reauthenticated the servers. This is a small user base HMC so I just recreated the users and call home but I could have used restupgdata and pointed at the files I had backed up to my NIM server earlier. I then upgraded to 951 and put on MH01893 without an issue.
The next piece of the adventure was updating the back-level BMC and PNOR. The BMC is the baseboard management controller and provides the management and monitoring functions for the boot process and the OPAL hypervisor. It fulfills many of the functions that the ASMI provided on the other Power Systems servers. The CEC PNOR provides the host boot and OPAL (hypervisor for Linux only servers) components. It’s upgraded after the BMC firmware. The OPAL hypervisor provides the hardware abstraction and run time services for the HMC software.
The first step is getting the BMC and PNOR firmware. To download the new firmware to your desktop, go to Fix Central. For product group select “Power”, then for product scroll all the way to the bottom and select “Power Hardware Management Console”, and then “7063-CR1” (or 7063-cr2 if that is your HMC) for the console and lastly “All” for the final selection option. You then click on continue and Fix Central will give you a list of firmware updates. Click on “description” for the one you want and this will give you the readme file—save this to a PDF and read it!
If you then click on the firmware level it will provide you with your download options. I always check the “include prerequisites” box and then finally continue. In order to download the firmware, you will need to provide the type (7063) and seven-digit serial number of the HMC. It now prompts for type and serial number and you enter those and click on continue. You then have to click on “agree” for the terms and conditions and it then provides a list of files that you can download.
The primary files you need to download to use the BMC GUI to update the firmware and CEC PNOR for the 3.26 BMC and 3.11 PNOR firmware level are:
P8DTU20200610_IBM_7063CR1_sign.pnor (32 MB)
and
SMT_P8_326.bin
In my case, I am using the BMC GUI to do the update so I only need the .bin and .pnor files. These get downloaded to your desktop and will be uploaded to the HMC during the update process.
Updating the Firmware
The default username and password for the BMC are ADMIN and ADMIN. If you change the password, make a note of it as you will need it in the future.
To get to the BMC GUI on a 7063-cr1 Power HMC you need to use either Google Chrome or Mozilla Firefox as a browser. There are two ways to get to the BMC.
- Login to the HMC on the web and look for the option to “Launch BMC.” Once it’s launched, you’ll login as ADMIN.
- Find out what IP was assigned to the BMC and use https:// to connect directly to the BMC. This is how I normally connect.
- It will show you the current firmware and some other information. Make a note of the firmware revision line and the PNOR build time.
They were at:
Firmware Revision : 03.08 IP address : xxxxxx Firmware Build Time : 20190522 BMC MAC address : xxxxxx PNOR Build Time : 20190513 CPLD Version : B2.81.01
Unless the readme says otherwise, we update the BMC and then the PNOR. Across the top there are multiple tabs and one of them is “Maintenance.” Click on “Maintenance”, click on “Firmware update” and then click on “Enter update mode.” You’ll be prompted again and you click on “Enter Update Mode” and then click on ok when it asks if you really want to do this. You’ll now get the “Firmware Upload” box and will need to click on browse to go find the firmware to be uploaded. This is where you will go and find the .bin file you downloaded earlier. Once you see it listed to the left of Browse then click on “Upload Firmware” to upload the firmware. Once the firmware is uploaded you will see a screen that shows what was uploaded and your current level. Leave the “Preserve Configuration” checked and the “Preserve SDR” unchecked. In order to upgrade the firmware you will need to click on “Start Upgrade.”
You’ll see the progress updated regularly and when the upgrade is completed you will see a message saying “Upgrade complete. Please wait for 1 minute.” Click on OK and you will see the HMC is now rebooting. After the reboot, you can log back in and you will now see the BMC firmware (Firmware Revision line) has been updated.
Now go back to the Maintenance tab and then select “PNOR Update.” Here you go through the PNOR upload process in the same way you did for the firmware upload. After the PNOR file is uploaded, it will show you the current and new levels and you will need to click on “Start Upgrade.”
Again, you’ll see the progress indicator and once the update is complete it will reboot again. After the reboot you will see:
System Firmware Revision : 03.26 IP address : xxxxxx Firmware Build Time : 20210816 BMC MAC address : xxxxxx PNOR Build Time : 20200610 CPLD Version : B2.81.01
At this point your HMC firmware and CEC PNOR are now up to date. As with all firmware updates, be sure to check the readme prior to performing the update to ensure that the HMC level and the firmware to be updated are a supported combination.
Problems With BMC and PNOR Update
It turns out that there is an issue with this update, especially since it’s such a major jump. It’s not in the readme but the best way to do this update is to shut the HMC down first (which I did not) otherwise there will be issues with the date on the HMC. After I updated the BMC and PNOR I noticed the date on the HMC was back in 1980 but it would not let me change it to anything more recent than 1984. The workaround is to shut the HMC down, fix the time in the BMC then bring the HMC up and you should be able to fix the date there. According to IBM support “Following the update of the BMC, the user must reconfigure the BMC date/time under the Configuration menu, before starting the HMC operating system.” However, this was not in the readme that I downloaded that morning.
If I was to do this again I would start by rebooting the HMC, then I would probably update the BMC and PNOR before I did the 950 upgrade (after the mandatory pre-patch). And I would power down the HMC while I was updating the BMC and PNOR.
Notes About HMC v10
IBM just recently announced HMC v10R1M1010. This level only runs on the 7063-cr1, 7063-cr2 or the vHMC on POWER8, POWER9 or Power10 servers. HMC v10 will not run on any Intel servers, only POWER servers. In order to upgrade to v10 your HMC must be at a minimum of either v9r1m910 plus MH01858 or v9r2m950. Finally HMC v10 only supports servers that are POWER8 or higher. POWER7 and POWER7+ are not supported.
Keep Software and Firmware up to Date
The new POWER HMC needs to be treated just like any other POWER server that you have in your data center. This means that you need to keep not just the HMC software up to date, but you also need to keep the firmware and CEC PNOR up to date. It’s a straightforward process but needs to be added to your regular maintenance window as an additional step in your HMC maintenance planning. The really cool thing is that this new HMC allows you to recover remotely when things go wrong. If I was on an Intel HMC and had to do the same recovery someone would have had to drive to the datacenter to put the USB key back in. Because I had access to the BMC I was able to do the recovery with no one needing to go onsite.
References
- Fix Central
- 7063-cr1 HMC announcement
- HMC BIOS Updates
- Power HMC Firmware updates for 7063-cr1
- Power HMC Firmware updates for 7063-cr2
- Power HMC 3.26/3.11 Firmware Readme for 7063-cr1 (changes when you select it in Fix Central)
- Power HMC OP940 Firmware Readme for 7063-cr2 (changes when you select it in Fix Central)
- HMC v10 Recovery Image Readme
- HMC v10 Announcement
- Upgrading from HMC v9 to HMC v10
Notes on the 7063-CR2
It’s a different technology to the 7063-cr1 so you may find the following links to be useful:
- Configuring a new 7063-cr2
- Configuring the BMC on a 7063-cr2
- Links to many articles on the 7063-cr2
- 7063-cr2 announcement