HMC, BMC, VMI, eBMC, FSP, Oh My!
In the beginning of the HMC era life was simple. We had an x86 HMC connected to the FSP (Flexible Service Processor) of the server. You could have redundant HMCs if you wanted, as well. With the migration to the Power HMCs until Power10 we have a BMC (baseboard management controller) in the HMC, and we still connect to the FSP on the server. With Power10, we added the VMI (virtualization management interface) and the eBMC (enterprise BMC). This article will clear up some of the confusion around these.
HMC Basics
HMC (Hardware Management Console)
The HMC is an appliance that is used to manage partitions locally or remotely. The HMC can be an x86 HMC (7042), a Power based HMC (7063-CR1 or CR2) or a virtual HMC. Servers can run without an HMC but you need an HMC to do remote power on/off, LPM (live partition mobility), get a remote console and many other actions. The HMC has one connection on the public or administrative network and a second private connection that is used to talk to the FSPs or eBMCs for the attached servers. This private connection can be direct connect if there is only one server or via a private switch or VLAN. The private network uses DHCP, so it is important that it be a private network that is non-routable. If you are still running x86 based HMCs (7042-cr8 and cr9) they cannot run HMC code above v9r2m953 which goes out of support on April 30, 2023, so it is time to transition to the Power HMC as the Power10 servers require version 10 of the HMC code. If you have POWER7 servers around then move everything else to a Power HMC and keep the POWER7 on the x86 HMC so it does not stop you updating the Power HMC to v10.
BMC (Baseboard Management Controller)
The BMC is new with the Power HMCs. It allows you to remotely control the HMC, including monitoring during reboots and upgrades. It also allows you to attach iso images as virtual media and they can then be used to do updates. The BMC also provides a console to the HMC. For the Power HMC, we now put the public interface on eth0 and the private on any of eth1, eth2 or eth3. This is a change from x86, but it allows the BMC to share a port with the public connection and saves on cables and switch ports.
Both the public HMC interface and the BMC require an IP address each on the same subnet. The BMC can be accessed using ssh or https to the BMC IP address. The BMC is also maintained separately to the HMC code. The updates can be downloaded from Fix Central and are found under the Power firmware all the way down the bottom of the dropdown. BMC updates for the 7063-CR1 are different to the ones for the 7063-CR2 so make sure you download the correct ones. These updates are applied through the BMC https interface.
The first time you login to the BMC the username is root, and the password is 0penBmc (zero at the front not O). It will force you to change the password. I highly recommend adding a second admin account as well as the bmccred one. When configuring the BMC from the HMC, you set it to shared mode on eth0 and provide it with a static IP. On the 7063-CR2, you can’t do this until you are at v10 of the HMC code. It provides the option and lets you edit it but does not make the change.
BMC Inband Credentials
The HMC needs to be able to communicate with the BMC to self-monitor for problem reporting, poll for platform events and several other tasks. In order to do this, you need to set up the inband interface so the HMC can talk to the BMC —this requires the inband credentials. When setting up the HMC you will need to create an account (I do it on both the HMC and the BMC) for the inband credentials. I normally call in bmccred.
FSP (Flexible Service Processor)
The FSP is the point for pre-Power 10 servers (usually labelled HMC1 or HMC2 on the server) to which the HMC will connect to on the private network. The HMC uses DHCP on this network to provide an IP to the FSP from a range you select and uses it to manage the servers. To add to the confusion, the documentation often refers to HMC1 and HMC2 as eth0 and eth1. The key is that these are the HMC ports on the servers. With Power10 we no longer have an FSP; there is an eBMC instead.
ASMI (Advanced System Management Interface)
The ASMI is used to connect to the service processor and is normally accessed through the HMC https interface. It is used to perform general administrator tasks such as reading service processor logs, reading vital product data, taking FSP dumps and controlling the system Power. For Power10 you can connect to it using the eBMC IP. For Power10, the default username and password are admin/admin. It is important to write down the passwords for these and test them regularly so that you know what they are if you need to work with IBM on problems.
VMI (Virtualization Management Interface)
The VMI is new for Power10 and is used for virtualization management for the Power10 servers. The VMI resides in the Power hypervisor on the Power10 server. It is assigned an IP on the private network by the HMC. This is the IP the HMC uses to communicate with the hypervisor for partition management and consoles. Users do not interact directly with this IP, and other than having two IPs on the service network instead of one, there is no difference in terms of managing the system. All traffic between the HMC and VMI is encrypted with TLS using a system unique certificate. The VMI supports both DHCP and static IP configurations. After the server is connected to the HMC and configured by setting the access passwords you should configure the VMI interface. The VMI shares the same physical connection as the eBMC going back to the HMC so there is no additional cabling needed.
The default VMI connection setting (not the eBMC connection) for a new Power scale-out or midrange server is static. Since it is on the private network it should be changed to use DHCP. This should be done before you Power on the server. When configuring the VMI it uses the terms eth0 and eth1—this refers to the HMC1 and HMC2 ports on the eBMC, not the ports on the HMC. The VMI is configured on the HMC under actions>view all actions>operations>VMI configuration.
You can list the interfaces for the VMI using:
lssysconn -r vmi -m <managed system>
Note: The VMI should be configured before powering on the system or the system will go into a “no connection” state when it reaches standby mode. The VMI IP address is not active until the server is powered on (standby or operating).
In his article “What’s new in HMC 10.1.1020.0,” Hariganesh Muralidharan provides some great diagrams on the connectivity. Additionally, Shawn Bodily posted a blog on his experiences with installing a Power10 server and what he learned about the HMC, eBMC, VMI, etc. There is also a great article by Raghavendra Malapati, “Managing Power10 scale-out and midrange systems using HMC.”
eBMC (Enterprise Baseboard Management Controller)
This is the replacement for the previous FSP and is used in the Power10 servers. eBMC Power servers introduce a new connection type between the HMC and the managed server which means they require two IP addresses per BMC ethernet port. The new interface is called the VMI which we described above.
As with FSP based systems, there are two ethernet ports that can be used to connect to redundant HMCs—these are referred to as HMC1 and HMC2 or eth0 and eth1. Each HMC is cabled to one eBMC port and redundant HMCs will use a total of four IP addresses provided by the HMC. The default setting for the eBMC IP address is DHCP client; however, as mentioned above, the default setting for the VMI IP address is currently set to static and should be changed and configured prior to powering on the system. The eBMC IP address is active when the server has completed AC power apply initialization (power off state).
Networking
As mentioned above there are now at least four IP addresses involved in setting up your HMC with a Power10 server. These look something like this:
ETH0 on HMC 192.168.2.30 IP for HMC
192.168.2.31 IP for BMC
ETH1, 2 or 3 DHCP range IP range used to serve an IP to each server for its FSP or eBMC
Same DHCP range IP range used to server an IP to the VMI on each Power10 server
If you have redundant HMCs then there will be a second set of IP addresses for that HMC.
HMC and BMC Maintenance
HMC
HMC v10 is required to manage Power10 systems. However, it only runs on the 7063-CR1, 7063-CR2 and virtual HMC appliance models. It is not supported on the x86 7042 models, and it only supports servers that are POWER8 and above. Version 10 provides a significant number of updates—there are links to these in the references below. The latest version as of 3/9/2023 is v10r2m1030 plus patch MF70632. The minimum required for all Power10 servers except the E1080 is v10r2m1020. The E1080 requires v10r1m1010 as a minimum. I highly recommend going to v10r2m1030 plus MF70632. V10r2m1030 can be updated to directly from v9r2m950 plus MH01942 or V10r1m1010 plus MF70431. This is covered in the readme and may get updated so always redownload and check the readme before doing any patching.
Be aware that the GUI changes at v10r2m1030—you can revert to the old dashboard for that login only by clicking on the login name above and selecting the old dashboard. This is only for this login. After a while the new dashboard becomes intuitive, but I wanted to warn you as it is significantly different.
When planning HMC updates you should read the readme/description section provided on Fix Central to make sure that the firmware you are running and the servers will coexist happily. You can also go to the FLRT tables (see references) to see a cross-reference map of support levels. If you are updating though several levels of the HMC code, then you may have to do a multistep update. There may also be mandatory patches at various steps. As an example, when going from v9r1m941 to v9r2m953 there are multiple steps:
bkprofdata
saveupgdata -r disk
bkconsdata
V9R1M942
Use updhmc as it is an update
Reboot HMC
MH01898
Use updhmc
Reboot HMC
saveupgdata -r disk
MH01857 – this must be the last fix prior to the v9r2m950 update
Use updhmc
saveupgdata -r disk
If you don’t do the above saveupgdata your getupgfiles will fail
V2r9m950 (Make sure to download the network files specific to the CR9)
Use getupgfiles
chhmc -c altdiskboot -s enable --mode upgrade
Reboot HMC
saveupgdata -r disk
MH01900
Use updhmc
Reboot HMC
V9r2m953
Use updhmc as it is an update
saveupgdata -r disk
MH01943
Use updhmc
Reboot HMC
Take another backup
bkprofdata
saveupgdata -r disk
bkconsdata
The above is just an example—you should check on Fix Central for the latest patches and check the readmes to find the correct process for the updates.
I have learned the hard way that you should take a “saveupgdata -r disk” at every step to avoid problems. Also make sure you back up the profiles and then the HMC itself prior to starting. I do this to my NIM server as follows:
bkprofdata -m Server-8286-41A-XXXXX -f backup-828641a-feb022023 saveupgdata -r disk saveupgdata -r disksftp -h nimserver -u jlynch -d /backups/hmcsave bkconsdata -r sftp -h nimserver -u jlynch -d /backups/hmcbkup
You should also plan to update your BMC at the same time so that it does not get back level.
BMC
Maintaining the BMC is fairly straightforward and frequently forgotten. You download the updates from Fix Central to your desktop and then go to the BMC through the web interface and perform the maintenance there. On a 7063-CR1 you will see it refer to BMC and PNOR updates. In the download you will have both a .bin (BMC) and a .pnor file. For the 7063-CR2 you will receive a couple of tar files in the download. During maintenance you upload these files from your desktop to the HMC as part of the process and then do the update. The BMC is done first then the PNOR on the 7063-CR1. On the 7063-CR2 you will do the BMC, which is the tar file that ends mtd.tar, first, and then the PNOR, which is the tar file that has pnor in its name. For the latest levels the files are:
7063-CR2
PNOR mowgli-IBM-OP9_v2.5_4.125_prod.pnor.squashfs.tar
BMC obmc-mowgli-op940.hmc-24.ubi.mtd.tar
7063-CR1
PNOR P8DTU20200610_IBM_7063CR1_sign.pnor
BMC SMT_P8_326.bin
Make sure to read the readme file to ensure there are no interim steps if you are very back level. You should also check that the time on the BMC and the HMC are the same. This requires a reboot of the HMC to correct. This should be checked before and after any BMC and/or HMC updates.
The latest level for the 7063-CR2 is OP940.50 which shows on the BMC as:
Firmware version IBM-mowgli-ibm-OP9_v2.5_4.125-prod Firmware Version op940.hmc-24-0-g44e2acf8e6 lshmc -b will show it as: "bios=mowgli-ibm-OP9_v2.5_4.125-prod”
For a 7063-CR1 you will see the following on the web interface to the BMC:
Firmware Revision : 03.26 Firmware Build Time : 20210816 PNOR Build Time : 20200610 CPLD Version : B2.81.01
The above is BMC and PNOR 3.26/3.11, which is the latest.
lshmc -b shows it as:
"bios=open-power-IBM-P8DTU-7063CR1-20200610”
Useful Commands
lshmc -n
Provides network information for all the LPARs.
lshmc -v
Provides version information but also provides information on the model HMC and its serial number.
lssysconn -r all
Provides a list of all connected servers, and information about them. Note how the Power10 version also includes the VMI information.
lssysconn -r all resource_type=sys,type_model_serial_num=9009-22G*XXXXXXX,sp_type=fsp,sp=primary,sp_phys_loc=U78D3.001.WZS0JVB-P1-C1,ipaddr=172.16.222.41,alt_ipaddr=unavailable,state=Connected resource_type=sys,type_model_serial_num=9105-41B*XXXXXXX,sp_type=ebmc,ipaddr=172.16.222.45,user_name=admin,alt_ipaddr=unavailable,state=Connected,vmi_ipaddr=172.16.222.44,vmi_state=Connected
To check the vmi configuration:
lssysconn -r vmi -m Server-9105-41B-XXXX resource_type=vmi,type_model_serial_num=9105-41B*XXXXXXX,interface=eth1,ipv4addr=0.0.0.0,gateway=0.0.0.0,net_mask=0.0.0.0,ipv4addr_orig=static resource_type=vmi,type_model_serial_num=9105-41B*XXXXXXX,interface=eth0,ipv4addr=172.16.222.44,gateway=0.0.0.0,net_mask=255.255.0.0,ipv4addr_orig=dhcp
To check for configuration errors:
lssyscfg -r sys -F name,state,state_detail
This will provide a list of servers and their status.
lshmc -V
From a 7063-CR1
lshmc -V "version= Version: 10 Release: 2 Service Pack: 1030 HMC Build level 2211152246 ","base_version=V10R2"
lshmc -b
From a 7063-CR1
lshmc -b "bios=open-power-IBM-P8DTU-7063CR1-20200610”
From a 7063-CR2
lshmc -b "bios=mowgli-ibm-OP9_v2.5_4.125-prod"
General Update Advice
It is important to keep your HMC and BMC up to date to avoid known problems, support issues and security issues. According to the FLRT site, all V10 levels of the HMC code prior to V10R1M1020 went out of support by 10/24/2022. V9R2M953 is the highest level supported on the 7042 x86 HMCs and it goes out of support on April 30, 2023. In order to remain supported, it is critical that you update your HMCs and BMCs to the latest level. If you still have old POWER7 servers around and cannot get rid of them then move them to a 7042-CR8 or CR9 HMC and put all your POWER8 and above servers on the 7063 Power HMCs. That will allow you to update those Power HMCs and BMCs to the latest supported and patched levels.
You should also note that IBM still allows anonymous access to download HMC code; however, you now have to use SSL based ftp (ftp -s on AIX). This may require you to open some additional ports on your firewall. ftp -s uses port 21 to open the session, but requires you switch to passive mode. This will require ports 65024 – 65535 to be open.
IBM is constantly updating and patching the HMC. On March 10, 2023, they released HMC V10R2M1031 as a service pack that can be updated to from V10R2M1030.
There was no change to the BMC and PNOR levels for either 7063 HMC.
I hope this helps explain some of the new technology, and happy updating!
References
How to configure a new 7063-CR2 HMC
Connecting a Power10 eBMC system to an HMC:
https://www.ibm.com/support/pages/node/6833430
https://www.ibm.com/support/pages/connecting-power-10-ebmc-system-hmc
Add user to the BMC on the 7063-CR2 HMC
IBM Power HMC 7063-CR2 Console Inband Communications Credentials
Shawn Bodily: Blog on Power10 Install
Raghavendra Malapati: Managing Power10 Scaleout and midrange systems using the HMC
Hariganesh Muralidharan Blogs:
HMC V10R1M1010
HMC V10R2M1030
Redbooks and Power10:
IBM Power S1014, S1022s, S1022, and S1024 Technical Overview and Introduction
IBM Power E1050: Technical Overview and Introduction
IBM Power E1080 Technical Overview and Introduction
Power10 Performance Quick Start Guides
Power10 Performance Best Practices
HMC Levels supported according to FLRT
Jaqui Lynch article on AIX 7.3 and HMC v10 Updates (Jan 2022)
Location of FTP files for HMC Updates
ftp -s public.dhe.ibm.com
Network files for getupgfiles for v1030
cd software/server/hmc/network/v1030/ppc
Patches such as MF70632_ppc.iso
cd software/server/hmc/fixes/
Updates such as V10R2M1031
cd software/server/hmc/updates