Configuring Hybrid Network Virtualization for AIX
An overview and description of Hybrid Network Virtualization—or HNV, as it is more commonly known—can be found in this blog written by Chuck Graham, the lead IBM architect for SR-IOV. It’s an excellent blog, which I highly recommend you read before attempting to configure HNV in your environment.
As described in the blog, HNV was introduced to allow administrators to use SR-IOV “…for optimal performance and mobility…” As we know, native SR-IOV adapter logical ports offer the best possible network performance, when compared with Virtual NIC and Virtual Ethernet Adapters (backed by VIOS SEA). So, if you need the absolute best network performance from an Ethernet adapter (that can still be shared with other LPARs), then native SR-IOV is the best choice for your configuration. From Chuck’s blog:
“PowerVM’s SR-IOV technology allows a partition to have direct access to a shared physical adapter to provide low overhead, high performance networking…”
Having said that, the downside to employing SR-IOV logical ports is that it cannot be used in environments that require Live Partition Mobility (LPM) or Simplified Remote Restart (SRR). As we know, in an LPM/SRR environment, all devices must be virtual. An SR-IOV logical port is not a true virtual adapter, it is considered a physical adapter, and, as such, will prevent a partition from being LPM capable. From Chuck’s blog:
“Previously, partition mobility operations were not allowed when a partition’s configuration included direct access to physical I/O adapters…”
So, to take advantage of the speed and efficiency of SR-IOV logical ports (which map to SR-IOV Virtual Functions or VFs), and allow an LPAR to participate in mobility operations, IBM introduced HNV. Again, from Chuck’s blog:
“One of the key advantages of the HNV approach is the integration of the dynamic remove and add of the Migratable logical port as part of active LPM, inactive LPM or SRR operations…”
The HNV capability is enabled by selecting a new Migratable option when you configure a new SR-IOV logical port.
The aim of this article is to provide a “how-to” guide for configuring HNV, with an AIX LPAR in mind. We’ll show you how to configure an existing LPAR so it can start using a HNV configuration. We’ll also describe what happens during a LPM operation.
First, a little about the environment. In our lab we have two POWER9, S924 systems, configured with the following:
System Type/Model: S924/9009-42G
System firmware: FW950.50 (VL950_105)
HMC: V10R1 M1021
AIX: 7300-01-01-2246
VIOS: 3.1.3.21
SR-IOV adapter: EN0H. PCIe3 4-port (2×10 Gb FCoE, 2×1 GbE) SFP+ adapter (FC EL38, FC EL56, FC EN0H, and FC EN0J; CCIN 2B93)
In this example, we will reconfigure an existing AIX LPAR to use HNV. The LPAR is currently configured with one VEA, ent0. The VEA is configured on VLAN 691, with an IP address of 10.1.1.123:
root@aix73-sriov # entstat -d ent0 | grep -i vlan
Invalid VLAN ID Packets: 0
Port VLAN ID: 691
VLAN Tag IDs: None
root@aix73-sriov # ifconfig -a
en0: flags=1e084863,81cc0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 10.1.1.123 netmask 0xffffff00 broadcast 10.1.1.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
...
At a high level, we will perform the following steps to configure the LPAR for HNV. First, we will assign a SR-IOV logical port to the LPAR, marking it as migratable. We will also specify the VLAN id tag of 691 (which is required for our network). Then, we will configure a backing device for the migratable logical port; the backing device, in this example, will be another VEA, connected to the VLAN 691.
Let’s look at each step in detail.
Configuration Steps for HNV for an AIX LPAR
1. Assign a SR-IOV Logical Port to the LPAR. Click Hardware Virtualized I/O for your LPAR. Click Add Port.
Click Select an SR-IOV physical port.
Select the SR-IOV physical port and click OK.
2. Click Migratable. Click Virtual Ethernet Adapter.
| Note: If you click on the yellow triangle exclamation icon, it provides an informational message about Linux (which we can ignore because we are configuring HNV for an AIX LPAR).
|
.png "Picture5-(1).png")
3. Click Virtual Ethernet Adapter. Click Configure backup device.
Note: You can also configure a vNIC device as backup device for HNV. Please refer to the IBM online documentation for details.
4. Select the VLAN for the VEA. In our case VLAN 691.
Enter the PVID of the VLAN that the SR-IOV LP will be connected to on the network. Again, in our example, VLAN 691 is the desired VLAN. Click OK.
That’s it! The HNV configuration is complete!
You’ll see a new SR-IOV logical port listed for your LPAR, under Hardware Virtualised I/O.
You can view details of your newly configured port by selecting the port, clicking Action and View Logical Port.
In the view panel, you will see details of your logical port, including the ticked Migratable option and the Backing Device Type (in this case VEA). You can click View to see information for the backup network device.
Clicking View provides details of the Backing device type (virtual Ethernet adapter) settings, such as the virtual network name, the VEA (slot) ID, the virtual switch name and the Port VLAN ID (PVID).
You can also view similar information in the partition profile for the LPAR.
Under Partition Actions, click Profiles, Manage Profiles.
Click on the profile name.
Click on the SR-IOV Logical Ports tab.
Select the logical port and click SR-IOV Menu and Edit Logical Port.
You’re presented with the Logical Port Properties panel. It displays information relating to the SR-IOV logical port, such as the Migratable option selected, the Backing Device Type (VEA) and the Virtual Network Attached.
Click Cancel and Close to exit the partition profile.
Under Virtual Networks for the LPAR, you’ll find another view of the virtual Ethernet adapters assigned to the LPAR. This view shows the SR-IOV Backup adapter column(s) containing the location code of the SR-IOV logical port, for which this VEA is its backup adapter.
The HMC CLI also provides a command to view the HNV configuration for a partition. The lshwres command, with the –rsubtype logport flags, displays the SR-IOV logical port information, including the HNV specific configuration attributes, migratable, backup_device_slot_num, backup_device_type and backup_veth_vnetwork.
hmc1:~> lshwres -r sriov -m pvc-p9b-SN784FC00 --rsubtype logport --level eth config_id=0,lpar_name=aix73_sriov-e6cdd198-0000002a,lpar_id=8,lpar_state=Running,is_required=1,adapter_id=1,logical_port_id=27004008,logical_port_type=eth,drc_name=PHB 4104,location_code=U78D2.001.WZS124V-P1-C7-T1-S8,functional_state=1,phys_port_id=0,debug_mode=0,diag_mode=0,huge_dma_window_mode=0,capacity=2.0,max_capacity=100.0,promisc_mode=0,mac_addr=26dab42a4500,curr_mac_addr=26dab42a4500,allowed_os_mac_addrs=all,allowed_vlan_ids=all,port_vlan_id=691,migratable=1,backup_device_slot_num=4,backup_device_type=veth,backup_veth_vnetwork=691-ETHERNET0,is_vnic_backing_device=0
Automatic AIX Configuration Changes for HNV
So, what configuration was automatically performed on the AIX LPAR to support the HNV setup?
On the AIX LPAR we observe that several new network adapters were automatically configured by the HNV setup.
root@aix73-sriov / # lsdev -Cc adapter | grep ^ent ent0 Available Virtual I/O Ethernet Adapter (l-lan) ent1 Available 06-00 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) ent2 Available EtherChannel / IEEE 802.3ad Link Aggregation ent3 Available Virtual I/O Ethernet Adapter (l-lan)
The ent2 EtherChannel device is automatically configured by HNV, as an AIX network interface backup (NIB) device. The primary adapter is ent1, which is the SR-IOV logical port, VF adapter. The backup adapter is ent3, which is a new VEA.
root@aix73-sriov / # lsattr -El ent2 adapter_names ent1 EtherChannel Adapters True alt_addr 0x000000000000 Alternate EtherChannel Address True auto_recovery yes Enable automatic recovery after failover True backup_adapter ent3 Adapters to use when the primary channel fails True delay_log 0 Delay error logging (in seconds) during open True hash_mode default Determines how outgoing adapter is chosen True hcn_id 0xae01fbf Hybrid Converged Network ID True interval long Determines interval value for IEEE 802.3ad mode True mac_swap no Enable MAC address swap between primary and backup on failover True mode standard EtherChannel mode of operation True netaddr 0 Address to ping True noloss_failover yes Enable lossless failover after ping failure True num_retries 3 Times to retry ping before failing True retry_time 1 Wait time (in seconds) between pings True use_alt_addr no Enable Alternate EtherChannel Address True use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames True root@aix73-sriov / # topas -E Topas Monitor for host: aix73-sriov Interval: 2 Mon Apr 17 02:57:34 2023 =============================================================================== Network KBPS I-Pack O-Pack KB-In KB-Out ent2 (EC) 0.1 0.6 0.6 0.1 0.1 |--ent1 (PRIM) 0.1 0.6 0.6 0.1 0.1 --ent3 (VETH BACK) 0.0 0.0 0.0 0.0 0.0 en0 0.0 0.0 0.0 0.0 0.0 lo0 0.0 0.0 0.0 0.0 0.0
The VEA adapter, ent3, is configured with virtual adapter (slot) id, 4.
root@aix73-sriov / # lscfg -vpl ent3 | grep ent ent3 U9009.42G.78443EX-V8-C4-T1 Virtual I/O Ethernet Adapter (l-lan)
Testing the HNV Network Setup on AIX (subhead)
At this point, the HNV setup has configured the AIX NIB interface, but the IP address is still assigned to the existing VEA, en0. So, next we will temporarily move (transfer) the IP address from the VEA interface to the NIB interface, for testing.
We will need to disconnect the network to make this change, so, we connect to the terminal console of the LPAR (via the HMC CLI, with vtmenu) and run the ifconfig transfer command. Then we test the network is functioning, on the NIB interface, by pinging an address on our network.
root@aix73-sriov / # tty
/dev/vty0
root@aix73-sriov / # ifconfig en0 10.1.1.123 transfer en2
root@aix73-sriov / # ifconfig -a
en0: flags=1e084863,81cc0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
en2: flags=1e084863,18c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 10.1.1.123 netmask 0xffffff00 broadcast 10.1.1.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
inet6 ::1%1/64
tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
root@aix73-sriov / # ping 10.8.4.100
PING 10.8.4.100: (10.8.4.100): 56 data bytes
64 bytes from 10.8.4.100: icmp_seq=0 ttl=63 time=0 ms
64 bytes from 10.8.4.100: icmp_seq=1 ttl=63 time=0 ms
64 bytes from 10.8.4.100: icmp_seq=2 ttl=63 time=0 ms
64 bytes from 10.8.4.100: icmp_seq=3 ttl=63 time=0 ms
^C
--- 10.8.4.100 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0/0/0 ms
Testing LPM with HNV
With the IP now assigned to the NIB en2 interface and a successful ping test completed, we’re ready to test LPM with the HNV configuration.
Remember, when we use LPM to migrate the LPAR to another server, we’re expecting the NIB interface to failover to the VEA, ent3, and when the LPM is finished, the NIB should fall back to the primary adapter, ent1 (which is the SR-IOV VF adapter). All of this should happen automatically.
Our AIX LPAR currently resides on the Power server named pvc-p9b-SN784FC00.
We will use LPM to move the LPAR to another Power server named pvc-p9a-SN78443EX.
hmc1:~> migrlpar -o v -m pvc-p9b-SN784FC00 -t pvc-p9a-SN78443EX -p aix73_sriov-e6cdd198-0000002a --migsriov 1 hmc1:~> echo $? 0 hmc1:~> migrlpar -o m -m pvc-p9b-SN784FC00 -t pvc-p9a-SN78443EX -p aix73_sriov-e6cdd198-0000002a --migsriov 1 hmc1:~> echo $? 0
The last migrlpar flag (shown in the commands above) is a new option called migsriov. When validating or migrating partitions, use this option to specify whether the HMC is required to recreate each partition’s migratable SR-IOV logical ports on the destination managed system.
We can use this option to specify whether the HMC is required to recreate each partition’s HNV migratable SR-IOV logical port on the destination server. You can specify either 1 or 2 with this flag.
If you specify 1, the HMC is required to recreate the HNV migratable SR-IOV logical port on the destination server.
If you specify 2, the HMC is not required to recreate the HNV adapter but will try to do so if possible. But if it is unable to, it will migrate the LPAR regardless.
If you specify 1, and the HMC is unable to configure the HNV adapter on the destination server, then an error is displayed and the LPM operation will fail for the LPAR.
The default value for this option is 1, so it is only included here for demonstration purposes.
Source system – LPAR is migrating away
Destination system – LPAR is arriving
The AIX entstat command shows that the NIB failover has occurred before LPM to the other system.
root@aix73-sriov / # entstat -d ent2 | grep channel Active channel: primary channel < Before LPM root@aix73-sriov / # entstat -d ent2 | grep channel Active channel: backup channel < During LPM
The AIX entstat command and the AIX errpt both report that the NIB failover and fallback occurred. With failover occurring before the LPM was running, and the fallback occurring after the LPM was successfully completed.
root@aix73-sriov / # entstat -d ent2 | grep channel Active channel: primary channel root@aix73-sriov / # errpt IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION 8650BE3F 0417030423 I H ent2 ETHERCHANNEL RECOVERY 159D686C 0417030323 I S pmig Client Partition Migration Completed F2646817 0417030323 I S pmig Client Partition Migration Started 59224136 0417030323 P H ent2 ETHERCHANNEL FAILOVER root@aix73-sriov / # errpt -a | grep All All primary EtherChannel adapters failed: switching over to backup adapter root@aix73-sriov / # errpt -a | grep prim A primary adapter in the EtherChannel recovered: returning to main channel
We observe that the HNV config was successfully recreated during LPM by reviewing the partition profile details and looking under the SR-IOV Logical Ports tab.
We can also view the LPAR’s HNV configuration on the destination system using the HMC CLI and the lshwres command, as shown below.
hmc1:~> lshwres -r sriov -m pvc-p9a-SN78443EX --rsubtype logport --level eth config_id=0,lpar_name=aix73_sriov-e6cdd198-0000002a,lpar_id=8,lpar_state=Running,is_required=1,adapter_id=1,logical_port_id=27004003,logical_port_type=eth,drc_name=PHB 4099,location_code=U78D2.001.WZS0WH0-P1-C7-T1-S3,functional_state=1,phys_port_id=0,debug_mode=0,diag_mode=0,huge_dma_window_mode=0,capacity=2.0,max_capacity=100.0,promisc_mode=0,mac_addr=26dab42a4500,curr_mac_addr=26dab42a4500,allowed_os_mac_addrs=all,allowed_vlan_ids=all,port_vlan_id=691,migratable=1,backup_device_slot_num=4,backup_device_type=veth,backup_veth_vnetwork=691-ETHERNET0,is_vnic_backing_device=0
Changing AIX IP Configuration (Permanently) for HNV
Recall that we temporarily assigned (transferred) the IP address from en0 to en2. In this state, if we rebooted/restarted the LPAR, the IP configuration would revert and the IP address would again be configured on the en0 interface, which is not desirable.
The en0 VEA interface ODM entry still contains the IP address information.
root@aix73-sriov / # lsattr -El en0 alias4 IPv4 Alias including Subnet Mask True alias6 IPv6 Alias including Prefix Length True arp on Address Resolution Protocol (ARP) True authority Authorized Users True broadcast Broadcast Address True monitor off Enable/Disable monitor for virtual Ethernet True mtu 1500 Maximum IP Packet Size for This Device True mtu_bypass on Enable/Disable largesend for virtual Ethernet True netaddr 10.1.1.123 Internet Address True netaddr6 IPv6 Internet Address True netmask 255.255.255.0 Subnet Mask True prefixlen Prefix Length for IPv6 Internet Address True remmtu 576 Maximum IP Packet Size for REMOTE Networks True rfc1323 Enable/Disable TCP RFC 1323 Window Scaling True security no Security Level True state up Current Interface Status True tcp_mssdflt Set TCP Maximum Segment Size True tcp_nodelay Enable/Disable TCP_NODELAY Option True tcp_recvspace Set Socket Buffer Space for Receiving True tcp_sendspace Set Socket Buffer Space for Sending True thread off Enable/Disable thread attribute True
The en2 NIB interface ODM entry does not contain the IP address information.
root@aix73-sriov / # lsattr -El en2 alias4 IPv4 Alias including Subnet Mask True alias6 IPv6 Alias including Prefix Length True arp on Address Resolution Protocol (ARP) True authority Authorized Users True broadcast Broadcast Address True monitor off Enable/Disable monitor for virtual Ethernet True mtu 1500 Maximum IP Packet Size for This Device True mtu_bypass on Enable/Disable largesend for virtual Ethernet True netaddr Internet Address True netaddr6 IPv6 Internet Address True netmask Subnet Mask True prefixlen Prefix Length for IPv6 Internet Address True remmtu 576 Maximum IP Packet Size for REMOTE Networks True rfc1323 Enable/Disable TCP RFC 1323 Window Scaling True security none Security Level True state down Current Interface Status True tcp_mssdflt Set TCP Maximum Segment Size True tcp_nodelay Enable/Disable TCP_NODELAY Option True tcp_recvspace Set Socket Buffer Space for Receiving True tcp_sendspace Set Socket Buffer Space for Sending True thread off Enable/Disable thread attribute True root@aix73-sriov / #
To configure the IP address permanently on en2, and to ensure it is preserved on system restarts, we need to remove the IP configuration from the AIX ODM, for en0 (VEA). Then we need to add the IP configuration details to the ODM for en2 (the NIB).
We use the AIX chdev command to configure the IP information for the en2 interface into the ODM. Note that we use the -P flag to only update the ODM. We do this because the IP address is already configured and active on the en2 interface and we only need to store the information in the ODM, not configure it online.
root@aix73-sriov / # chdev -l en2 -a netaddr=10.1.1.123 -a netmask=255.255.255.0 -a state=up -P en2 changed root@aix73-sriov / # lsattr -El en2 alias4 IPv4 Alias including Subnet Mask True alias6 IPv6 Alias including Prefix Length True arp on Address Resolution Protocol (ARP) True authority Authorized Users True broadcast Broadcast Address True monitor off Enable/Disable monitor for virtual Ethernet True mtu 1500 Maximum IP Packet Size for This Device True mtu_bypass on Enable/Disable largesend for virtual Ethernet True netaddr 10.1.1.123 Internet Address True netaddr6 IPv6 Internet Address True netmask 255.255.255.0 Subnet Mask True ...
We use the AIX rmdev command to remove the en0 interface from the ODM and then we run the cfgmgr command to reconfigure the device and ensure the IP information is no longer in the ODM.
root@aix73-sriov / # rmdev -dl en0 en0 deleted root@aix73-sriov / # cfgmgr root@aix73-sriov / # lsattr -El en0 alias4 IPv4 Alias including Subnet Mask True alias6 IPv6 Alias including Prefix Length True arp on Address Resolution Protocol (ARP) True authority Authorized Users True broadcast Broadcast Address True monitor off Enable/Disable monitor for virtual Ethernet True mtu 1500 Maximum IP Packet Size for This Device True mtu_bypass on Enable/Disable largesend for virtual Ethernet True netaddr Internet Address True netaddr6 IPv6 Internet Address True netmask Subnet Mask True ...
Reviewing the AIX NIB Configuration
To view the AIX NIB details and statistics, we can use the AIX entstat command. For example, the command below displays the configured VLAN/PVID for the NIB devices, ent1 (VF) and ent3 (VEA).
root@aix73-sriov / # entstat -d ent2 | grep -i vlan Port VLAN (Priority:ID): 7:0691 <<< SR-IOV VF VLAN ACL Status: Disabled Enabled VLAN IDs: None Invalid VLAN ID Packets: 0 Port VLAN ID: 691 <<<< VEA VLAN Tag IDs: None ..
In the example below the entstat command displays both the Primary and Backup Adapters for the NIB, as well as the VLAN/PVID for each adapter.
root@aix73-sriov / # entstat -d ent2
-------------------------------------------------------------
ETHERNET STATISTICS (ent2) :
Device Type: EtherChannel
Hardware Address: ee:ed:fe:01:34:04
Elapsed Time: 0 days 0 hours 29 minutes 27 seconds
Transmit Statistics: Receive Statistics:
-------------------- -------------------
Packets: 1381 Packets: 1673
Bytes: 260608 Bytes: 157053
Interrupts: 18 Interrupts: 1570
Transmit Errors: 0 Receive Errors: 0
Packets Dropped: 0 Packets Dropped: 0
...
Statistics for every adapter in the EtherChannel:
-------------------------------------------------
Number of primary adapters: 1
Number of backup adapters: 1
Active channel: primary channel
Operating mode: Standard mode
MAC swap: disabled
=============================================================
PRIMARY ADAPTERS
=============================================================
-------------------------------------------------------------
ETHERNET STATISTICS (ent1) :
Device Type: 10GbE SFP+ SR PCIe3 Converged Network Adapter VF
Hardware Address: ee:ed:fe:01:34:04
...
10GbE SFP+ SR PCIe3 Converged Network Adapter VF
-------------------------------------------------------------
Device Statistics:
------------------
Device ID: df1028e214100f04
Version: 1
Physical Port Link Status: Up
Logical Port Link Status: Up
Physical Port Speed: 10 Gbps Full Duplex
...
Port VLAN (Priority:ID): 7:0691
...
=============================================================
BACKUP ADAPTERS
=============================================================
-------------------------------------------------------------
ETHERNET STATISTICS (ent3) :
Device Type: Virtual I/O Ethernet Adapter (l-lan)
Hardware Address: ee:ed:fe:01:34:04
...
Port VLAN ID: 691
...
AIX Alog Configuration Logs
During the automatic configuration of the AIX NIB interface, AIX captures debug information to the cfg alog file. You can view the commands that were run to configure the devices and the NIB, using the alog command. The alog log files are saved to /var/adm/ras on your AIX system.
For example, the output below shows some of the initial commands used to dynamically configure the NIB interface.
Creating the NIB device with primary adapter ent1 (VF):
root@aix73-sriov /var/adm/ras # alog -ot cfg | grep ent2 MS 11862378 11928014 21:34:30.231 /usr/lib/methods/chgvlan -l ent2 -a base_adapter=ent1 vlan_tag_id=691 MS 11862380 11928014 21:34:30.237 /usr/lib/methods/cfgvlan -l ent2 MS 11927808 7209440 21:52:54.474 /usr/lib/methods/chgent -l ent2 -a adapter_names=ent1 M0 10682842 21:52:54.502 cfghcn.c 139 hcn_add_prim: Created new ech ent2 with adapter_names ent1 ...
Creating the backup device ent3, slot 4 (VEA):
root@aix73-sriov /var/adm/ras # alog -ot cfg | grep U9009.42G.784FC00-V8-C4 CS 10617238 11010446 21:52:54.987 /usr/sbin/drmgr -a -c slot -s U9009.42G.784FC00-V8-C4 -w 5 -d 1 CS 10617238 11010446 21:52:54.999 /usr/sbin/drmgr -a -c slot -s U9009.42G.784FC00-V8-C4 -w 5 -d 1 C4 10617238 21:52:55.000 drc_name: U9009.42G.784FC00-V8-C4 C4 10617238 21:52:55.015 checking for U9009.42G.784FC00-V8-C4 in all=0x2002fd48 ... MS 7864614 8716722 00:43:44.131 /usr/lib/methods/cfgvioent -l ent3 M4 7864614 00:43:44.131 Parallel mode = 1 M4 7864614 00:43:44.131 Get CuDv for ent3 M4 7864614 00:43:44.131 Get device PdDv, uniquetype=adapter/vdevice/IBM,l-lan M4 7864614 00:43:44.131 Get parent CuDv, name=vio0 ...
Viewing HNV Configuration for a Managed System
You can view the HNV configuration for a managed system by looking at the information displayed under PowerVM -> Hardware Virtualized I/O.
In this view you can see all the SR-IOV logical ports that have been configured on the entire system, where they have been assigned and whether they are migratable (i.e., HNV capable).
Click on Hardware Virtualized I/O and select the SR-IOV adapter from the drop-down list.
The Logical Ports view provides a list of all the partitions that have SR-IOV logical ports assigned and if they are Migratable. In the visual below, we can see that the LPAR named aix73_sriov-e6cdd198-0000002a has a logical port assigned and it is Migratable (Yes). This LPAR is configured for HNV, can take advantage of the high performance SRI-OV adapter and participate in LPM (mobility) operations.
Monitoring Dynamic HNV NIB Changes on AIX During LPM
During a mobility operation, such as LPM, the HNV capable LPAR will automatically failover to the backup adapter for the HNV adapter.
For example, we observed that when the LPM operation was initiated, the HNV adapter automatically performed a failover to the backup adapter, which in our example was the VEA (in slot 4). Once the NIB failover was completed successfully, the primary adapter, which was the SR-IOV logical port, was dynamically removed from the LPAR. At this point the LPAR could live migrate to the other server. Once the LPAR arrived and was operating on the destination server, a new SR-IOV logical port (VF adapter) was assigned to the LPAR and was, once again, introduced into the HNV NIB configuration as the primary adapter. At which point, the NIB automatically performed a fallback to the primary interface. This completed the LPM operation for the HNV capable LPAR.
We can monitor this process during a LPM operation. If we run the following small script, we can monitor the dynamic removal and addition of the VF adapter on the AIX LPAR.
Small while loop to monitor the NIB status:
# while true do echo echo == lsdev -Cc adapter | grep ^ent lscfg -vpl ent1 | grep ent entstat -d ent2 | grep channel echo == sleep 5 echo done == ent0 Available Virtual I/O Ethernet Adapter (l-lan) ent1 Available 0E-00 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) < Primary adapter for NIB before LPM ent2 Available EtherChannel / IEEE 802.3ad Link Aggregation < NIB adapter ent3 Available Virtual I/O Ethernet Adapter (l-lan) < Backup adapter for NIB ent1 U78D2.001.WZS124V-P1-C7-T1-S15 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) < Primary adapter slot and logical port numbers Active channel: primary channel < Primary adapter is active before LPM == ... == ent0 Available Virtual I/O Ethernet Adapter (l-lan) ent1 Available 0E-00 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) < Before LPM ent2 Available EtherChannel / IEEE 802.3ad Link Aggregation ent3 Available Virtual I/O Ethernet Adapter (l-lan) ent1 U78D2.001.WZS124V-P1-C7-T1-S15 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) Active channel: primary channel == ... == ent0 Available Virtual I/O Ethernet Adapter (l-lan) ent2 Available EtherChannel / IEEE 802.3ad Link Aggregation < Primary SR-IOV adapter removed by DLPAR. Ready for LPM ent3 Available Virtual I/O Ethernet Adapter (l-lan) lscfg: device ent1 not found. < Primary adapter, ent1 has been removed for LPM Active channel: backup channel < Backup adapter is now active. Ready for LPM == … == ent0 Available Virtual I/O Ethernet Adapter (l-lan) ent1 Available 0E-00 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) < SR-IOV VF adapter added by DLPAR after LPM ent2 Available EtherChannel / IEEE 802.3ad Link Aggregation ent3 Available Virtual I/O Ethernet Adapter (l-lan) ent1 U78D2.001.WZS124V-P1-C7-T1-S7 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) < SR-IOV LP dynamically configured after LPM Active channel: primary channel < Primary adapter active again after LPM ==
Alternatively, we can simply monitor for the removal and addition of the primary adapter to the NIB device. Below is another small script to monitor the changes to the primary and backup adapters in the NIB interface, during LPM:
# while true do echo echo == lsattr -El ent2 | grep -E 'backup_|er_' entstat -d ent2 | grep channel echo == sleep 5 echo done == adapter_names ent1 EtherChannel Adapters True < Primary adapter, ent1, SR-IOV VF adapter before LPM backup_adapter ent3 Adapters to use when the primary channel fails True < Backup adapter, ent3, VEA Active channel: primary channel < Primary adapter, ent1, is active before LPM == == adapter_names NONE EtherChannel Adapters True < Primary adapter dynamically removed during LPM backup_adapter ent3 Adapters to use when the primary channel fails True Active channel: backup channel < Backup adapter now active during LPM == … == adapter_names NONE EtherChannel Adapters True backup_adapter ent3 Adapters to use when the primary channel fails True Active channel: backup channel == … == adapter_names NONE EtherChannel Adapters True backup_adapter ent3 Adapters to use when the primary channel fails True Active channel: backup channel == ... == adapter_names ent1 EtherChannel Adapters True < Primary (SR-IOV VF) adapter dynamically added after LPM backup_adapter ent3 Adapters to use when the primary channel fails True Active channel: primary channel < Primary adapter active again, after LPM ==
This article has provided an example of configuring HNV for an existing AIX LPAR. It also demonstrated Live Partition Mobility with HNV and AIX. If you would like to learn more about PowerVM’s SR-IOV, vNIC or HNV technology, please consider attending the following formal IBM training classes. Both classes cover SR-IOV in detail and are highly recommended:
AN30G IBM PowerVM: Implementing Virtualization
AN31G IBM PowerVM II: Virtualization: Advanced PowerVM Management and Performance
Thanks for reading!