Skip to main content

Configuring Hybrid Network Virtualization for AIX

In this article, IBM’s Chris Gibson explains how to configure Hybrid Network Virtualization for an AIX partition

An overview and description of Hybrid Network Virtualization—or HNV, as it is more commonly known—can be found in this blog written by Chuck Graham, the lead IBM architect for SR-IOV. Its an excellent blog, which I highly recommend you read before attempting to configure HNV in your environment.

As described in the blog, HNV was introduced to allow administrators to use SR-IOV “…for optimal performance and mobility…” As we know, native SR-IOV adapter logical ports offer the best possible network performance, when compared with Virtual NIC and Virtual Ethernet Adapters (backed by VIOS SEA). So, if you need the absolute best network performance from an Ethernet adapter (that can still be shared with other LPARs), then native SR-IOV is the best choice for your configuration. From Chucks blog:

PowerVMs SR-IOV technology allows a partition to have direct access to a shared physical adapter to provide low overhead, high performance networking…”

Having said that, the downside to employing SR-IOV logical ports is that it cannot be used in environments that require Live Partition Mobility (LPM) or Simplified Remote Restart (SRR). As we know, in an LPM/SRR environment, all devices must be virtual. An SR-IOV logical port is not a true virtual adapter, it is considered a physical adapter, and, as such, will prevent a partition from being LPM capable. From Chucks blog:

Previously, partition mobility operations were not allowed when a partition’s configuration included direct access to physical I/O adapters…”

So, to take advantage of the speed and efficiency of SR-IOV logical ports (which map to SR-IOV Virtual Functions or VFs), and allow an LPAR to participate in mobility operations, IBM introduced HNV. Again, from Chucks blog:

One of the key advantages of the HNV approach is the integration of the dynamic remove and add of the Migratable logical port as part of active LPM, inactive LPM or SRR operations…”

The HNV capability is enabled by selecting a new Migratable option when you configure a new SR-IOV logical port.

The aim of this article is to provide a how-to” guide for configuring HNV, with an AIX LPAR in mind. Well show you how to configure an existing LPAR so it can start using a HNV configuration. Well also describe what happens during a LPM operation.

First, a little about the environment. In our lab we have two POWER9, S924 systems, configured with the following:

System Type/Model:              S924/9009-42G
System firmware:                   FW950.50 (VL950_105)
HMC:                                       V10R1 M1021
AIX:                                         7300-01-01-2246
VIOS:                                       3.1.3.21
SR-IOV adapter:                      EN0H. PCIe3 4-port (2×10 Gb FCoE, 2×1 GbE) SFP+ adapter (FC EL38, FC EL56, FC EN0H, and FC EN0J; CCIN 2B93)


In this example, we will reconfigure an existing AIX LPAR to use HNV. The LPAR is currently configured with one VEA, ent0. The VEA is configured on VLAN 691, with an IP address of 10.1.1.123:

root@aix73-sriov # entstat -d ent0 | grep -i vlan
Invalid VLAN ID Packets: 0
Port VLAN ID: 691
VLAN Tag IDs: None

root@aix73-sriov # ifconfig -a
en0: flags=1e084863,81cc0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 10.1.1.123 netmask 0xffffff00 broadcast 10.1.1.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
...

At a high level, we will perform the following steps to configure the LPAR for HNV. First, we will assign a SR-IOV logical port to the LPAR, marking it as migratable. We will also specify the VLAN id tag of 691 (which is required for our network). Then, we will configure a backing device for the migratable logical port; the backing device, in this example, will be another VEA, connected to the VLAN 691.

Lets look at each step in detail.

Configuration Steps for HNV for an AIX LPAR 

1. Assign a SR-IOV Logical Port to the LPAR. Click Hardware Virtualized I/O for your LPAR. Click Add Port.

Click Select an SR-IOV physical port.

Select the SR-IOV physical port and click OK.

2. Click Migratable. Click Virtual Ethernet Adapter.

Note: If you click on the yellow triangle exclamation icon, it provides an informational message about Linux (which we can ignore because we are configuring HNV for an AIX LPAR).

3. Click Virtual Ethernet Adapter. Click Configure backup device.
Note: You can also configure a vNIC device as backup device for HNV. Please refer to the IBM online documentation for details.

4. Select the VLAN for the VEA. In our case VLAN 691.

Figure 9. Attach Virtual Network
Figure 9. Attach Virtual Network view (click to expand)

Enter the PVID of the VLAN that the SR-IOV LP will be connected to on the network. Again, in our example, VLAN 691 is the desired VLAN. Click OK.

Thats it! The HNV configuration is complete!

Youll see a new SR-IOV logical port listed for your LPAR, under Hardware Virtualised I/O.

You can view details of your newly configured port by selecting the port, clicking Action and View Logical Port.

In the view panel, you will see details of your logical port, including the ticked Migratable option and the Backing Device Type (in this case VEA). You can click View to see information for the backup network device.

Clicking View provides details of the Backing device type (virtual Ethernet adapter) settings, such as the virtual network name, the VEA (slot) ID, the virtual switch name and the Port VLAN ID (PVID).

You can also view similar information in the partition profile for the LPAR.

Under Partition Actions, click Profiles, Manage Profiles.

Click on the profile name.

Click on the SR-IOV Logical Ports tab.

Select the logical port and click SR-IOV Menu and Edit Logical Port.

Youre presented with the Logical Port Properties panel. It displays information relating to the SR-IOV logical port, such as the Migratable option selected, the Backing Device Type (VEA) and the Virtual Network Attached.

Click Cancel and Close to exit the partition profile.

Under Virtual Networks for the LPAR, youll find another view of the virtual Ethernet adapters assigned to the LPAR. This view shows the SR-IOV Backup adapter column(s) containing the location code of the SR-IOV logical port, for which this VEA is its backup adapter.

Note: The virtual Ethernet adapter for the HNV configuration does not appear in the traditional Virtual Adapters tab in the partition profile. This information is only visible under the SR-IOV Logical Ports tab and is managed by the Power Hypervisor.

The HMC CLI also provides a command to view the HNV configuration for a partition. The lshwres command, with the –rsubtype logport flags, displays the SR-IOV logical port information, including the HNV specific configuration attributes, migratable, backup_device_slot_num, backup_device_type and backup_veth_vnetwork.

hmc1:~> lshwres -r sriov -m pvc-p9b-SN784FC00 --rsubtype logport --level eth
config_id=0,lpar_name=aix73_sriov-e6cdd198-0000002a,lpar_id=8,lpar_state=Running,is_required=1,adapter_id=1,logical_port_id=27004008,logical_port_type=eth,drc_name=PHB 4104,location_code=U78D2.001.WZS124V-P1-C7-T1-S8,functional_state=1,phys_port_id=0,debug_mode=0,diag_mode=0,huge_dma_window_mode=0,capacity=2.0,max_capacity=100.0,promisc_mode=0,mac_addr=26dab42a4500,curr_mac_addr=26dab42a4500,allowed_os_mac_addrs=all,allowed_vlan_ids=all,port_vlan_id=691,migratable=1,backup_device_slot_num=4,backup_device_type=veth,backup_veth_vnetwork=691-ETHERNET0,is_vnic_backing_device=0

Automatic AIX Configuration Changes for HNV

So, what configuration was automatically performed on the AIX LPAR to support the HNV setup?

On the AIX LPAR we observe that several new network adapters were automatically configured by the HNV setup.

root@aix73-sriov / # lsdev -Cc adapter | grep ^ent
ent0    Available       Virtual I/O Ethernet Adapter (l-lan)
ent1    Available 06-00 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04)
ent2    Available       EtherChannel / IEEE 802.3ad Link Aggregation
ent3    Available       Virtual I/O Ethernet Adapter (l-lan)

The ent2 EtherChannel device is automatically configured by HNV, as an AIX network interface backup (NIB) device. The primary adapter is ent1, which is the SR-IOV logical port, VF adapter. The backup adapter is ent3, which is a new VEA.

root@aix73-sriov / # lsattr -El ent2
adapter_names   ent1           EtherChannel Adapters                                          True
alt_addr        0x000000000000 Alternate EtherChannel Address                                 True
auto_recovery   yes            Enable automatic recovery after failover                       True
backup_adapter  ent3           Adapters to use when the primary channel fails                 True
delay_log       0              Delay error logging (in seconds) during open                   True
hash_mode       default        Determines how outgoing adapter is chosen                      True
hcn_id          0xae01fbf      Hybrid Converged Network ID                                    True
interval        long           Determines interval value for IEEE 802.3ad mode                True
mac_swap        no             Enable MAC address swap between primary and backup on failover True
mode            standard       EtherChannel mode of operation                                 True
netaddr         0              Address to ping                                                True
noloss_failover yes            Enable lossless failover after ping failure                    True
num_retries     3              Times to retry ping before failing                             True
retry_time      1              Wait time (in seconds) between pings                           True
use_alt_addr    no             Enable Alternate EtherChannel Address                          True
use_jumbo_frame no             Enable Gigabit Ethernet Jumbo Frames                           True
 
root@aix73-sriov / # topas -E
Topas Monitor for host:    aix73-sriov Interval:   2   Mon Apr 17 02:57:34 2023
===============================================================================
Network                                KBPS   I-Pack   O-Pack    KB-In   KB-Out
ent2 (EC)                               0.1      0.6      0.6      0.1      0.1
  |--ent1 (PRIM)                       0.1      0.6      0.6      0.1      0.1
   --ent3 (VETH BACK)                  0.0      0.0      0.0      0.0      0.0
en0                                     0.0      0.0      0.0      0.0      0.0
lo0                                     0.0      0.0      0.0      0.0      0.0

The VEA adapter, ent3, is configured with virtual adapter (slot) id, 4.

root@aix73-sriov / # lscfg -vpl ent3 | grep ent
ent3             U9009.42G.78443EX-V8-C4-T1  Virtual I/O Ethernet Adapter (l-lan)

Testing the HNV Network Setup on AIX (subhead)

At this point, the HNV setup has configured the AIX NIB interface, but the IP address is still assigned to the existing VEA, en0. So, next we will temporarily move (transfer) the IP address from the VEA interface to the NIB interface, for testing.

We will need to disconnect the network to make this change, so, we connect to the terminal console of the LPAR (via the HMC CLI, with vtmenu) and run the ifconfig transfer command. Then we test the network is functioning, on the NIB interface, by pinging an address on our network.

root@aix73-sriov / # tty
/dev/vty0
 
root@aix73-sriov / # ifconfig en0 10.1.1.123 transfer en2
root@aix73-sriov / # ifconfig -a
en0: flags=1e084863,81cc0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
en2: flags=1e084863,18c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 10.1.1.123 netmask 0xffffff00 broadcast 10.1.1.255
         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>
        inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
        inet6 ::1%1/64
         tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
 
root@aix73-sriov / # ping 10.8.4.100
PING 10.8.4.100: (10.8.4.100): 56 data bytes
64 bytes from 10.8.4.100: icmp_seq=0 ttl=63 time=0 ms
64 bytes from 10.8.4.100: icmp_seq=1 ttl=63 time=0 ms
64 bytes from 10.8.4.100: icmp_seq=2 ttl=63 time=0 ms
64 bytes from 10.8.4.100: icmp_seq=3 ttl=63 time=0 ms
^C
--- 10.8.4.100 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0/0/0 ms

Testing LPM with HNV

With the IP now assigned to the NIB en2 interface and a successful ping test completed, were ready to test LPM with the HNV configuration.

Remember, when we use LPM to migrate the LPAR to another server, were expecting the NIB interface to failover to the VEA, ent3, and when the LPM is finished, the NIB should fall back to the primary adapter, ent1 (which is the SR-IOV VF adapter). All of this should happen automatically.

Our AIX LPAR currently resides on the Power server named pvc-p9b-SN784FC00.

We will use LPM to move the LPAR to another Power server named pvc-p9a-SN78443EX.

hmc1:~> migrlpar -o v -m pvc-p9b-SN784FC00 -t pvc-p9a-SN78443EX -p aix73_sriov-e6cdd198-0000002a --migsriov 1
hmc1:~> echo $?
0
 
hmc1:~> migrlpar -o m -m pvc-p9b-SN784FC00 -t pvc-p9a-SN78443EX -p aix73_sriov-e6cdd198-0000002a --migsriov 1
hmc1:~> echo $?
0

The last migrlpar flag (shown in the commands above) is a new option called migsriov. When validating or migrating partitions, use this option to specify whether the HMC is required to recreate each partitions migratable SR-IOV logical ports on the destination managed system.

We can use this option to specify whether the HMC is required to recreate each partitions HNV migratable SR-IOV logical port on the destination server. You can specify either 1 or 2 with this flag.

If you specify 1, the HMC is required to recreate the HNV migratable SR-IOV logical port on the destination server.

If you specify 2, the HMC is not required to recreate the HNV adapter but will try to do so if possible. But if it is unable to, it will migrate the LPAR regardless.

If you specify 1, and the HMC is unable to configure the HNV adapter on the destination server, then an error is displayed and the LPM operation will fail for the LPAR.

The default value for this option is 1, so it is only included here for demonstration purposes.

Source system – LPAR is migrating away

Destination system – LPAR is arriving

The AIX entstat command shows that the NIB failover has occurred before LPM to the other system.

root@aix73-sriov / # entstat -d ent2 | grep channel
Active channel: primary channel                        < Before LPM
root@aix73-sriov / # entstat -d ent2 | grep channel
Active channel: backup channel                         < During LPM

The AIX entstat command and the AIX errpt both report that the NIB failover and fallback occurred. With failover occurring before the LPM was running, and the fallback occurring after the LPM was successfully completed.

root@aix73-sriov / # entstat -d ent2 | grep channel
Active channel: primary channel
 
root@aix73-sriov / # errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
8650BE3F   0417030423 I H ent2           ETHERCHANNEL RECOVERY
159D686C   0417030323 I S pmig           Client Partition Migration Completed
F2646817   0417030323 I S pmig           Client Partition Migration Started
59224136   0417030323 P H ent2           ETHERCHANNEL FAILOVER
 
root@aix73-sriov / # errpt -a | grep All
All primary EtherChannel adapters failed: switching over to backup adapter
 
root@aix73-sriov / # errpt -a | grep prim
A primary adapter in the EtherChannel recovered: returning to main channel

We observe that the HNV config was successfully recreated during LPM by reviewing the partition profile details and looking under the SR-IOV Logical Ports tab.

We can also view the LPAR’s HNV configuration on the destination system using the HMC CLI and the lshwres command, as shown below.

hmc1:~> lshwres -r sriov -m pvc-p9a-SN78443EX --rsubtype logport --level eth
config_id=0,lpar_name=aix73_sriov-e6cdd198-0000002a,lpar_id=8,lpar_state=Running,is_required=1,adapter_id=1,logical_port_id=27004003,logical_port_type=eth,drc_name=PHB 4099,location_code=U78D2.001.WZS0WH0-P1-C7-T1-S3,functional_state=1,phys_port_id=0,debug_mode=0,diag_mode=0,huge_dma_window_mode=0,capacity=2.0,max_capacity=100.0,promisc_mode=0,mac_addr=26dab42a4500,curr_mac_addr=26dab42a4500,allowed_os_mac_addrs=all,allowed_vlan_ids=all,port_vlan_id=691,migratable=1,backup_device_slot_num=4,backup_device_type=veth,backup_veth_vnetwork=691-ETHERNET0,is_vnic_backing_device=0

Changing AIX IP Configuration (Permanently) for HNV

Recall that we temporarily assigned (transferred) the IP address from en0 to en2. In this state, if we rebooted/restarted the LPAR, the IP configuration would revert and the IP address would again be configured on the en0 interface, which is not desirable.

The en0 VEA interface ODM entry still contains the IP address information.

 
root@aix73-sriov / # lsattr -El en0
alias4                      IPv4 Alias including Subnet Mask              True
alias6                      IPv6 Alias including Prefix Length            True
arp           on            Address Resolution Protocol (ARP)             True
authority                   Authorized Users                              True
broadcast                   Broadcast Address                             True
monitor       off           Enable/Disable monitor for virtual Ethernet   True
mtu           1500          Maximum IP Packet Size for This Device        True
mtu_bypass    on            Enable/Disable largesend for virtual Ethernet True
netaddr       10.1.1.123  Internet Address                              True
netaddr6                    IPv6 Internet Address                         True
netmask       255.255.255.0 Subnet Mask                                   True
prefixlen                   Prefix Length for IPv6 Internet Address       True
remmtu        576           Maximum IP Packet Size for REMOTE Networks    True
rfc1323                     Enable/Disable TCP RFC 1323 Window Scaling    True
security      no          Security Level                                True
state         up            Current Interface Status                      True
tcp_mssdflt                 Set TCP Maximum Segment Size                  True
tcp_nodelay                 Enable/Disable TCP_NODELAY Option             True
tcp_recvspace               Set Socket Buffer Space for Receiving         True
tcp_sendspace               Set Socket Buffer Space for Sending           True
thread        off           Enable/Disable thread attribute               True

The en2 NIB interface ODM entry does not contain the IP address information.

root@aix73-sriov / # lsattr -El en2
alias4             IPv4 Alias including Subnet Mask              True
alias6             IPv6 Alias including Prefix Length            True
arp           on   Address Resolution Protocol (ARP)             True
authority          Authorized Users                              True
broadcast          Broadcast Address                             True
monitor       off  Enable/Disable monitor for virtual Ethernet   True
mtu           1500 Maximum IP Packet Size for This Device        True
mtu_bypass    on   Enable/Disable largesend for virtual Ethernet True
netaddr            Internet Address                              True
netaddr6           IPv6 Internet Address                         True
netmask            Subnet Mask                                   True
prefixlen          Prefix Length for IPv6 Internet Address       True
remmtu        576  Maximum IP Packet Size for REMOTE Networks    True
rfc1323            Enable/Disable TCP RFC 1323 Window Scaling    True
security      none Security Level                                True
state         down Current Interface Status                      True
tcp_mssdflt        Set TCP Maximum Segment Size                  True
tcp_nodelay        Enable/Disable TCP_NODELAY Option             True
tcp_recvspace      Set Socket Buffer Space for Receiving         True
tcp_sendspace      Set Socket Buffer Space for Sending           True
thread        off  Enable/Disable thread attribute               True
root@aix73-sriov / #

To configure the IP address permanently on en2, and to ensure it is preserved on system restarts, we need to remove the IP configuration from the AIX ODM, for en0 (VEA). Then we need to add the IP configuration details to the ODM for en2 (the NIB).

We use the AIX chdev command to configure the IP information for the en2 interface into the ODM. Note that we use the -P flag to only update the ODM. We do this because the IP address is already configured and active on the en2 interface and we only need to store the information in the ODM, not configure it online.

root@aix73-sriov / # chdev -l en2 -a netaddr=10.1.1.123 -a netmask=255.255.255.0 -a state=up -P
en2 changed
root@aix73-sriov / # lsattr -El en2
alias4                      IPv4 Alias including Subnet Mask              True
alias6                      IPv6 Alias including Prefix Length            True
arp           on            Address Resolution Protocol (ARP)             True
authority                   Authorized Users                              True
broadcast                   Broadcast Address                             True
monitor       off           Enable/Disable monitor for virtual Ethernet   True
mtu           1500          Maximum IP Packet Size for This Device        True
mtu_bypass    on            Enable/Disable largesend for virtual Ethernet True
netaddr       10.1.1.123  Internet Address                              True
netaddr6                    IPv6 Internet Address                         True
netmask       255.255.255.0 Subnet Mask                                   True
...

We use the AIX rmdev command to remove the en0 interface from the ODM and then we run the cfgmgr command to reconfigure the device and ensure the IP information is no longer in the ODM.

root@aix73-sriov / # rmdev -dl en0
en0 deleted
 
root@aix73-sriov / # cfgmgr
root@aix73-sriov / # lsattr -El en0
alias4             IPv4 Alias including Subnet Mask              True
alias6             IPv6 Alias including Prefix Length            True
arp           on   Address Resolution Protocol (ARP)             True
authority          Authorized Users                              True
broadcast          Broadcast Address                             True
monitor       off  Enable/Disable monitor for virtual Ethernet   True
mtu           1500 Maximum IP Packet Size for This Device        True
mtu_bypass    on   Enable/Disable largesend for virtual Ethernet True
netaddr            Internet Address                              True
netaddr6           IPv6 Internet Address                         True
netmask            Subnet Mask                                   True
...

Reviewing the AIX NIB Configuration

To view the AIX NIB details and statistics, we can use the AIX entstat command. For example, the command below displays the configured VLAN/PVID for the NIB devices, ent1 (VF) and ent3 (VEA).

root@aix73-sriov / # entstat -d ent2 | grep -i vlan
Port VLAN (Priority:ID): 7:0691                               <<< SR-IOV VF
VLAN ACL Status: Disabled
Enabled VLAN IDs: None
Invalid VLAN ID Packets: 0
Port VLAN ID:   691                                            <<<< VEA
VLAN Tag IDs:  None
..

In the example below the entstat command displays both the Primary and Backup Adapters for the NIB, as well as the VLAN/PVID for each adapter.

root@aix73-sriov / # entstat -d ent2
-------------------------------------------------------------
ETHERNET STATISTICS (ent2) :
Device Type: EtherChannel
Hardware Address: ee:ed:fe:01:34:04
Elapsed Time: 0 days 0 hours 29 minutes 27 seconds
 
Transmit Statistics:                          Receive Statistics:
--------------------                          -------------------
Packets: 1381                                 Packets: 1673
Bytes: 260608                                 Bytes: 157053
Interrupts: 18                                Interrupts: 1570
Transmit Errors: 0                            Receive Errors: 0
Packets Dropped: 0                            Packets Dropped: 0
...
Statistics for every adapter in the EtherChannel:
-------------------------------------------------
 
Number of primary adapters: 1
Number of backup adapters: 1
Active channel: primary channel
Operating mode: Standard mode
MAC swap: disabled
=============================================================
 
                     PRIMARY ADAPTERS
 
=============================================================
 
-------------------------------------------------------------
 
ETHERNET STATISTICS (ent1) :
Device Type: 10GbE SFP+ SR PCIe3 Converged Network Adapter VF
Hardware Address: ee:ed:fe:01:34:04
...
10GbE SFP+ SR PCIe3 Converged Network Adapter VF
-------------------------------------------------------------
 
Device Statistics:
------------------
Device ID: df1028e214100f04
Version: 1
Physical Port Link Status: Up
Logical Port Link Status: Up
Physical Port Speed: 10 Gbps Full Duplex
...
Port VLAN (Priority:ID): 7:0691
...
=============================================================
 
                      BACKUP ADAPTERS
 
=============================================================
 
-------------------------------------------------------------
 
ETHERNET STATISTICS (ent3) :
Device Type: Virtual I/O Ethernet Adapter (l-lan)
Hardware Address: ee:ed:fe:01:34:04
...
Port VLAN ID:   691
...

AIX Alog Configuration Logs

During the automatic configuration of the AIX NIB interface, AIX captures debug information to the cfg alog file. You can view the commands that were run to configure the devices and the NIB, using the alog command. The alog log files are saved to /var/adm/ras on your AIX system.

For example, the output below shows some of the initial commands used to dynamically configure the NIB interface.

Creating the NIB device with primary adapter ent1 (VF):

root@aix73-sriov /var/adm/ras # alog -ot cfg | grep ent2
MS 11862378 11928014 21:34:30.231 /usr/lib/methods/chgvlan -l ent2 -a base_adapter=ent1 vlan_tag_id=691
MS 11862380 11928014 21:34:30.237 /usr/lib/methods/cfgvlan -l ent2
MS 11927808 7209440 21:52:54.474 /usr/lib/methods/chgent -l ent2 -a adapter_names=ent1
M0 10682842 21:52:54.502 cfghcn.c 139 hcn_add_prim: Created new ech ent2 with adapter_names ent1
...

Creating the backup device ent3, slot 4 (VEA):

root@aix73-sriov /var/adm/ras # alog -ot cfg | grep U9009.42G.784FC00-V8-C4
CS 10617238 11010446 21:52:54.987 /usr/sbin/drmgr -a -c slot -s U9009.42G.784FC00-V8-C4 -w 5 -d 1
CS 10617238 11010446 21:52:54.999 /usr/sbin/drmgr -a -c slot -s U9009.42G.784FC00-V8-C4 -w 5 -d 1
C4 10617238 21:52:55.000 drc_name: U9009.42G.784FC00-V8-C4
C4 10617238 21:52:55.015 checking for U9009.42G.784FC00-V8-C4 in all=0x2002fd48
...
MS 7864614 8716722 00:43:44.131 /usr/lib/methods/cfgvioent -l ent3
M4 7864614 00:43:44.131 Parallel mode = 1
M4 7864614 00:43:44.131 Get CuDv for ent3
M4 7864614 00:43:44.131 Get device PdDv, uniquetype=adapter/vdevice/IBM,l-lan
M4 7864614 00:43:44.131 Get parent CuDv, name=vio0
...

Viewing HNV Configuration for a Managed System

You can view the HNV configuration for a managed system by looking at the information displayed under PowerVM -> Hardware Virtualized I/O.

In this view you can see all the SR-IOV logical ports that have been configured on the entire system, where they have been assigned and whether they are migratable (i.e., HNV capable).

Click on Hardware Virtualized I/O and select the SR-IOV adapter from the drop-down list.

The Logical Ports view provides a list of all the partitions that have SR-IOV logical ports assigned and if they are Migratable. In the visual below, we can see that the LPAR named aix73_sriov-e6cdd198-0000002a has a logical port assigned and it is Migratable (Yes). This LPAR is configured for HNV, can take advantage of the high performance SRI-OV adapter and participate in LPM (mobility) operations.

Monitoring Dynamic HNV NIB Changes on AIX During LPM

During a mobility operation, such as LPM, the HNV capable LPAR will automatically failover to the backup adapter for the HNV adapter.

For example, we observed that when the LPM operation was initiated, the HNV adapter automatically performed a failover to the backup adapter, which in our example was the VEA (in slot 4). Once the NIB failover was completed successfully, the primary adapter, which was the SR-IOV logical port, was dynamically removed from the LPAR. At this point the LPAR could live migrate to the other server. Once the LPAR arrived and was operating on the destination server, a new SR-IOV logical port (VF adapter) was assigned to the LPAR and was, once again, introduced into the HNV NIB configuration as the primary adapter. At which point, the NIB automatically performed a fallback to the primary interface. This completed the LPM operation for the HNV capable LPAR.

We can monitor this process during a LPM operation. If we run the following small script, we can monitor the dynamic removal and addition of the VF adapter on the AIX LPAR.

Small while loop to monitor the NIB status:

# while true
do
 echo
 echo ==
 lsdev -Cc adapter | grep ^ent
 lscfg -vpl ent1 | grep ent
 entstat -d ent2 | grep channel
 echo ==
sleep 5
echo
done
 
 
==
ent0    Available       Virtual I/O Ethernet Adapter (l-lan)
ent1    Available 0E-00 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) < Primary adapter for NIB before LPM
ent2    Available       EtherChannel / IEEE 802.3ad Link Aggregation < NIB adapter
ent3    Available       Virtual I/O Ethernet Adapter (l-lan)         < Backup adapter for NIB
 
ent1 U78D2.001.WZS124V-P1-C7-T1-S15  PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04)                            < Primary adapter slot and logical port numbers
Active channel: primary channel                      < Primary adapter is active before LPM
==
...
==
ent0    Available       Virtual I/O Ethernet Adapter (l-lan)
ent1    Available 0E-00 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) < Before LPM
ent2    Available       EtherChannel / IEEE 802.3ad Link Aggregation
ent3    Available       Virtual I/O Ethernet Adapter (l-lan)
  ent1             U78D2.001.WZS124V-P1-C7-T1-S15  PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04)
Active channel: primary channel
==
...
==
ent0    Available  Virtual I/O Ethernet Adapter (l-lan)
ent2    Available  EtherChannel / IEEE 802.3ad Link Aggregation < Primary SR-IOV adapter removed by DLPAR. Ready for LPM
ent3    Available  Virtual I/O Ethernet Adapter (l-lan)
lscfg: device ent1 not found.                        < Primary adapter, ent1 has been removed for LPM
Active channel: backup channel                       < Backup adapter is now active. Ready for LPM
==
…
==
ent0    Available       Virtual I/O Ethernet Adapter (l-lan)
ent1    Available 0E-00 PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04) < SR-IOV VF adapter added by DLPAR after LPM
ent2    Available       EtherChannel / IEEE 802.3ad Link Aggregation
ent3    Available       Virtual I/O Ethernet Adapter (l-lan)
  ent1             U78D2.001.WZS124V-P1-C7-T1-S7  PCIe3 10GbE SFP+ SR 4-port Converged Network Adapter VF (df1028e214100f04)     < SR-IOV LP dynamically configured after LPM
Active channel: primary channel < Primary adapter active again after LPM
==

Alternatively, we can simply monitor for the removal and addition of the primary adapter to the NIB device. Below is another small script to monitor the changes to the primary and backup adapters in the NIB interface, during LPM:

# while true
do
 echo
 echo ==
 lsattr -El ent2 | grep -E 'backup_|er_'
 entstat -d ent2 | grep channel
 echo ==
sleep 5
echo
done
 
 
==
adapter_names   ent1  EtherChannel Adapters   True  < Primary adapter, ent1, SR-IOV VF adapter before LPM
backup_adapter  ent3  Adapters to use when the primary channel fails  True   < Backup adapter, ent3, VEA
Active channel: primary channel                               < Primary adapter, ent1, is active before LPM
==
 
==
adapter_names   NONE  EtherChannel Adapters   True  < Primary adapter dynamically removed during LPM
backup_adapter  ent3  Adapters to use when the primary channel fails  True
Active channel: backup channel                       < Backup adapter now active during LPM
==
…
 
==
adapter_names   NONE  EtherChannel Adapters   True
backup_adapter  ent3  Adapters to use when the primary channel fails  True
Active channel: backup channel
==
…
 
==
adapter_names   NONE  EtherChannel Adapters   True
backup_adapter  ent3  Adapters to use when the primary channel fails  True
Active channel: backup channel
==
...
 
==
adapter_names   ent1  EtherChannel Adapters   True  < Primary (SR-IOV VF) adapter dynamically added after LPM
backup_adapter  ent3           Adapters to use when the primary channel fails                 True
Active channel: primary channel                               < Primary adapter active again, after LPM
==

This article has provided an example of configuring HNV for an existing AIX LPAR. It also demonstrated Live Partition Mobility with HNV and AIX. If you would like to learn more about PowerVMs SR-IOV, vNIC or HNV technology, please consider attending the following formal IBM training classes. Both classes cover SR-IOV in detail and are highly recommended:

AN30G IBM PowerVM: Implementing Virtualization

AN31G IBM PowerVM II: Virtualization: Advanced PowerVM Management and Performance

Thanks for reading!

References:

Adding SR-IOV logical ports

migrlpar