AIX SR-IOV VF and Cisco CDP
Yesterday, IBM released Technology Level (TL) 2 for AIX 7.3. With TL2, a new capability is delivered for AIX systems using native Single Root I/O Virtualization (SR-IOV) adapters (logical ports). With this update AIX administrators can now display information for Cisco network devices (switches) that are directly connected to their AIX logical partition (LPAR). This is made possible by exploiting the Cisco Discovery Protocol (CDP). AIX provides the cdpd daemon which can receive incoming data packets or messages by using CDP and discovering the physically connected Cisco devices. CDP is a Layer 2 (data link layer) protocol that is media-independent and network-independent. It is used by networking applications that run on Cisco devices to discover other Cisco devices that are directly connected to the device. AIX support for CDP was first introduced in AIX 7.2 TL5, however this was not supported with SR-IOV logical ports. TL2 now supports CDP with SR-IOV logical ports (also known as virtual functions or VFs).
Defining Cisco Discovery Protocol
All Cisco devices transmit CDP packets periodically (the default time interval value is 60 seconds though this is adjustable). If you’re not familiar with CDP, here’s how Cisco defines it:
“CDP is a protocol that runs over Layer 2 (the data link layer) on all Cisco routers, bridges, access servers, and switches. CDP allows network management applications to discover Cisco devices that are
neighbors of already known devices, in particular, neighbors running lower-layer, transparent protocols. With CDP, network management applications can learn the device type and the SNMP agent address of neighboring devices. CDP enables applications to send SNMP queries to neighboring devices”
You can find more information on CDP here.
So, if your AIX LPAR is directly connected to a Cisco network switch—through a physical Ethernet adapter or (now) through a native SR-IOV VF then you can display information about the Cisco switch that the adapter is connected to.
This is handy for troubleshooting network issues, as it allows an AIX admin to identify the network switch that an AIX server is connected to on their network.
Note: The CDP support has been available on AIX and VIOS for several years but it has never supported SR-IOV VF adapters. AIX 7.3 TL2 allows the VF adapter to work with CDP. Also note that this approach is not supported with Virtual Ethernet Adapters (VEAs).
Testing an LPAR on AIX 7.3 TL2 SP1
Here’s what I found in my testing with an AIX LPAR running 7.3 TL2 SP1.
# oslevel -s 7300-02-01-2346
An SR-IOV logical port is assigned to this LPAR and is connected to my network.
# lsdev -Cc adapter | grep VF
ent1 Available 07-00 PCIe3 100/1000 Base-TX 4-port Converged Network Adapter VF (df1028e214103c04)
# ifconfig en1
en1: flags=1e084863,18c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 10.8.12.11 netmask 0xffffff00 broadcast 10.8.12.255
tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
We start the cdpd daemon on AIX.
# startsrc -s cdpd 0513-059 The cdpd Subsystem has been started. Subsystem PID is 20054430.
We add the SR-IOV VF network interface (en1) to the cdpd monitoring framework so that it can receive data packets and messages from the physically connected Cisco switch, using CDP.
# cdpctl add en1 cdpctl: 0812-005 successfully added port en1
We then display the list of network interfaces that are configured to receive incoming data using CDP on this AIX LPAR.
# cdpctl show portlist 1.en1
Finally, we display the CDP information (such as the network switch and router capabilities) that are being received on the configured CDP-capable network interfaces (i.e., en1).
# cdpctl show port en1
Waiting for CDP advertise (default 60 seconds)......
Device ID : route1-n1.local(AGE19190TKY)
Address : 10.10.10.2
Port ID : Ethernet3/43
Capabilities
: Router Level 3
: Level 2 Switch
Cisco switch OS Version : Cisco Nexus Operating System (NX-OS) Software, Version 9.3(9)
Platform : N9K-C9508
Native VLAN ID : 32
Trusted Bitmap : N/A
AVVID untrusted ports : N/A
Duplex : Full
MTU : 576
System Name : route1-n1
System Object ID : N/A
Management Addresses : 10.1.2.10
CDP record received on dev en1 also stored at /tmp/cdp_record_en1
The information is also saved to a file in /tmp called cdp_record_enX, where X is the interface number.
# cat /tmp/cdp_record_en1
CDP Advertise captured on Tue Oct 24 21:09:20 2023
Device ID : route1-n1.local(AGE19190TKY)
Address : 10.10.10.2
Port ID : Ethernet3/43
Capabilities
: Router Level 3
: Level 2 Switch
Cisco switch OS Version : Cisco Nexus Operating System (NX-OS) Software, Version 9.3(9)
Platform : N9K-C9508
Native VLAN ID : 32
Trusted Bitmap : N/A
AVVID untrusted ports : N/A
Duplex : Full
MTU : 576
System Name : route1-n1
System Object ID : N/A
Management Addresses : 10.1.2.10
Please refer to the AIX 7.3 command reference pages for more information:
Also, please check out this awesome post by IBM Champion Andrey Klyachkin on AIX and CDP: Cisco Discovery Protocol and IBM AIX.
The cdpd daemon and the cdpctl command were first introduced in AIX 7.2 TL5, in 2020.
From the “AIX 7.2 What’s New” page:
“Added information about the cdpd daemon and cdpctl command that control the Cisco Discovery Protocol (CDP) incoming data packets or messages from physically connected Cisco devices”
From the AIX 7.2 TL5 release notes:
“bos.net.tcp.cdp
This fileset contains new commands cdpd and cdpctl that are used to receive Cisco Discovery Protocol (CDP) messages for discovering other Cisco devices that are directly connected to the AIX logical partition. The cdpd command runs as a daemon to receive CDP packets that are sent by the resource controller. The cdpctl command controls the CDP daemon by configuring the network interface that is physically connected to the Cisco device”
Note that AIX 7.3 TL2 introduced new Link Layer Discovery Protocol (LLDP) enhancements to report additional details for EtherChannel and underlying adapters when the AIX LLDP daemon is running. View the announcement here.
Initially I considered using lldpd to obtain similar information to cdpd, but there’s no “lldpsvc” attribute for a SR-IOV VF adapter. Find more information about LLDP here or here.
The cdpd changes were also made available with AIX 7.2 TL5 SP7, which also became available in November 2023. The related AIX 7.2 APAR is IJ48479, “cdpd does not add multicast address during port add operation,” “Port information was not available in case of SRIOV VF interfaces. Interface added/enabled for multicast address to handle SRIOV VF interface.” The related AIX 7.3 APAR is IJ48390.
The CDP method is a great alternative to using the classic tcpdump approach (which, by the way, doesn’t work with SR-IOV logical ports). For example:
# tcpdump -nn -v -i en0 -s 1500 -c 1 'ether[20:2] == 0x2000'
The tcpdump technique still works well with physical adapters on AIX and is discussed in detail here:
Using tcpdump to Discover Network Information
Note: I tested this (cdpd/cdpctl) against one of the Shared Ethernet Adapters (SEA) on my VIOS and it also worked as advertised. However, I note that this was always working with an SEA consisting of an underlying physical Ethernet adapter; even on previous releases of VIOS code.
$ ioslevel
3.1.3.21
$ oem_setup_env
# lsdev -Cc adapter | grep ent5
ent5 Available Shared Ethernet Adapter
# ifconfig en5
en5: flags=1e084863,2000c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 10.4.7.201 netmask 0xfffff000 broadcast 10.4.7.255
# startsrc -s cdpd
0513-059 The cdpd Subsystem has been started. Subsystem PID is 30146884.
# cdpctl add en5
cdpctl: 0812-005 successfully added port en5
# cdpctl show port en5
Waiting for CDP advertise (default 60 seconds)......
Device ID : labrouter.local(AHA09360044)
Address : 10.1.7.100
Port ID : mgmt0
Capabilities
: Router Level 3
: Level 2 Switch
Cisco switch OS Version : Cisco Nexus Operating System (NX-OS) Software, Version 6.2(13a)
Platform : DS-C9513
Duplex : Full
System Name : labrouter
System Object ID : N/A
Management Addresses : 10.1.7.100
CDP record received on dev en5 also stored at /tmp/cdp_record_en5