AIX SR-IOV VF and Cisco CDP
IBM’s Chris Gibson highlights the latest AIX updates
Defining Cisco Discovery ProtocolAll Cisco devices transmit CDP packets periodically (the default time interval value is 60 seconds though this is adjustable). If you’re not familiar with CDP, here’s how Cisco defines it:
“CDP is a protocol that runs over Layer 2 (the data link layer) on all Cisco routers, bridges, access servers, and switches. CDP allows network management applications to discover Cisco devices that are
neighbors of already known devices, in particular, neighbors running lower-layer, transparent protocols. With CDP, network management applications can learn the device type and the SNMP agent address of neighboring devices. CDP enables applications to send SNMP queries to neighboring devices”
You can find more information on CDP here.
So, if your AIX LPAR is directly connected to a Cisco network switch—through a physical Ethernet adapter or (now) through a native SR-IOV VF then you can display information about the Cisco switch that the adapter is connected to.
This is handy for troubleshooting network issues, as it allows an AIX admin to identify the network switch that an AIX server is connected to on their network.
Note: The CDP support has been available on AIX and VIOS for several years but it has never supported SR-IOV VF adapters. AIX 7.3 TL2 allows the VF adapter to work with CDP. Also note that this approach is not supported with Virtual Ethernet Adapters (VEAs).
Testing an LPAR on AIX 7.3 TL2 SP1Here's what I found in my testing with an AIX LPAR running 7.3 TL2 SP1.
# oslevel -s 7300-02-01-2346An SR-IOV logical port is assigned to this LPAR and is connected to my network.
# lsdev -Cc adapter | grep VF ent1 Available 07-00 PCIe3 100/1000 Base-TX 4-port Converged Network Adapter VF (df1028e214103c04) # ifconfig en1 en1: flags=1e084863,18c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN> inet 10.8.12.11 netmask 0xffffff00 broadcast 10.8.12.255 tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0We start the cdpd daemon on AIX.
# startsrc -s cdpd 0513-059 The cdpd Subsystem has been started. Subsystem PID is 20054430.We add the SR-IOV VF network interface (en1) to the cdpd monitoring framework so that it can receive data packets and messages from the physically connected Cisco switch, using CDP.
# cdpctl add en1 cdpctl: 0812-005 successfully added port en1We then display the list of network interfaces that are configured to receive incoming data using CDP on this AIX LPAR.
# cdpctl show portlist 1.en1Finally, we display the CDP information (such as the network switch and router capabilities) that are being received on the configured CDP-capable network interfaces (i.e., en1).
# cdpctl show port en1 Waiting for CDP advertise (default 60 seconds)...... Device ID : route1-n1.local(AGE19190TKY) Address : 10.10.10.2 Port ID : Ethernet3/43 Capabilities : Router Level 3 : Level 2 Switch Cisco switch OS Version : Cisco Nexus Operating System (NX-OS) Software, Version 9.3(9) Platform : N9K-C9508 Native VLAN ID : 32 Trusted Bitmap : N/A AVVID untrusted ports : N/A Duplex : Full MTU : 576 System Name : route1-n1 System Object ID : N/A Management Addresses : 10.1.2.10 CDP record received on dev en1 also stored at /tmp/cdp_record_en1The information is also saved to a file in /tmp called cdp_record_enX, where X is the interface number.
# cat /tmp/cdp_record_en1 CDP Advertise captured on Tue Oct 24 21:09:20 2023 Device ID : route1-n1.local(AGE19190TKY) Address : 10.10.10.2 Port ID : Ethernet3/43 Capabilities : Router Level 3 : Level 2 Switch Cisco switch OS Version : Cisco Nexus Operating System (NX-OS) Software, Version 9.3(9) Platform : N9K-C9508 Native VLAN ID : 32 Trusted Bitmap : N/A AVVID untrusted ports : N/A Duplex : Full MTU : 576 System Name : route1-n1 System Object ID : N/A Management Addresses : 10.1.2.10Please refer to the AIX 7.3 command reference pages for more information:
Also, please check out this awesome post by IBM Champion Andrey Klyachkin on AIX and CDP: Cisco Discovery Protocol and IBM AIX.
The cdpd daemon and the cdpctl command were first introduced in AIX 7.2 TL5, in 2020.
From the “AIX 7.2 What’s New” page:
“Added information about the cdpd daemon and cdpctl command that control the Cisco Discovery Protocol (CDP) incoming data packets or messages from physically connected Cisco devices”
From the AIX 7.2 TL5 release notes:
This fileset contains new commands cdpd and cdpctl that are used to receive Cisco Discovery Protocol (CDP) messages for discovering other Cisco devices that are directly connected to the AIX logical partition. The cdpd command runs as a daemon to receive CDP packets that are sent by the resource controller. The cdpctl command controls the CDP daemon by configuring the network interface that is physically connected to the Cisco device”
Note that AIX 7.3 TL2 introduced new Link Layer Discovery Protocol (LLDP) enhancements to report additional details for EtherChannel and underlying adapters when the AIX LLDP daemon is running. View the announcement here.
Initially I considered using lldpd to obtain similar information to cdpd, but there’s no “lldpsvc” attribute for a SR-IOV VF adapter. Find more information about LLDP here or here.
The cdpd changes were also made available with AIX 7.2 TL5 SP7, which also became available in November 2023. The related AIX 7.2 APAR is IJ48479, “cdpd does not add multicast address during port add operation," “Port information was not available in case of SRIOV VF interfaces. Interface added/enabled for multicast address to handle SRIOV VF interface.” The related AIX 7.3 APAR is IJ48390.
The CDP method is a great alternative to using the classic tcpdump approach (which, by the way, doesn’t work with SR-IOV logical ports). For example:
# tcpdump -nn -v -i en0 -s 1500 -c 1 'ether[20:2] == 0x2000'The tcpdump technique still works well with physical adapters on AIX and is discussed in detail here:
Using tcpdump to Discover Network Information
Note: I tested this (cdpd/cdpctl) against one of the Shared Ethernet Adapters (SEA) on my VIOS and it also worked as advertised. However, I note that this was always working with an SEA consisting of an underlying physical Ethernet adapter; even on previous releases of VIOS code.
$ ioslevel 188.8.131.52 $ oem_setup_env # lsdev -Cc adapter | grep ent5 ent5 Available Shared Ethernet Adapter # ifconfig en5 en5: flags=1e084863,2000c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN> inet 10.4.7.201 netmask 0xfffff000 broadcast 10.4.7.255 # startsrc -s cdpd 0513-059 The cdpd Subsystem has been started. Subsystem PID is 30146884. # cdpctl add en5 cdpctl: 0812-005 successfully added port en5 # cdpctl show port en5 Waiting for CDP advertise (default 60 seconds)...... Device ID : labrouter.local(AHA09360044) Address : 10.1.7.100 Port ID : mgmt0 Capabilities : Router Level 3 : Level 2 Switch Cisco switch OS Version : Cisco Nexus Operating System (NX-OS) Software, Version 6.2(13a) Platform : DS-C9513 Duplex : Full System Name : labrouter System Object ID : N/A Management Addresses : 10.1.7.100 CDP record received on dev en5 also stored at /tmp/cdp_record_en5
About the author
Chris Gibson is an AIX and PowerVM specialist. He's an IBM Champion for Power Systems, IBM CATE and a technical editor.
See more by Chris Gibson