Skip to main content

AIX Virtual Fibre Channel Enhancement

IBM’s Chris Gibson introduces the new AIX 7.3 vfcstat command and discusses some of the (not so recent) enhancements to the fcstat command

A new virtual Fibre Channel (VFC) client adapter enhancement was included with the release of AIX 7.3 Technology Level 2 Service Pack 1 (November 2023). A new command, called vfcstat, was introduced. This allows administrators to access information for VFC adapters, such as statistics, status and activities.

New vfcstat Command

Prior to this command, administrators could use the cat command to view the information stored in the /proc file system (in /proc/sys/adapter/fc/fcsX). The vfcstat command makes it easier for admins to view this information (without needing to know the location of the various files in /proc).

This command is available for AIX 7.3 systems that are Virtual I/O (VIO) clients, of VIOS running version 3.1.4.x and later (VIOS 3.1.4 provides support for NPIV multi-queue).

From the AIX 7.3 command reference for vfcstat:

“The vfcstat command reports information such as virtual FC adapter state, status, NPort ID, I/O queues information, read/write statistics, binding information with the host (VIOS), link errors, path timeouts and so on. The required information is fetched in real time and displayed in the output. You can reset some of the details such as statistics and error counts. You can also retrieve read/write statistics in formatted output for scripting purposes by using the -F option. The vfcstat command displays the statistics in a specified time interval for the specific number of times. The vfcstat command works on virtual FC client adapter only.”

The new command is delivered with the version 7.3.2.1, devices.vdevice.IBM.vfc-client.rte fileset.

# lslpp -w /usr/bin/vfcstat
  File                                        Fileset               Type
  ----------------------------------------------------------------------------
  /usr/bin/vfcstat
                          devices.vdevice.IBM.vfc-client.rte        Symlink
 
# ls -ltr /usr/bin/vfcstat
lrwxrwxrwx    1 bin      bin              17 Oct 23 18:23 /usr/bin/vfcstat -> /usr/sbin/vfcstat
 
# lslpp -w /usr/sbin/vfcstat
  File                                        Fileset               Type
  ----------------------------------------------------------------------------
  /usr/sbin/vfcstat
                          devices.vdevice.IBM.vfc-client.rte        File
 
# lslpp -lc devices.vdevice.IBM.vfc-client.rte
#Fileset:Level:PTF Id:State:Type:Description:EFIX Locked
/usr/lib/objrepos:devices.vdevice.IBM.vfc-client.rte:7.3.2.1::COMMITTED:F:Virtual Fibre Channel Client Support:
/etc/objrepos:devices.vdevice.IBM.vfc-client.rte:7.3.2.1::COMMITTED:F:Virtual Fibre Channel Client Support:

I tested the vfcstat command on an AIX system running 7300-02-01-2346.

You can use the command to list all the VFC adapters on your AIX system.

# vfcstat -l
fcs0
fcs1

To list all the functions available for a given VFC adapter, run the following:

# vfcstat  -d fcs0
activity
capabilities
hostinfo
link
nport_id
status
tunables
wwpn

To display error counts for a given VFC adapter:

 # vfcstat  -d fcs0 -e
Current time                    : Jan 22 2024, 22:00:52.294
Time since last reset           : 5885149 sec ( 2 months 8 days 2 hours 45 minutes 49 seconds )
 
Link Errors:
           Total errors: 0
        Last 10 minutes: 0
        Last 60 minutes: 0
          Last 24 hours: 0
 
Path Timeouts:
                 Domain: Admin
         Total timeouts: 0
        Last 10 minutes: 0
        Last 60 minutes: 0
          Last 24 hours: 0
 
Path Timeouts:
                 Domain: SCSI
         Total timeouts: 0
        Last 10 minutes: 0
        Last 60 minutes: 0
          Last 24 hours: 0
 
Path Timeouts:
                 Domain: NVMeOF
         Total timeouts: 0
        Last 10 minutes: 0
        Last 60 minutes: 0
          Last 24 hours: 0

Note there are three domains for VFC adapter elements. One is for VFC Admin commands, the second is for SCSI commands and the third is for NVME commands.

Use the -s flag to display various statistics for a given VFC adapter. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/stats. We’ve included the –F option, in the example below, as it displays formatted statistic output (which is easier to read!).

# vfcstat  -d fcs0 -s -F
Current time=Jan 22 2024, 22:38:58.102
Time since last reset=5887435 sec ( 2 months 8 days 3 hours 23 minutes 55 seconds )
Queue_Start
Domain=SCSI
Queue number=0
Read reqs=9814
Read bytes=196898541
Write reqs=2696118
Write bytes=253070213192
No DMA res count=0
No SGE count=0
h_send_dropped=0
h_send_closed=0
h_send_busy=0
No cmd res count=0
Queue_End

To display tunables for a given VFC adapter, use the -d and -f tunables options. This is same as reading proc file /proc/sys/adapter/fc/fcs0/tunables.

# vfcstat  -d fcs0 -f tunables
lg_term_dma     ( Requested / Granted ):     0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size   ( Requested / Granted ):     0x100000 (1M) / 0x100000 (1M)
num_cmd_elems   ( Requested / Granted ):     2048 / 2048
num_sp_cmd_elem ( Requested / Granted ):     1024 / 1024
num_io_queues   ( Requested / Granted ):     8 / 0
num_nvme_queues ( Requested / Granted ):     4 / 0

To display VIOS host information for a given VFC adapter, use the -f hostinfo option. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/hostinfo.

# vfcstat -d fcs0 -f hostinfo
VFC client adapter name                   : fcs0
Host partition name  (VIOS)               : vios1
VFC host adapter name                     : vfchost0
VFC host adapter location code            : U8286.42A.2153B1V-V1-C30
FC adapter name on VIOS                   : fcs0
FC adapter location code  on VIOS         : U78C9.001.WZS00XE-P1-C1-T1

To display WWPN information for a given VFC adapter, use the -f wwpn option. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/wwpn.

# vfcstat -d fcs0 -f wwpn
0xc852100000000011

To reset all fields in the stats function for a given VFC adapter:

# vfcstat  -d fcs0 -s –z

To reset all error counts for a given VFC adapter:

# vfcstat  -d fcs0 -e –z

Here’s a little script I wrote to collect and display tunables output from each of the available functions for all VFC adapters on a system:

myvfcs=$(/usr/bin/vfcstat -l)
vfcsfuncs=$(echo $myvfcs | awk '{print $1}' | xargs /usr/bin/vfcstat -d)
 
echo Displaying functions that are available for a virtual FC client adapter: $myvfcs
echo
echo Functions available for $myvfcs are:
echo $vfcsfuncs
echo
 
echo
for xvfcs in $myvfcs
do
  for eachvfunc in $vfcsfuncs
  do
    echo "VFCSTAT $eachvfunc for $xvfcs" | xargs /usr/local/bin/box
    /usr/bin/vfcstat -d $xvfcs -f $eachvfunc
    echo
  done
done

The output is:

# ./xvfcs
Displaying functions that are available for a virtual FC client adapter: fcs0 fcs1
 
Functions available for fcs0 fcs1 are:
activity capabilities hostinfo link nport_id status tunables wwpn
 
 
#############################
# VFCSTAT activity for fcs0 #
#############################
Admin Queue
 
Admin I/O Queue
 
 
#################################
# VFCSTAT capabilities for fcs0 #
#################################
Requested capabilities         : CAN_MIGRATE, CAN_HANDLE_FPIN
Host responded capabilities    :
 
#############################
# VFCSTAT hostinfo for fcs0 #
#############################
VFC client adapter name                   : fcs0
Host partition name  (VIOS)               : vios1
VFC host adapter name                     : vfchost0
VFC host adapter location code            : U8286.42A.2153B1V-V1-C30
FC adapter name on VIOS                   : fcs0
FC adapter location code  on VIOS         : U78C9.001.WZS00YD-P1-C7-T1
 
 
#########################
# VFCSTAT link for fcs0 #
#########################
Link       UP
topology   FABRIC
speed      8Gb
 
#############################
# VFCSTAT nport_id for fcs0 #
#############################
0xe016a0
 
###########################
# VFCSTAT status for fcs0 #
###########################
State                     0x10 (ONLINE)
Opened                    true
cfg_opn_state             0x0
prev_cfg_channels         0
flags                     0x200082 ( CHANNELS_NOT_SUPPORTED, LINK_SETUP, NONVM_FROM_CFG )
total_io_dma_size         0x20000000
total_dma_size_used       0x9900000
no_chs_reason_code        0x2 ( NO_PROCESSOR_COMPATABILITY )
migrated_count            0
online_time               373 (lbolt)
UP time                   5978367 sec ( 2 months 9 days 4 hours 39 minutes 27 seconds )
 
Admin Queue
    active_cmds           0
    spl_active_cmds       0
    pending_cmds          0
    spl_pending_cmds      0
    resp_cmds             0
 
Admin I/O Queue
    active_cmds           0
    spl_active_cmds       0
    pending_cmds          0
    spl_pending_cmds      0
    waiting_cmds          0
    resp_cmds             0
    offlevel_resp_cmds    0
    cancel_resp_cmds      0
 
#############################
# VFCSTAT tunables for fcs0 #
#############################
lg_term_dma     ( Requested / Granted ):     0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size   ( Requested / Granted ):     0x100000 (1M) / 0x100000 (1M)
num_cmd_elems   ( Requested / Granted ):     2048 / 2048
num_sp_cmd_elem ( Requested / Granted ):     1024 / 1024
num_io_queues   ( Requested / Granted ):     8 / 0
num_nvme_queues ( Requested / Granted ):     4 / 0
 
#########################
# VFCSTAT wwpn for fcs0 #
#########################
0xc852100000000011
 
#############################
# VFCSTAT activity for fcs1 #
#############################
Admin Queue
 
Admin I/O Queue
 
 
#################################
# VFCSTAT capabilities for fcs1 #
#################################
Requested capabilities         : CAN_MIGRATE, CAN_HANDLE_FPIN
Host responded capabilities    :
 
#############################
# VFCSTAT hostinfo for fcs1 #
#############################
VFC client adapter name                   : fcs1
Host partition name  (VIOS)               : vios2
VFC host adapter name                     : vfchost0
VFC host adapter location code            : U8286.42A.2153B1V-V2-C30
FC adapter name on VIOS                   : fcs0
FC adapter location code  on VIOS         : U78C9.001.WZS00XE-P1-C1-T1
 
#########################
# VFCSTAT link for fcs1 #
#########################
Link       UP
topology   FABRIC
speed      8Gb
 
#############################
# VFCSTAT nport_id for fcs1 #
#############################
0xe016c0
 
###########################
# VFCSTAT status for fcs1 #
###########################
State                     0x10 (ONLINE)
Opened                    true
cfg_opn_state             0x0
prev_cfg_channels         0
flags                     0x200082 ( CHANNELS_NOT_SUPPORTED, LINK_SETUP, NONVM_FROM_CFG )
total_io_dma_size         0x20000000
total_dma_size_used       0x9900000
no_chs_reason_code        0x2 ( NO_PROCESSOR_COMPATABILITY )
migrated_count            0
online_time               373 (lbolt)
UP time                   5978367 sec ( 2 months 9 days 4 hours 39 minutes 27 seconds )
 
Admin Queue
    active_cmds           0
    spl_active_cmds       0
    pending_cmds          0
    spl_pending_cmds      0
    resp_cmds             0
 
Admin I/O Queue
    active_cmds           0
    spl_active_cmds       0
    pending_cmds          0
    spl_pending_cmds      0
    waiting_cmds          0
    resp_cmds             0
    offlevel_resp_cmds    0
    cancel_resp_cmds      0
 
#############################
# VFCSTAT tunables for fcs1 #
#############################
lg_term_dma     ( Requested / Granted ):     0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size   ( Requested / Granted ):     0x100000 (1M) / 0x100000 (1M)
num_cmd_elems   ( Requested / Granted ):     2048 / 2048
num_sp_cmd_elem ( Requested / Granted ):     1024 / 1024
num_io_queues   ( Requested / Granted ):     8 / 0
num_nvme_queues ( Requested / Granted ):     4 / 0
 
#########################
# VFCSTAT wwpn for fcs1 #
#########################
0xc852100000000013

Enhancements to fcstat

In a somewhat related set of enhancements, the fcstat command was also updated with several new options that provide additional information to the administrator. You may or may not be aware, but these new options have been available for some time with AIX 7.3. The new options are:

  • -p Displays a time-series report of traffic statistics for a specific transport protocol (TP) that is specified with the protocol parameter.
  • -t Displays a time-series report of the traffic statistics continuously with a time interval between two consecutive reports, which is specified with the interval parameter.
  • -e Displays all the statistics, that includes the device-specific statistics such as driver statistics, link statistics, FC4 types and vital small form-factor pluggable (SFP) parameters.
  • -d Displays the statistics by opening the adapter in diagnostic mode.

Here’s an example of using the -t and -p options. In the example below we have chosen to view the SCSI protocol statistics for the fcs0 adapter and to display the time-series report of traffic statistics with a one second interval.

# fcstat -t 1 -p scsi fcs0
 
FIBRE CHANNEL STATISTICS REPORT: fcs0
Device Type: Virtual Fibre Channel Client Adapter (adapter/vdevice/IBM,vfc-client)
 
TP         Read Reqs(K)     Write Reqs(K)   Read (GB)  Write (GB)
===== ================= ================= =========== ===========
SCSI                  9              2314           0         203
SCSI                  9              2315           0         203
SCSI                  9              2316           0         203
SCSI                  9              2317           0         203
SCSI                  9              2318           0         203
SCSI                  9              2319           0         203
SCSI                  9              2320           0         203
SCSI                  9              2322           0         204
SCSI                  9              2323           0         204
SCSI                  9              2324           0         204
SCSI                  9              2326           0         204
SCSI                  9              2328           0         204
SCSI                  9              2329           0         205
SCSI                  9              2330           0         205
SCSI                  9              2332           0         205
SCSI                  9              2334           0         205
SCSI                  9              2335           0         205
SCSI                  9              2336           0         206
SCSI                  9              2338           0         206
SCSI                  9              2340           0         206

In the output above the time-series report shows the following statistics:

  • Read Reqs(K): The number of read or input requests, in thousands.
  • Write Reqs(K): The number of write or output requests, in thousands.
  • Read(GB): The number of input bytes that are read, in gigabytes.
  • Write(GB): The number of output bytes that are written, in gigabytes.

If you do not have any NVME adapters/devices on your AIX system (or you are only interested in FC SCSI traffic), you can remove the -p flag and shorten the command to, fcstat -t 1 fcs0.

The -e option displays vital small form-factor pluggable (SFP) parameters such as vendor information, temperature, voltage information, as well as, the device-specific statistics such as driver statistics, link statistics and FC4 protocol types.

Previously, some of this information was only available using a special tool (called emfc_power) provided by IBM support. This option is useful with physical (not virtual) fibre channel adapters. It appears this enhancement may have been delivered based on this customer enhancement request. In the example below, we are displaying fcstat output for a 32GB FC adapter (note that older adapters may not support this option, producing no FC SFP output).

# fcstat -e fcs0
FIBRE CHANNEL STATISTICS REPORT: fcs0
 
Device Type: PCIe3 2-Port 32Gb FC Adapter (df1000e314101506) (adapter/pciex/df1000e31410150)
...etc...
FC SFP Information
  Vendor Name: AVAGO
  Vendor OUI:  0000176a
  Vendor PN:   AFBR-57G5MZ-ELX
  Temperature: 42.371 C     [Range -128 C - +128 C]
  Voltage:     3.331 V      [Range 0 V - +6.55 V]
  TX Bias:     5.466 mA     [Range 0 mA - 131 mA]
  TX Power:   -1.4837 dBm   [Range -40 dBm - +8.2 dBm]
               0.7106 mW    [Range 0 mW - 6.5535 mW]
  RX Power:   -3.6866 dBm   [Range -40 dBm - +8.2 dBm]
               0.4279 mW    [Range 0 mW - 6.5535 mW ]

The fcstat command opens the device in diagnostic mode by using the -d flag, as shown below. When the link is down and the device is opened in nondiagnostic mode, the fcstat command delays in generating the output. You can use the -c flag to remove this delay. If the device is already opened and the fcstat command is started with the -d flag, the open operation on the device fails with an EACCESS error. Meaning you can now run fcstat on an adapter, with a link that is down, and still obtain some useful information relating to link status, failure and loss of sync/signal counts.

# fcstat -d fcs0
 
FIBRE CHANNEL STATISTICS REPORT: fcs0
 
Device Type: 8Gb PCI Express Dual Port FC Adapter (df1000f114108a03) (adapter/pciex/df1000f114108a0)
Serial Number: 1A202029F7
Option ROM Version: 0278213D
ZA: U2D2.10X13
World Wide Node Name: 0x20000000C9D55D1E
World Wide Port Name: 0x10000000C9D55D1E
 
FC-4 TYPES:
  Supported: 0x0000012000000000000000000000000000000000000000000000000000000000
  Active:    0x0000010000000000000000000000000000000000000000000000000000000000
Class of Service: 3
Port Speed (supported): 8 GBIT
Port Speed (running):   0 GBIT
Port FC ID: 0x000000
Port Type: UNKNOWN
Attention Type:   Link Down
Topology:  UNKNOWN
 
Seconds Since Last Reset: 0
 
        Transmit Statistics     Receive Statistics
        -------------------     ------------------
Frames: 0                       0
Words:  0                       0
 
LIP Count: 0
NOS Count: 0
Error Frames:  0
Dumped Frames: 0
Link Failure Count: 4
Loss of Sync Count: 4
Loss of Signal: 5
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
 
FC SCSI Adapter Driver Information
  No DMA Resource Count: 0
  No Adapter Elements Count: 0
  No Command Resource Count: 0
 
FC SCSI Traffic Statistics
  Input Requests:   0
  Output Requests:  0
  Control Requests: 0
  Input Bytes:  0
  Output Bytes: 0

You can read more about these new options for fcstat in this blog from Yadagiri Rajaboina, from IBM AIX Storage Drivers Development: Displaying the Fibre channel HBA SFP information with AIX/VIOS.

I encourage you to read the command reference guides, and also the “what’s new” links below, to learn more about the latest enhancements for AIX 7.3.

References:

vfcstat Command Reference

fcstat Command Reference

What’s New in Commands – November 2023

What’s New in Commands – November 2022