AIX Virtual Fibre Channel
Enhancement
In this post I want to discuss two topics. First, I want to tell you all about a new AIX 7.3 command called vfcstat. Second, I want to show some of the (not so recent) enhancements to the fcstat command.
New vfcstat command
A new Virtual Fibre Channel (VFC) client adapter enhancement was introduced with the release of AIX 7.3 Technology Level 2 Service Pack 1 (November 2023). A new command, called vfcstat, was introduced. This allows administrators to access information for VFC adapters, such as statistics, status, and activities.
Prior to this command, administrators could use the cat command to view the information stored in the /proc file system (in /proc/sys/adapter/fc/fcsX). The vfcstat command makes it easier for admins to view this information (without needing to know the location of the various files in /proc).
This command is available for AIX 7.3 systems that are Virtual I/O (VIO) clients, of VIOS running version 3.1.4.x and later (VIOS 3.1.4 provides support for NPIV multi-queue, https://www.ibm.com/support/knowledgecenter/POWER9/p9hb1/p9hb1_npiv_multi_queue_support.htm).
From the AIX 7.3 command reference for vfcstat:
“The vfcstat
command reports information such as virtual FC adapter state, status, NPort ID,
I/O queues information, read/write statistics, binding information with the
host (VIOS), link errors, path timeouts, and so on. The required information is
fetched in real time and displayed in the output. You can reset some of the
details such as statistics and error counts. You can also retrieve read/write
statistics in formatted output for scripting purposes by using the -F option. The vfcstat command displays
the statistics in a specified time interval for the specific number of times.
The vfcstat
command works on virtual FC client adapter only.”
The new command is delivered with the devices.vdevice.IBM.vfc-client.rte fileset.
# lslpp
-w /usr/bin/vfcstat
File
Fileset Type
----------------------------------------------------------------------------
/usr/bin/vfcstat
devices.vdevice.IBM.vfc-client.rte
Symlink
# ls
-ltr /usr/bin/vfcstat
lrwxrwxrwx 1 bin
bin 17 Oct 23 18:23
/usr/bin/vfcstat -> /usr/sbin/vfcstat
# lslpp
-w /usr/sbin/vfcstat
File
Fileset Type
----------------------------------------------------------------------------
/usr/sbin/vfcstat
devices.vdevice.IBM.vfc-client.rte
File
# lslpp
-lc devices.vdevice.IBM.vfc-client.rte
#Fileset:Level:PTF
Id:State:Type:Description:EFIX Locked
/usr/lib/objrepos:devices.vdevice.IBM.vfc-client.rte:7.3.2.1::COMMITTED:F:Virtual
Fibre Channel Client Support:
/etc/objrepos:devices.vdevice.IBM.vfc-client.rte:7.3.2.1::COMMITTED:F:Virtual
Fibre Channel Client Support:
I tested the vfcstat command on an AIX system running 7300-02-01-2346.
You can use the command to list all the VFC adapters on your AIX system.
# vfcstat
-l
fcs0
fcs1
To list all the functions available for the given VFC adapter, run the following:
# vfcstat
-d fcs0
activity
capabilities
hostinfo
link
nport_id
status
tunables
wwpn
To display error counts for a given VFC adapter.
# vfcstat
-d fcs0 -e
Current
time : Jan 22 2024,
22:00:52.294
Time
since last reset : 5885149 sec
( 2 months 8 days 2 hours 45 minutes 49 seconds )
Link
Errors:
Total errors: 0
Last 10 minutes: 0
Last 60 minutes: 0
Last 24 hours: 0
Path
Timeouts:
Domain: Admin
Total timeouts: 0
Last 10 minutes: 0
Last 60 minutes: 0
Last 24 hours: 0
Path
Timeouts:
Domain: SCSI
Total timeouts: 0
Last 10 minutes: 0
Last 60 minutes: 0
Last 24 hours: 0
Path
Timeouts:
Domain: NVMeOF
Total timeouts: 0
Last 10 minutes: 0
Last 60 minutes: 0
Last 24 hours: 0
Note there are three domains for VFC adapter elements. One is for VFC Admin commands, the second is for SCSI commands and the third is for NVME commands.
Use the -s flag to display
various statistics for a given VFC adapter. This is same as reading the proc
file /proc/sys/adapter/fc/fcs0/stats. We’ve included the –F option as it displays formatted
statistic output (which is easier to read!).
# vfcstat
-d fcs0 -s -F
Current
time=Jan 22 2024, 22:38:58.102
Time
since last reset=5887435 sec ( 2 months 8 days 3 hours 23 minutes 55 seconds )
Queue_Start
Domain=SCSI
Queue
number=0
Read
reqs=9814
Read
bytes=196898541
Write
reqs=2696118
Write
bytes=253070213192
No
DMA res count=0
No
SGE count=0
h_send_dropped=0
h_send_closed=0
h_send_busy=0
No
cmd res count=0
Queue_End
To display tunables of a given VFC adapter, use the -d and -f tunables options. This is same as reading proc
file /proc/sys/adapter/fc/fcs0/tunables.
# vfcstat
-d fcs0 -f tunables
lg_term_dma ( Requested / Granted ): 0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size ( Requested / Granted ): 0x100000 (1M) / 0x100000 (1M)
num_cmd_elems ( Requested / Granted ): 2048 / 2048
num_sp_cmd_elem
( Requested / Granted ): 1024 / 1024
num_io_queues ( Requested / Granted ): 8 / 0
num_nvme_queues
( Requested / Granted ): 4 / 0
To display VIOS host information of a given VFC adapter, use the -f hostinfo option. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/hostinfo.
# vfcstat
-d fcs0 -f hostinfo
VFC
client adapter name :
fcs0
Host
partition name (VIOS) : vios1
VFC
host adapter name :
vfchost0
VFC
host adapter location code :
U8286.42A.2153B1V-V1-C30
FC
adapter name on VIOS :
fcs0
FC
adapter location code on VIOS : U78C9.001.WZS00XE-P1-C1-T1
To display WWPN information of a given VFC adapter, use the -f wwpn option. This is same as reading the proc file /proc/sys/adapter/fc/fcs0/wwpn.
# vfcstat
-d fcs0 -f wwpn
0xc852100000000011
To reset all fields in the stats function for a given VFC adapter.
#
vfcstat -d fcs0 -s –z
To reset all error counts for a given VFC adapter.
# vfcstat
-d fcs0 -e –z
Here's a little script I wrote to collect and display output from each of the available functions for all VFC adapters on a system:
|
myvfcs=$(/usr/bin/vfcstat
-l) vfcsfuncs=$(echo
$myvfcs | awk '{print $1}' | xargs /usr/bin/vfcstat -d) echo
Displaying functions that are available for a virtual FC client adapter:
$myvfcs echo echo
Functions available for $myvfcs are: echo
$vfcsfuncs echo echo for
xvfcs in $myvfcs do for eachvfunc in $vfcsfuncs do echo "VFCSTAT $eachvfunc for
$xvfcs" | xargs /usr/local/bin/box /usr/bin/vfcstat -d $xvfcs -f $eachvfunc echo done done |
The output is:
# ./xvfcs
Displaying
functions that are available for a virtual FC client adapter: fcs0 fcs1
Functions
available for fcs0 fcs1 are:
activity
capabilities hostinfo link nport_id status tunables wwpn
#############################
#
VFCSTAT activity for fcs0 #
#############################
Admin
Queue
Admin
I/O Queue
#################################
#
VFCSTAT capabilities for fcs0 #
#################################
Requested
capabilities : CAN_MIGRATE,
CAN_HANDLE_FPIN
Host
responded capabilities :
#############################
#
VFCSTAT hostinfo for fcs0 #
#############################
VFC
client adapter name : fcs0
Host
partition name (VIOS) : vios1
VFC
host adapter name :
vfchost0
VFC
host adapter location code :
U8286.42A.2153B1V-V1-C30
FC
adapter name on VIOS :
fcs0
FC
adapter location code on VIOS : U78C9.001.WZS00YD-P1-C7-T1
#########################
#
VFCSTAT link for fcs0 #
#########################
Link UP
topology FABRIC
speed 8Gb
#############################
#
VFCSTAT nport_id for fcs0 #
#############################
0xe016a0
###########################
#
VFCSTAT status for fcs0 #
###########################
State 0x10 (ONLINE)
Opened true
cfg_opn_state 0x0
prev_cfg_channels 0
flags 0x200082 ( CHANNELS_NOT_SUPPORTED,
LINK_SETUP, NONVM_FROM_CFG )
total_io_dma_size 0x20000000
total_dma_size_used 0x9900000
no_chs_reason_code 0x2 ( NO_PROCESSOR_COMPATABILITY )
migrated_count 0
online_time 373 (lbolt)
UP
time 5978367 sec ( 2
months 9 days 4 hours 39 minutes 27 seconds )
Admin
Queue
active_cmds 0
spl_active_cmds 0
pending_cmds 0
spl_pending_cmds 0
resp_cmds 0
Admin
I/O Queue
active_cmds 0
spl_active_cmds 0
pending_cmds 0
spl_pending_cmds 0
waiting_cmds 0
resp_cmds 0
offlevel_resp_cmds 0
cancel_resp_cmds 0
#############################
#
VFCSTAT tunables for fcs0 #
#############################
lg_term_dma ( Requested / Granted ): 0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size ( Requested / Granted ): 0x100000 (1M) / 0x100000 (1M)
num_cmd_elems ( Requested / Granted ): 2048 / 2048
num_sp_cmd_elem
( Requested / Granted ): 1024 / 1024
num_io_queues ( Requested / Granted ): 8 / 0
num_nvme_queues
( Requested / Granted ): 4 / 0
#########################
#
VFCSTAT wwpn for fcs0 #
#########################
0xc852100000000011
#############################
#
VFCSTAT activity for fcs1 #
#############################
Admin
Queue
Admin
I/O Queue
#################################
#
VFCSTAT capabilities for fcs1 #
#################################
Requested
capabilities : CAN_MIGRATE,
CAN_HANDLE_FPIN
Host
responded capabilities :
#############################
#
VFCSTAT hostinfo for fcs1 #
#############################
VFC
client adapter name :
fcs1
Host
partition name (VIOS) : vios2
VFC
host adapter name :
vfchost0
VFC
host adapter location code :
U8286.42A.2153B1V-V2-C30
FC
adapter name on VIOS :
fcs0
FC
adapter location code on VIOS : U78C9.001.WZS00XE-P1-C1-T1
#########################
#
VFCSTAT link for fcs1 #
#########################
Link UP
topology FABRIC
speed 8Gb
#############################
#
VFCSTAT nport_id for fcs1 #
#############################
0xe016c0
###########################
#
VFCSTAT status for fcs1 #
###########################
State 0x10 (ONLINE)
Opened true
cfg_opn_state 0x0
prev_cfg_channels 0
flags 0x200082 (
CHANNELS_NOT_SUPPORTED, LINK_SETUP, NONVM_FROM_CFG )
total_io_dma_size 0x20000000
total_dma_size_used 0x9900000
no_chs_reason_code 0x2 ( NO_PROCESSOR_COMPATABILITY )
migrated_count 0
online_time 373 (lbolt)
UP
time 5978367 sec ( 2
months 9 days 4 hours 39 minutes 27 seconds )
Admin
Queue
active_cmds 0
spl_active_cmds 0
pending_cmds 0
spl_pending_cmds 0
resp_cmds 0
Admin
I/O Queue
active_cmds 0
spl_active_cmds 0
pending_cmds 0
spl_pending_cmds 0
waiting_cmds 0
resp_cmds 0
offlevel_resp_cmds 0
cancel_resp_cmds 0
#############################
#
VFCSTAT tunables for fcs1 #
#############################
lg_term_dma ( Requested / Granted ): 0x1000000 (16M) / 0x1000000 (16M)
max_xfer_size ( Requested / Granted ): 0x100000 (1M) / 0x100000 (1M)
num_cmd_elems ( Requested / Granted ): 2048 / 2048
num_sp_cmd_elem
( Requested / Granted ): 1024 / 1024
num_io_queues ( Requested / Granted ): 8 / 0
num_nvme_queues
( Requested / Granted ): 4 / 0
#########################
#
VFCSTAT wwpn for fcs1 #
#########################
0xc852100000000013
Enhancements to fcstat
In a somewhat related set of enhancements, the fcstat command was also updated with several new options that provide additional information to the administrator. You may or may not be aware, but these new options have been available for some time with AIX 7.3. The new options are:
· -p Displays a time-series report of traffic statistics for a specific transport protocol (TP) that is specified with the Protocol parameter.
· -t Displays a time-series report of the traffic statistics continuously with a time interval between two consecutive reports, which is specified with the Interval parameter.
· -e Displays all the statistics, that includes the device-specific statistics such as driver statistics, link statistics, FC4 types, and vital Small Form-factor Pluggable (SFP) parameters.
· -d Displays the statistics by opening the adapter in diagnostic mode.
Here’s an example of using the -t and -p options. In the example below we have chosen to view the SCSI protocol statistics for the fcs0 adapter and to display the time-series report of traffic statistics with a one second interval.
# fcstat
-t 1 -p scsi fcs0
FIBRE
CHANNEL STATISTICS REPORT: fcs0
Device
Type: Virtual Fibre Channel Client Adapter (adapter/vdevice/IBM,vfc-client)
TP Read Reqs(K) Write Reqs(K) Read (GB)
Write (GB)
=====
================= ================= =========== ===========
SCSI 9 2314 0 203
SCSI 9 2315 0 203
SCSI 9 2316 0 203
SCSI 9 2317 0 203
SCSI 9 2318 0 203
SCSI 9 2319 0 203
SCSI 9 2320 0 203
SCSI 9 2322 0 204
SCSI 9 2323 0 204
SCSI 9 2324 0 204
SCSI 9 2326 0 204
SCSI 9 2328 0 204
SCSI 9 2329 0 205
SCSI 9 2330 0 205
SCSI 9 2332 0 205
SCSI 9 2334 0 205
SCSI 9 2335 0 205
SCSI 9 2336 0 206
SCSI 9 2338 0 206
SCSI 9 2340 0 206
SCSI 9 2341 0 206
SCSI 9 2342 0 206
SCSI 9 2344 0 207
SCSI 9 2346 0 207
SCSI 9 2347 0 207
SCSI 9 2348 0 207
SCSI 9 2350 0 207
SCSI 9 2352 0 208
SCSI 9 2353 0 208
SCSI 9 2354 0 208
In the output above the time-series report shows the following statistics:
· Read Reqs(K): The number of read or input requests, in thousands.
· Write Reqs(K): The number of write or output requests, in thousands.
· Read(GB): The number of input bytes that are read, in gigabytes.
· Write(GB): The number of output bytes that are written, in gigabytes.
If you do not have any NVME adapters/devices on your AIX system (or you are only interested in FC SCSI traffic), you can remove the -p flag and shorten the command to, fcstat -t 1 fcs0.
The -e option displays vital Small Form-factor Pluggable (SFP) parameters such as vendor information, temperature, voltage information, as well as, the device-specific statistics such as driver statistics, link statistics, and FC4 protocol types.
Previously, some of this information was only available using a special tool (called emfc_power) provided by IBM support. This option is useful with physical (not virtual) fibre channel adapters. It appears this enhancement may have been delivered based on this customer enhancement request, https://ibm-power-systems.ideas.ibm.com/ideas/AIX-I-688.
# fcstat
-e fcs0
FIBRE
CHANNEL STATISTICS REPORT: fcs0
Device
Type: PCIe3 2-Port 32Gb FC Adapter (df1000e314101506)
(adapter/pciex/df1000e31410150)
...etc...
FC
SFP Information
Vendor Name: AVAGO
Vendor OUI:
0000176a
Vendor PN:
AFBR-57G5MZ-ELX
Temperature: 42.371 C [Range -128 C - +128 C]
Voltage:
3.331 V [Range 0 V - +6.55 V]
TX Bias:
5.466 mA [Range 0 mA - 131 mA]
TX Power:
-1.4837 dBm [Range -40 dBm -
+8.2 dBm]
0.7106 mW [Range 0 mW - 6.5535 mW]
RX Power:
-3.6866 dBm [Range -40 dBm -
+8.2 dBm]
0.4279 mW [Range 0 mW - 6.5535 mW ]
The fcstat command opens the device in diagnostic mode by using the -d flag, as shown below. When the link is down and the device is opened in nondiagnostic mode, the fcstat command delays in generating the output. You can use the -c flag to remove this delay. If the device is already opened and the fcstat command is started with the -d flag, the open operation on the device fails with an EACCESS error. Meaning you can now run fcstat on an adapter, with a link that is down, and still obtain some useful information relating to link status, failure, and loss of sync/signal counts.
# fcstat
-d fcs0
FIBRE
CHANNEL STATISTICS REPORT: fcs0
Device
Type: 8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
(adapter/pciex/df1000f114108a0)
Serial
Number: 1A202029F7
Option
ROM Version: 0278213D
ZA:
U2D2.10X13
World
Wide Node Name: 0x20000000C9D55D1E
World
Wide Port Name: 0x10000000C9D55D1E
FC-4
TYPES:
Supported:
0x0000012000000000000000000000000000000000000000000000000000000000
Active:
0x0000010000000000000000000000000000000000000000000000000000000000
Class
of Service: 3
Port
Speed (supported): 8 GBIT
Port
Speed (running): 0 GBIT
Port
FC ID: 0x000000
Port
Type: UNKNOWN
Attention
Type: Link Down
Topology: UNKNOWN
Seconds
Since Last Reset: 0
Transmit Statistics Receive Statistics
------------------- ------------------
Frames:
0 0
Words: 0 0
LIP
Count: 0
NOS
Count: 0
Error
Frames: 0
Dumped
Frames: 0
Link
Failure Count: 4
Loss
of Sync Count: 4
Loss
of Signal: 5
Primitive
Seq Protocol Error Count: 0
Invalid
Tx Word Count: 0
Invalid
CRC Count: 0
FC
SCSI Adapter Driver Information
No DMA Resource Count: 0
No Adapter Elements Count: 0
No Command Resource Count: 0
FC
SCSI Traffic Statistics
Input Requests: 0
Output Requests: 0
Control Requests: 0
Input Bytes:
0
Output Bytes: 0
You can read more about these new options for fcstat in this blog from Yadagiri Rajaboina from IBM AIX Storage Drivers Development.
Displaying the Fibre channel HBA SFP information with
AIX/VIOS
I encourage you to read the command reference guides, and also the “What’s new” links below, to learn more about the latest enhancements.
References:
vfcstat Command Reference
https://www.ibm.com/docs/en/aix/7.3?topic=v-vfcstat-command
fcstat Command Reference
https://www.ibm.com/docs/en/aix/7.3?topic=f-fcstat-command
What's new in Commands - November 2023
https://www.ibm.com/docs/en/aix/7.3?topic=commands-whats-new#commands_whats_new__title__3
What's new in Commands - November 2022
https://www.ibm.com/docs/en/aix/7.3?topic=commands-whats-new#commands_whats_new__title__4