Finding out the actual value of num_cmd_elems of a Fiber Channel adapter using kdb on AIX….in 2022

Way back in 2011, 2012 and 2013, I shared how you could use the AIX kdb (and lsattr) commands to determine the running value of a device attribute against the value stored in the AIX ODM. Here are the posts I’m alluding to:

As you can imagine, a decade or so later, things have changed a little!

I recently received a query from a colleague that referenced some of my older material. Here’s the email:

From: IBMer

Date: Wednesday, 13 July 2022 at 6:46 am

To: Chris Gibson

Subject: Re: Finding out the actual value of num_cmd_elems of a Fiber adapter using kdb

Here’s an older post I saw from Chris Gibson.

http://gibsonnet.net/blog/dwarchive/kdb%20and%20fscsi%20attributes%20(Chris%27s%20AIX%20Blog).html

He does note - “The following relies on internal structures that are subject to change”

Chris, would you happen to have this for AIX v7.3?

#oslevel -s

7300-00-02-2220

(0)> efscsi fscsi0

Unable to find <efsc_scsi_ptrs>

I replied to the email with my thoughts and advice. This prompted me to write a new blog post and provide some of my updated tips in this area. Below is my reply to the query.

Hi IBMer, I’m assuming VFC/NPIV here.

FYI, AIX development introduced a hard-coded "floor" of 512 for the adapter num_cmd_elems, starting with AIX 7.2 TL5 and higher (as part of the NPIV scaling improvements through Multiple-Queue). The ODM allows a lower value to be set, but at configuration time it is set to 512 if the ODM setting is less. Note however that this is only for the adapter layer (fcsX), at the protocol (fscsi) layer the commands are throttled to the ODM setting. So, you will find that the high-water mark (fcstat -D) is limited to the value set for the fcsX adapter in the ODM and, thus, from the SAN point of view the host is still throttled to the specified value. So, yes, things have changed!

https://www.ibm.com/docs/en/power9/9080-M9S?topic=channel-npiv-multiple-queue-support

Having said that….

In the examples below, num_cmd_elems is set to 400 and 1024 (in the ODM) respectively. kdb reports num_cmd_elems as 380 and 1004, less 20 reserved for special internal use, by the kernel.

Note: The device driver of VFC adapters was changed from efscsi to emfscsi starting with 7200-05.

# odmget -q name=fscsi0 CuDv

CuDv:

name = "fscsi0"

status = 1

chgstatus = 2

ddins = "emfscsidd"

location = "C4-T1-01"

parent = "fcs0"

connwhere = "1"

PdDvLn = "driver/vio_npiv/emfscsi"

# genkex | grep fscsi

f10009d5a01e5000 f8000 /usr/lib/drivers/emfscsidd

# oslevel -s

7200-05-04-2220

# echo emfscsi -d fscsi0 | kdb -script | grep num_cmd

int num_cmd_elems = 0x17C

# printf "%d\n" 0x17C

380

# lsattr -El fcs0 -a num_cmd_elems

num_cmd_elems 400 Maximum Number of COMMAND Elements True

# oslevel -s

7300-00-02-2220

# echo emfscsi -d fscsi0 | kdb -script | grep num_cmd | grep -v 0x0

int num_cmd_elems = 0x3EC

# printf "%d\n" 0x3EC

1004

# lsattr -El fcs0 -a num_cmd_elems

num_cmd_elems 1024 Maximum Number of COMMAND Elements True

If the intent here is to confirm the "running value” versus the “ODM configured value” for num_cmd_elems, an alternative could be to use the "lsattr -Pl fcsX" command.

# chdev -l fcs0 -a num_cmd_elems=1024 -P

fcs0 changed

# lsattr -El fcs0 -a num_cmd_elems

num_cmd_elems 1024 Maximum Number of COMMAND Elements True << ODM

# lsattr -Pl fcs0 -a num_cmd_elems

num_cmd_elems 400 Maximum Number of COMMAND Elements True << Running

http://gibsonnet.net/blog/cgaix/html/Thanks%20kdb%20but%20lsattr's%20got%20me%20covered!.html

;Additionally on 7.3 (only?), another way to check the configured and running value...

# grep num_cmd /proc/sys/adapter/fc/fcs0/tunables

num_cmd_elems ( Requested / Granted ): 1024 / 1024

I hope this helps.

Also, while I have your attention, please take a quick look at this recent APAR in relation to NPIV & Multi-Queue performance, https://www.ibm.com/support/pages/apar/IJ40942. And check out one of my more recent posts about some of the goodies you can find in /proc/sys/adapter/fc/fcsX/hostinfo and /proc/sys/adapter/fc/fcsX/ on AIX 7.2 TL5 SP3 and 7.3.

http://gibsonnet.net/blog/cgaix/html/lsviofcs.html