Maximum value for num_cmd_elems on AIX/VIOS
Author: Chris Gibson
Published: 4th November 2019
I’ve received questions, similar to the one below, on more than one occasion, so I thought I’d post something about this for future reference. The question is in relation to setting the maximum value for num_cmd_elems for a fibre channel adapter on AIX (and VIOS).
“I have a customer that wants to set their num_cmd_ele on the VIO's as 4096, but every time they set it, they still show 3200. Any ideas?”
This is not a new issue and is most likely because their adapter type doesn't support 4096. But AIX allows you to set it to a higher value and then adjusts it back down to the max of 3200 (usually on boot up).
The following information (from APAR Technote IV88990) summarises the problem and the expected behaviour for AIX.
In some instances the maximum allowed value of num_cmd_elems
shown in the ODM for a fibre adapter cannot actually be
set. There won't be an error message displayed at the
time of chdev. However, the user will find that, after
reconfiguring the adapter (or, after rebooting the host),
the num_cmd_elems value will be less than what the user
entered in the chdev command.
AIX Development found that the num_cmd_elems value
supported by the adapter firmware varied somewhat between
different adapter models from the same vendor. The
supported value could also vary for other reasons, such
as how many ports and PCI functions are configurable on
the adapter, changes across firmware updates, etc. The
maximum value of 4096 in the PdAt ODM object class is the
maximum supported by most of our PCIe2 fibre adapters,
but the 'actual' supported value may vary at times depending
on what the adapter firmware reports at the time the
adapter is configured. In those cases, the 'true' maximum
supported value will be set after the user attempts a
chdev operation to change num_cmd_elems to 4096.
This is a limitation of the adapter firmware, and not
a limitation within the AIX adapter driver.
There are critical reasons NOT to indicate a failure for
the chdev operation. So, rather than failing the chdev,
the driver simply sets the 'true' maximum allowed
num_cmd_elems, and the user will see it after reboot or
reconfiguration of the adapter.
Documentation does not have information on
num_cmd_elems of FC adapter
Essentially this is an adapter hardware/firmware resource limitation. It is not a fabric issue. From my understanding this is how some FC adapters parcel out resources in their ASIC between different ports (PCI functions). The only new adapter that reliably allows more than 3200 num_cmd_elems is the 32Gb FC adapter, which allows 6144.
For more information, please refer to the following links:
IV88990: SOME FIBRE ADAPTERS NUM_CMD_ELEMS CANNOT BE SET TO MAX ODM VALUE
IJ10890: RULES SETTING FOR NUM_CMD_ELEMS ON 16G FC ADAPTER INCORRECT