Maximum value for num_cmd_elems on AIX/VIOS

 

Author: Chris Gibson

Published: 4^th November 2019

 

I’ve received questions, similar to the one below, on more than one occasion, so I thought I’d post something about this for future reference. The question is in relation to setting the maximum value for num_cmd_elems for a fibre channel adapter on AIX (and VIOS).

 

“I have a customer that wants to set their num_cmd_ele on the VIO's as 4096, but every time they set it, they still show 3200. Any ideas?”

 

This is not a new issue and is most likely because their adapter type doesn't support 4096. But AIX allows you to set it to a higher value and then adjusts it back down to the max of 3200 (usually on boot up).

 

The following information (from APAR Technote IV88990) summarises the problem and the expected behaviour for AIX.

 

Error description

 

    In some instances the maximum allowed value of num_cmd_elems

    shown in the ODM for a fibre adapter cannot actually be

    set.  There won't be an error message displayed at the

    time of chdev. However, the user will find that, after

    reconfiguring the adapter (or, after rebooting the host),

    the num_cmd_elems value will be less than what the user

    entered in the chdev command.

 

    AIX Development found that the num_cmd_elems value

    supported by the adapter firmware varied somewhat between

    different adapter models from the same vendor. The

    supported value could also vary for other reasons, such

    as how many ports and PCI functions are configurable on

    the adapter, changes across firmware updates, etc. The

    maximum value of 4096 in the PdAt ODM object class is the

    maximum supported by most of our PCIe2 fibre adapters,

    but the 'actual' supported value may vary at times depending

    on what the adapter firmware reports at the time the

    adapter is configured. In those cases, the 'true' maximum

    supported value will be set after the user attempts a

    chdev operation to change num_cmd_elems to 4096.

 

    This is a limitation of the adapter firmware, and not

    a limitation within the AIX adapter driver.

 

    There are critical reasons NOT to indicate a failure for

    the chdev operation. So, rather than failing the chdev,

    the driver simply sets the 'true' maximum allowed

    num_cmd_elems, and the user will see it after reboot or

    reconfiguration of the adapter.

 

Local fix

 

    N/A

 

Problem summary

 

    Documentation does not have information on

    num_cmd_elems of FC adapter

 

Essentially this is an adapter hardware/firmware resource limitation. It is not a fabric issue. From my understanding this is how some FC adapters parcel out resources in their ASIC between different ports (PCI functions). The only new adapter that reliably allows more than 3200 num_cmd_elems is the 32Gb FC adapter, which allows 6144.

 

For more information, please refer to the following links:

 

IV88990: SOME FIBRE ADAPTERS NUM_CMD_ELEMS CANNOT BE SET TO MAX ODM VALUE

http://www-01.ibm.com/support/docview.wss?uid=isg1IV88990

 

IJ10890: RULES SETTING FOR NUM_CMD_ELEMS ON 16G FC ADAPTER INCORRECT

https://www-01.ibm.com/support/docview.wss?uid=isg1IJ10890