Make sure you are monitoring file system space on your VIOS.

 

Why?

 

If you run out of space in the root file system, odd things can happen when you try to map virtual devices to virtual adapters with mkvdev.

 

For example, a colleague of mine was attempting to map a new hdisk to a vhost adapter on a pair of VIOS. The VIOS was running a recent version of code. He received the following error message (see below). It wasn’t a very helpful message. At first I thought it was due to the fact that he had not set the reserve_policy attribute for the new disk to no_reserve on both VIOS. Changing the value for that attribute did not help.

 

$ ioslevel

2.2.1.3

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I enabled debug output from the padmin command line and discovered that the mkdev command was failing for some reason.

 

$ export CLI_DEBUG=33

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

*******************************************************************************

The command's response was not recognized.  This may or may not indicate a problem.

*******************************************************************************

 

I entered oem_setup_env and ran the mkdev command manually. I was presented with a very clear error message describing the problem. The root file system (/) was full!

 

$ oem_setup_env

 

# mkdev  -V hdisk1  -p vhost0  -l vhdisk1

mkdev: 0514-548 Cannot perform the requested function because

                the / filesystem is full or is out of inodes.

 

# df -g /

Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4           0.50      0.00  100%     7728    75% /

 

I found a 45MB core file (generated by cimserver) in the root file system. I moved the file to another file system (with plenty of free space) for further investigation.

 

# mv core /tmp/core.cg

# df -m .

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4         512.00     45.93   92%     7727    38% /

 

Then I ran the mkvdev command successfully.

 

$ mkvdev -vdev hdisk1 -vadapter vhost0 -dev vhdisk1

AIX: "lspv -l hdisk1 2>&1 | grep 0516-320"

AIX: "mkdev  -V hdisk1  -p vhost0  -l vhdisk1 "

vhdisk1 Available

AIX: "lsattr -El hdisk1 -a max_transfer |cut -d" " -f2"

 

$ lsmap -all

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name" | wc -l -c"

AIX: "lsdev -c adapter -t IBM,v-scsi-host -s vdevice -F "name""

AIX: "lsdev -C -l vhost0 -F "physloc""

AIX: "lsdev -p vhost0 -F "name" | wc -l -c"

AIX: "lsdev -p vhost0 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost0          U9119.FHA.87654B1-V6-C21                   0x0000002e

 

VTD                   vhdisk1

Status                Available

LUN                   0x8100000000000000

Backing device        hdisk1

Physloc               U5873.001.9SS005Y-P1-C9-T1-W50060E8006D0206A-L1000000000000

Mirrored              false

 

AIX: "lsdev -C -l vhost1 -F "physloc""

AIX: "lsdev -p vhost1 -F "name" | wc -l -c"

AIX: "lsdev -p vhost1 -F "name""

SVSA            Physloc                                      Client Partition ID

--------------- -------------------------------------------- ------------------

vhost1          U9119.FHA.87654B1-V6-C22                   0x0000002g

 

VTD                   NO VIRTUAL TARGET DEVICE FOUND

 

I found the same issue on the second VIOS i.e. a full root file system due to a core file (from cimserver). I also found no trace of a full file system event in the error report. Perhaps someone had taken it upon themselves to “clean house” at some point and had removed entries from the VIOS error log.

 

Make sure you monitor file system space on your VIOS. Who knows what else might fail if you run out of space in a critical file system.

Tags:  mkvdev vios aix the_command's_response_wa... this_may_or_may_not_indic... 0514-548 cli_debug=33 chris_gibson powervm root_filesystem_is_full