After updating my AIX 7.1 TL3 system to service pack 4, I noticed that each time I started a new ssh session with this system, there was a noticeable delay before the login prompt was displayed. I initially thought there was a network or host name resolution (DNS) problem, but after thoroughly checking related files, such as /etc/hosts, /etc/resolv.conf and /etc/netsvc.conf, I started looking for a problem elsewhere.

I used truss to assist me in my investigation. I found that the each time an ssh client connected to the sshd daemon, sshd would attempt to access a device named /dev/pkcs11. Each time this happened (once per login) there was a significant delay/pause before the ssh session continued to the login prompt. I also noticed that prior to applying SP4, this delay wasn’t present.

I ran truss with the following options (the -d flag provided me with a timestamp for each line of output, and helped my detect the delay!). Immediately after the pkcs11 device was opened, there was a several second delay before the process continued.

# truss -d ssh lpar9 date > cg.out 2>&1

…

0.1253:        kopen("/dev/pkcs11", O_RDONLY)   = 3

4.1939:        kioctl(3, 2, 0x2FF21848, 0x00000000) = 0

# time ssh lpar9 date

Wed Jan 14 15:15:04  2015

real    0m12.34s

user    0m0.02s

sys     0m0.01s

# oslevel -s

7100-03-04-1441

# lslpp -l security.pkcs11

  Fileset                      Level  State      Description

  ----------------------------------------------------------------------------

Path: /usr/lib/objrepos

  security.pkcs11           7.1.3.15  COMMITTED  PKCS11 Libraries

Path: /etc/objrepos

  security.pkcs11           7.1.3.15  COMMITTED  PKCS11 Libraries

I decided to un-install the security.pkcs11 fileset. This solved the issue and my ssh sessions started quickly with the login prompt appearing instantly again.

# installp -u security.pkcs11 -g

...

# time ssh lpar9 date

Wed Jan 14 15:13:56  2015

real    0m0.43s

user    0m0.02s

sys     0m0.01s

I’m still not sure what caused this problem. Prior to SP4, I did not encounter this issue with the security.pkcs11 fileset installed, so I can only assume that there may be some issue with this fileset at the 7.1.3.15 level. Here’s the truss output from a system running a lower level of security.pkcs11 (no delay).

0.1248:        kopen("/dev/pkcs11", O_RDONLY)   = 3

0.1320:        kioctl(3, 2, 0x2FF21848, 0x00000000) = 0

I also found some advice that suggested placing ‘UsePKCS no’ in the /etc/ssh/sshd_config file, but this did not help me in the tests that I conducted.

It was safe for me to remove this fileset as I was not using it for any purpose. Typically, this fileset is required when using special crypto cards in POWER servers.

IBM 4758 Model 2 Cryptographic Coprocessor

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.security/ibm_crypt_proc.htm

Public Key Cryptography Standards #11

http://www-01.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.security/pkcs_over.htm

Hopefully this will help others that may encounter this problem on their AIX systems.