SSH delay with security.pkcs11 installed.
After updating my AIX 7.1 TL3 system to service pack 4, I noticed that each time I started a new ssh session with this system, there was a noticeable delay before the login prompt was displayed. I initially thought there was a network or host name resolution (DNS) problem, but after thoroughly checking related files, such as /etc/hosts, /etc/resolv.conf and /etc/netsvc.conf, I started looking for a problem elsewhere.
I used truss to assist me in my investigation. I found that the each time an ssh client connected to the sshd daemon, sshd would attempt to access a device named /dev/pkcs11. Each time this happened (once per login) there was a significant delay/pause before the ssh session continued to the login prompt. I also noticed that prior to applying SP4, this delay wasn’t present.
I ran truss with the following options (the -d flag provided me with a timestamp for each line of output, and helped my detect the delay!). Immediately after the pkcs11 device was opened, there was a several second delay before the process continued.
# truss -d ssh lpar9 date > cg.out 2>&1 …
0.1253: kope 4.1939: kioctl(3, 2, 0x2FF21848, 0x00000000) = 0
# time ssh lpar9 date Wed Jan 14 15:15:04 2015
real 0m12.34s user 0m0.02s sys 0m0.01s
# oslevel -s 7100-03-04-1441
# lslpp -l security.pkcs11
File
---- Path: /usr/lib/objrepos
secu
Path: /etc/objrepos
secu
I decided to un-install the security.pkcs11 fileset. This solved the issue and my ssh sessions started quickly with the login prompt appearing instantly again.
# installp -u security.pkcs11 -g ... # time ssh lpar9 date Wed Jan 14 15:13:56 2015
real 0m0.43s user 0m0.02s sys 0m0.01s
I’m still not sure what caused this problem. Prior to SP4, I did not encounter this issue with the security.pkcs11 fileset installed, so I can only assume that there may be some issue with this fileset at the 7.1.3.15 level. Here’s the truss output from a system running a lower level of security.pkcs11 (no delay).
0.1248: kope 0.1320: kioctl(3, 2, 0x2FF21848, 0x00000000) = 0
I also found some advice that suggested placing ‘UsePKCS no’ in the /etc
It was safe for me to remove this fileset as I was not using it for any purpose. Typically, this fileset is required when using special crypto cards in POWER servers.
IBM 4758 Model 2 Cryptographic Coprocessor
http
Public Key Cryptography Standards #11
http
Hopefully this will help others that may encounter this problem on their AIX systems. |
Thanks for sharing
I had the same problem with AIX 7.1 TL04 SP02.
But in the truss log I do not have 'kopen("/dev/pkcs11'.
Anyway, the remove of the fileset security.pkcs11 resolved my issue.
@ZlatkoAIX
I can not remove pkcs11 device :
rmdev -l pkcs11
Method error (/usr/lib/methods/ucfgdevice):
0514-062 Cannot perform the requested function because the specified device is busy.
Thanks for providing this article.
I had the same problem with 7100-03-05-1524, and deinstall of security.pkcs11 solved the issue immediately.
Had
the same issue also at 7100-03-05-1524, removed the fileset
security.pkcs11 and normality (whatever that is) was restored. Thanks!
Now, is IBM going to fix this...? I can't see anything that mentions it yet.
I had the same issue after updating to 7100-03-05-1524.
Many
thanks, I did pretty much exact same troubleshooting and this blog was
the sole result google gave me for the offending 'kopen("/dev/pkcs11",
O_RDONLY)'. I had to remove the fileset to ultimately resolve it.
Thanks for sharing... I had the exact same problem and everything returned to normal once I uninstalled the problematic fileset.
Another workaround: # rmdev -l pkcs11 pkcs11 Defined The speed is back to normal.