Whilst working with one of my AIX customers recently I discovered a problem with NIMSH and SSL. The customer had updated their NIM master from AIX 7.1 TL4 SP1 to AIX 7.1 TL4 SP3. After the SP update, any attempt to connect to a NIM client (over NIMSH+SSL), from the NIM master, would simply hang. For example, we tried to list the filesets on the NIM client with this command, which never returned any output.


[root@750lpar4]/ # nim -o lslpp 750lpar9


The /var/adm/ras/nimsh.log file, on the NIM client, showed that the NIMSH session stopped here:


Thu Jan 12 14:31:49 2017        Loading certificates..

Thu Jan 12 14:31:49 2017        Loading private key file..

Thu Jan 12 14:31:49 2017        create BIO


NIM master: 750lpar4



NIM client: 750lpar9



[root@750lpar4]/ # lsnim -l 750lpar9


   class          = machines

   type           = standalone

   connect        = nimsh (secure)

   platform       = chrp

   netboot_kernel = 64

   if1            = 10_1_50 750lpar9 0

   cable_type1    = N/A

   Cstate         = ready for a NIM operation

   prev_state     = not running

   Mstate         = currently running

   cpuid          = 00F603CD4C00

   Cstate_result  = success


The root cause of the problem become apparent when we ran truss against the nim –o command.


[root@750lpar4]/ # truss -adef -o truss.lsnim.out -w all nim -o lslpp 750lpar9


[root@750lpar4]/ # cat truss.lsnim.out

13959372: C o u l d   n o t   l o a d   m o d u l e   / u s r / l i b / l

13959372: i b s s l . s o .\n S y s t e m   e r r o r :   N o   s u c h

13959372: f i l e   o r   d i r e c t o r y

19267612: C o u l d   n o t   l o a d   m o d u l e   / u s r / l i b / l

19267612: i b c r y p t o . s o .\n S y s t e m   e r r o r :   N o   s u

19267612: c h   f i l e   o r   d i r e c t o r y


The required shared library object files were missing on the NIM master.


[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

libssl.so not found

libcrypto.so not found


We fixed this issue by extracting the missing files from the (existing) /usr/lib/libssl.a and /usr/lib/libcrypto.a archives.


[root@750lpar4]/usr/lib # slibclean


[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libssl.a /usr/lib/libssl.so

x - /usr/lib/libssl.so


[root@750lpar4]/usr/lib # /bin/ar -v -x /usr/lib/libcrypto.a /usr/lib/libcrypto.so

x - /usr/lib/libcrypto.so


[root@750lpar4]/usr/lib # ls -ltr libssl.so libcrypto.so

-rwxr-xr-x    1 root     system       724913 Jan 18 09:08 libssl.so

-rwxr-xr-x    1 root     system      3031337 Jan 18 09:08 libcrypto.so


After that, the nim –o commands started working again.


[root@750lpar4]/usr/lib # nim -o showlog 750lpar9



So, the question is why did this happen? Well, in the past, the libssl.so.0.9.8 shared object was extracted by NIM, but more recent updates by the OpenSSL version has forced IBM to move to libssl.so. Usually, the extracted shared library object is added (if not currently present) when nimconfig -c is run.  But given that this is an existing NIM master, we did not want to run this again (as we would lose all of the current SSL key access). So extracting the objects is preferred. The problem is due to the fact that the libssl.so and libcrypto.so files are not populated when the AIX 7100-04-03 update is applied. This is a bug and will be officially addressed, soon, under APAR IV93152 NIM push operation to client hang on nimsh over SSL.


I believe this issue may also occur when you migrate your NIM master from AIX 7.1 to 7.2 (with nimadm for example). But I need to do more testing to reproduce and confirm the issue.


Here’s one good reason to setup NIMSH over SSL.





The following link is a great reference guide for configuring NIMSH over SSL.