_files/genppageinit.gif){kind=small}
_files/ibm-comm-wordmark.png){kind=small}
_files/blank.gif){kind=spacer}
_files/iconWarning.gif "IC Sunset")
The
developerWorks Connections platform will be sunset on December 31,
2019. On January 1, 2020, this blog will no longer be available. More details available on our FAQ.
SSH Host keys and AIX 7.1 Migration.Recent releases of AIX installation media (for 7.1 and 6.1) now contain the OpenSSH base installation filesets. This is very handy; we no longer need to download or locate the software from other sources.
One thing to consider is what this means for future AIX migrations.
If you are migrating a system (that already has a version of SSH installed) to AIX 7.1 then you may notice that the first time you attempt to connect to the server (after the 7.1 migration) the following ssh message appears:
root@nim1 : / # ssh aixlpar1 @@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is 59:6 Please contact your system administrator. Add correct host key in /.ssh/known_hosts to get rid of this message. Offending
RSA key in /.ss RSA host key for aixlpar1 has changed and you have requested strict checking. Host key verification failed.
In the output above I’m attempting to SSH from another system to the newly migrated AIX 7.1 LPAR. This is essentially informing us that the SSH host keys on the AIX 7.1 server don’t match the host key stored in the local systems /.ssh/known_hosts file. Something has changed.
Now of course I could simply accept this change and update my known_hosts files, like so:
root@nim1 : / # ssh-keygen -R aixlpar1 /.ssh/known_hosts updated. Original
contents retained as /.ss
With known_hosts updated, I’m able to SSH to the AIX 7.1 system successfully.
cgibson@nim1 : /home/cgibson $ ssh aixlpar1 date Mon Aug 20 19:44:20 EET 2012
But that’s just for my SSH known_hosts file only. What about all the users that connect to this system via SSH/SFTP/SCP? Do I really expect all of them to update their known_hosts file with the new host key information?
This could create problems for automated tasks, like file transfers. If these transfers stop working then their could be “hell to pay”. So the question I’m often asked is what can I do to prevent this from happening in the first place? Luckily there is a way.
In this example, we are using nimadm to migrate from AIX 5.3 to 7.1. The AIX 7.1 lpp_source resource was created using the AIX 7.1 installation media DVDs. All filesets were copied from the DVDs, verbatim, to the new 7.1 lpp_source resource on the NIM master.
First we verify that the openssh* filesets are in fact in the AIX 7.1 lpp_source on the NIM master.
root@nim1 : / # nim -o showres lpp_sourceaix710101 | grep -i ssh open open open open open open open open open
On the NIM client (running AIX 5.3), we verify there is an older version of SSH already installed. The migration will remove these filesets (and the associated /etc/ssh_host_* files). The newer version of SSH will be installed and new ssh_host_key* files will be generated (hence the problem with the remote SSH clients known_hosts files no longer holding the correct host keys).
aixlpar1 : / # oslevel -s 5300-12-04-1119
aixlpar1 : / # lslpp -l openssh* File
---- Path: /usr/lib/objrepos open open open open
The following SSH files are regenerated during the new fileset install:
aixlpar1
: /hom -rw-r--r-- 1 root
system 210 Feb 03
2006 /etc -rw------- 1 root
system 887 Feb 03
2006 /etc -rw-r--r-- 1 root
system 319 Feb 03
2006 /etc -rw------- 1 root
system 515 Feb 03
2006 /etc -rw-r--r-- 1 root
system 590 Feb 03
2006 /etc -rw------- 1 root
system 668 Feb 03
2006 /etc
You can
confirm the filesets are being installed during the nimadm operation by reviewing your clients nimadm log file in /var
root@nim1
: /var open open open open open open
aixlpar1 : / # ls -ltr /etc/ssh total 352 -rw-r--r-- 1
root system 210 Feb 03 2006 ssh_ -rw------- 1 root system 887 Feb 03 2006 ssh_host_rsa_key -rw-r--r-- 1 root system 319 Feb 03 2006 ssh_host_key.pub -rw------- 1 root system 515 Feb 03 2006 ssh_host_key -rw-r--r-- 1 root
system 590 Feb 03
2006 ssh_ -rw------- 1 root system 668 Feb 03 2006 ssh_host_dsa_key -rw-r--r-- 1 root system 2341 Apr 26 2007 ssh_prng_cmds -rw------- 1 root system 132839 Apr 26 2007 moduli -rw-r--r-- 1 root system 1288 May 01 2007 ssh_config -rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner -rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config -rw-r----- 1 root system 7 Aug 20 16:10 sshd.pid
To work around this issue, we can move the openssh* filesets out of the lpp_source directory (installp/ppc), like so:
root@nim1 : / # lsnim -l lpp_sourceaix710101 lpp_ class = resources type = lpp_source arch = power Rstate = ready for use prev_state = ready for use location
= /exp simages = yes alloc_count = 0 server = master
root@nim1
: / # cd /exp root@nim1
: /exp ./in ./in ./in ./in ./in ./in ./in
root@nim1
: /exp
root@nim1
: /exp -rw-r----- 1 root system 14336 Nov 16 2011 openssh.msg.en_US -rw-r----- 1 root system 138240 Nov 16 2011 openssh.man.en_US -rw-r----- 1 root system 6024192 Nov 16 2011 openssh.base -rw-r----- 1 root
system 17408 Nov 16
2011 open -rw-r----- 1 root
system 17408 Nov 16
2011 open -rw-r----- 1 root
system 153600 Nov 16
2011 open -rw-r----- 1 root
system 6298624 Nov 16
2011 open
I’ve chosen to move the filesets to a different directory (notice I’ve created a .toc file in the new location). I also need to copy (not move) the openssl* filesets to the new location as well.
root@nim1
: /exp root@nim1
: /exp root@nim1
: /exp
root@nim1
: /exp root@nim1
: /exp total 58648 -rw-r----- 1 root system 6298624 Nov 16 2011 openssl.base -rw-r----- 1 root system 14336 Nov 16 2011 openssh.msg.en_US -rw-r----- 1 root system 138240 Nov 16 2011 openssh.man.en_US -rw-r----- 1 root system 6024192 Nov 16 2011 openssh.base -rw-r----- 1 root system 2350080 Nov 16 2011 openssl.man.en_US -rw-r----- 1 root system 28672 Nov 16 2011 openssl.license -rw-r----- 1 root
system 2350080 Nov 16
2011 open -rw-r----- 1 root
system 3072 Nov 16
2011 open -rw-r----- 1 root
system 6307840 Nov 16
2011 open -rw-r----- 1 root
system 17408 Nov 16
2011 open -rw-r----- 1 root
system 17408 Nov 16
2011 open -rw-r----- 1 root
system 153600 Nov 16
2011 open -rw-r----- 1 root
system 6298624 Nov 16
2011 open -rw-r--r-- 1 root system 4966 Aug 20 20:41 .toc
Now that openssh is no longer in the lppsource, I check the AIX 7.1 lpp_source and update its TOC:
root@nim1 : / # nim -o check lpp_sourceaix710101
The openssh filesets are no longer listed as part of the lppsource:
root@nim1 : / # nim -o showres lpp_sourceaix710101 | grep -i ssh root@nim1 : / #
At this point we perform a standard nimadm operation to migrate from AIX 5.3 to 7.1. Openssh is not installed during the migration:
root@nim1: /var root@nim1: /var
After the migration, ssh connections to the server function just as they did before the migration.
cgibson@nim1 : /home/cgibson $ ssh aixlpar1 date Mon Aug 20 20:36:31 EETDT 2012
Now we can update the openssh filesets to the newer level. This will install the newer filesets but preserve the existing SSH host keys.
aixlpar1
: / # mount nim1
aixlpar1 : / # installp -Y -a -d /mnt ALL ...etc... Installation Summary ---- Nam ---- open open open open open open open open open open open
Rather than update these filesets manually after the migration, you can include this step as a post migration task with nimadm.
An alternative way to work around this problem (after the fact) would be to restore the original ssh_host_key* files from a backup. For example, I copied the original ssh_host_key* files to my home directory before starting the AIX migration.
aixlpar1 : / # cd /etc aixlpar1
: /etc # cp -pr ssh /hom
In the output below, I discover that my ssh_host_key* files have all been recreated during the migration.
aixlpar1 : /etc/ssh # ls -ltr total 352 -rw-r--r-- 1 root system 1288 May 01 2007 ssh_config -rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner -rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config -rw-r----- 1 root system 7 Aug 20 21:00 sshd.pid -rw-r--r-- 1 root system 2341 Aug 20 21:19 ssh_prng_cmds -rw------- 1 root system 132839 Aug 20 21:19 moduli -rw-r----- 1 root
system 382 Aug 20 21:45
ssh_ -rw------- 1 root system 1679 Aug 20 21:45 ssh_host_rsa_key -rw-r----- 1 root system 630 Aug 20 21:45 ssh_host_key.pub -rw------- 1 root system 965 Aug 20 21:45 ssh_host_key -rw-r----- 1 root
system 590 Aug 20 21:45
ssh_ -rw------- 1 root system 668 Aug 20 21:45 ssh_host_dsa_key
I copy the original files back to the /etc/ssh directory. The sshd subsystem is also restarted to pick up the updated ssh_host* files.
aixlpar1
: /etc/ssh # cp -p /hom
aixlpar1 : /etc/ssh # ls -ltr total 352 -rw-r--r-- 1 root
system 210 Feb 03
2006 ssh_ -rw------- 1 root system 887 Feb 03 2006 ssh_host_rsa_key -rw-r--r-- 1 root system 319 Feb 03 2006 ssh_host_key.pub -rw------- 1 root system 515 Feb 03 2006 ssh_host_key -rw-r--r-- 1 root
system 590 Feb 03
2006 ssh_ -rw------- 1 root system 668 Feb 03 2006 ssh_host_dsa_key -rw-r--r-- 1 root system 1288 May 01 2007 ssh_config -rw-r--r-- 1 root system 1155 May 04 2007 sshd_banner -rw-r--r-- 1 root system 2867 Oct 29 2008 sshd_config -rw-r----- 1 root system 7 Aug 20 21:00 sshd.pid -rw-r--r-- 1 root system 2341 Aug 20 21:19 ssh_prng_cmds -rw------- 1 root system 132839 Aug 20 21:19 moduli
aixlpar1 : /etc/ssh # stopsrc -s sshd 0513-044 The sshd Subsystem was requested to stop.
aixlpar1 : /etc/ssh # startsrc -s sshd 0513-059 The sshd Subsystem has been started. Subsystem PID is 3997822. |
_files/photo.png){kind=small}
Preserving the keys is good when it comes to prevent disruption, and the workaround will do. However it is also important to take into account why the key replacement was added in OpenSSH 5.8. The replacement was driven by two factors: a) weaknesses in the keys by using pseudo-random sources with unsufficient enthropy, and b) potential unauthorised access to host keys. Both vulnerabilities compromise the security of the encrypted connection, and key preservation will keep the servers exposed.
_files/photo_002.png){kind=small}
Chris, whats your thought , this procedure may vary if use different vendor ssh production in 5.3 and moving to open ssh in 7.1 ? ta
_files/photo_002.jpg){kind=small}
Hi Anthony and Dave, thanks for the comments. The wiki sounds like a great idea. We've got a decent list of tips and tricks already.
_files/photo.jpg){kind=small}
Chris, And tcp wrappers gets over written, well the hosts.allow/deny files inetd.conf and snmpd.conf gets over written. Good call on work around for SSH, bit of a bummer with sshd_confif, as my config file is heavily populated with chroot directives and allowed groups, fortunately I took local backups of my config files, so quickly got back to normal. DT
_files/photo_003.jpg){kind=small}
Chris, Maybe we should put together a developerWorks wiki on gotchas for migrations to AIX 7.1. So far, before the migration, check: ssh keys revert to default tuning parameters remove MPIO filesets save sendmail config aixpert save XML files After the migration: install MPIO filesets Replace Kerberos entry KRB5A in /usr/lib/security/methods.cfg with KRB5. Anthony
