_files/genppageinit.gif){kind=small}
_files/ibm-comm-wordmark.png){kind=small}
_files/blank.gif){kind=spacer}
_files/iconWarning.gif "IC Sunset")
The
developerWorks Connections platform will be sunset on December 31,
2019. On January 1, 2020, this blog will no longer be available. More details available on our FAQ.
How to live update your AIX system without rebooting the server.A little over a year ago, I wrote about the new AIX Live Update feature, available with the AIX 7.2. I discussed how to use this new feature to install an interim fix (ifix) on a running AIX system without a reboot. On October 11th, 2016, IBM announced the next exciting step in AIX Live Updates.
Introducing IBM AIX 7.2.1 Live Update
This is a significant enhancement, as it will allow AIX administrators to install TLs and SPs (and ifixes) without restarting their AIX systems. From the announcement:
"AIX Live Update for Technology Levels, Service Packs, and Interim Fixes
In this post I'll show you how to install updates without rebooting your AIX server. I recommend you first review my original article on Live Updates (from Oct 2015) in order to better understand the Live Update process, how it works and the requirements.
TL and SP Live Update support is delivered in AIX 7.2 TL1 (available November 11th 2016).
One of the biggest differences between the Live Update process for TL/SPs versus ifixes, is that you must backup your system prior to the update. This will be used in case you need to back out your system. The easiest way to do this is, is to create an alternate rootvg (alt disk clone).
On my system, I first applied TL1 for AIX 7.2 and verified the correct level was installed.
root@AIXmig / # oslevel -s 7200-01-00-0000 root@AIXmig / # cat /proc/version Oct 10 2016 11:53:17 1640C_72D @(#) _kdb_buildinfo unix_64 Oct 10 2016 11:53:17 1640C_72D
I had several "free" disks that I could use for the Live Update process. I'd need at least 3 disks, one for my alternate rootvg (back out), one for the mirror disk and one for the new rootvg. In this case I used hdisk6 (alt rootvg), hdisk3 (mdisk) and hdisk2 (ndisk). This was specified in the lvupdate.data configuration file. All three disks were large enough to hold a complete copy of my existing rootvg.
root@AIXmig / # lspv hdisk0 00f9 hdisk1 00f9 hdisk2 00f9 hdisk3 00f9 hdisk4 00f9 hdisk5 00f9 hdisk6 00f94f58def77f2c
root@AIXmig / # cat /var ... disks: nhdisk = hdisk2 mhdisk = hdisk3
Note, with 7.2 TL1, only two disks need to be specified in the lvupdate.data file. The tohdisk and tshdisk are not needed with TL1 unless you have a paging or dump device outside of rootvg. I cloned my rootvg to a spare disk (hdisk6) first. I specified the -B flag to ensure the boot list was not changed.
root@AIXmig / # alt_disk_copy -Bd hdisk6
The updates I needed to install were located in the /tmp directory.
root@AIXmig /tmp/cg/lku_ptfs # ls -ltr total 50328 -rw-r--r-- 1 root system 454656 Oct 20 00:49 bos. -rw-r--r-- 1 root system 41306112 Oct 20 00:50 bos.mp64.7.2.1.1.U -rw-r--r-- 1 root system 253952 Oct 20 00:50 bos. -rw-r--r-- 1 root system 369664 Oct 20 00:50 bos. -rw-r--r-- 1 root system 183296 Oct 20 00:50 bos. -rw-r--r-- 1 root system 1713152 Oct 20 00:50 bos. -rw-r--r-- 1 root system 155648 Oct 20 00:50 devi -rw-r--r-- 1 root system 2264 Oct 20 10:31 .toc
All of the filesets were currently installed at the 7.2.1.0 level.
root@AIXmig / # lslpp -l bos.liveupdate.rte bos.mp64 bos.net.tcp.ftp bos.rte.install bos.rte.libcfg bos.sysmgt.hmc devi File ---- Path: /usr/lib/objrepos bos. bos. bos. bos. bos. bos.sysmgt.hmc 7.2.1.0 COMMITTED HMC Client Runtime devi
Path: /etc/objrepos bos. bos. bos. bos. bos. bos. devi I ensured that my AIX partition could authenticate with my HMC (hsc02) and that I had a valid token.
root@AIXmig / # hmcauth -l Could not retrieve <tokens>.
root@AIXmig / # hmcauth -a hsc02 -u hscroot -p abc1234
root@AIXmig / # hmcauth -l Address : 10.1.50.30 User name: hscroot port : 12443
Next I performed a preview, without specifying any updates. This was to verify the environment was ready to accept a Live Update and shake out any problems.
root@AIXmig / # geninstall -k -p Validating live update input data.
Computing the estimated time for the live update operation: ---- LPAR: AIXmig Blackout_time(s): 18 Global_time(s): 708
Checking mirror vg device size: ---- Required device size: 7744 MB Given device size: 10239 MB PASSED: device size is sufficient.
Checking new root vg device size: ---- Required device size: 7744 MB Given device size: 10239 MB PASSED: device size is sufficient.
Checking temporary storage size for original LPAR: ---- Required device size: 0 or Undetermined MB Given device size: 2047 MB PASSED: device size is sufficient.
Checking temporary storage size for surrogate LPAR: ---- Required device size: 0 or Undetermined MB Given device size: 2047 MB PASSED: device size is sufficient.
Validating the adapters and their paths: ---- PASSED: adapters can be divided into two sets so that each has paths to all disks.
Checking lpar minimal memory size: ---- Required memory size: 2048 MB Current memory size: 2048 MB PASSED: memory size is sufficient.
Checking other requirements: ---- PASSED: sufficient space available in /var. PASSED: sufficient space available in /. PASSED: sufficient space available in /home. PASSED: no existing altinst_rvgLvup. PASSED: rootvg is not part of a snapshot. PASSED: pkcs11 is not installed. PASSED: DoD/DoDv2 profile is not applied. PASSED: Advanced Accounting is not on. PASSED: Virtual Trusted Platform Module is not on. PASSED: multiple semid lists is not on. PASSED: sufficient file system space for interim fix(es) is available. PASSED: The trustchk Trusted Execution Policy is not on. PASSED: The trustchk Trusted Library Policy is not on. PASSED: The trustchk TSD_FILES_LOCK policy is not on. PASSED: the boot disk is set to the current rootvg. PASSED: the mirrorvg name is available. PASSED: the rootvg is uniformly mirrored. PASSED: the rootvg does not have the maximum number of mirror copies. PASSED: the rootvg does not have stale logical volumes. PASSED: all of the mounted file systems are of a supported type. PASSED: this AIX instance is not diskless. PASSED: no Kerberos configured for NFS mounts. PASSED: multibos environment not present. PASSED: Trusted Computing Base not defined. PASSED: no local tape devices found. PASSED: live update not executed from console. PASSED: the execution environment is valid. PASSED: enough available space for /var to dump Component Trace buffers. PASSED: enough available space for /var to dump Light weight memory Trace buffers. PASSED: all devices are virtual devices. PASSED: No active workload partition found. PASSED: nfs configuration supported. PASSED: HMC token is present. PASSED: HMC requests for managed system successful. PASSED: HMC requests for lpar successful. PASSED: HMC is at a supported level. PASSED: HMC token is valid. PASSED: A virtual slot is available. PASSED: All VIOS are at supported level. PASSED: All storage adapter mappings are visible on VIOS. PASSED: All disk reserve policies set to NoReserve or PersistentStorage. PASSED: RSCT daemons are active. PASSED: no Kerberos configuration. PASSED: lpar is not remote restart capable. PASSED: the simplified remote restart configuration is supported. PASSED: no virtual log device configured. PASSED: lpar is using dedicated memory. PASSED: the disk configuration is supported. PASSED: no Generic Routing Encapsulation (GRE) tunnel configured. PASSED: Firmware level is supported. PASSED: vNIC resources available. PASSED: Consolidated system trace buffers size is within the limit of 64 MB. PASSED: SMT number is valid. PASSED: No process attached to vty0. PASSED: Audit is not enabled in stream mode. PASSED: No exclusive rsets (sysxrset) found. PASSED: Managed System state is operating. INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.
Then I performed another preview, this time specifying the location of the updates to be installed.
root@AIXmig / # geninstall -k -p -d /tmp/cg/lku_ptfs update_all
**** installp PREVIEW: installation will not actually occur. ****
+--- Pre-installation Verification... +--- Verifying selections...done Verifying requisites...done Results...
SUCCESSES --------- Filesets listed in this section passed pre-installation verification and will be installed.
Selected Filesets ----------------- bos.liveupdate.rte 7.2. bos.mp64 7.2. bos.net.tcp.ftp 7.2. bos.rte.install 7.2. bos.rte.libcfg 7.2. bos.sysmgt.hmc 7.2. devi
<< End of Success Section >>
+--- BUILDDATE Verification ... +--- Verifying build dates...done FILESET STATISTICS ------------------ 7 Selected to be installed, of which: 7 Passed pre-installation verification ---- 7 Total to be installed
RESOURCES --------- Estimated system resource requirements for filesets being installed: (All sizes are in 512-byte blocks) File /usr
/tmp
---- TOTA
NOTE: "Needed Space" values are calculated from data available prior to installation. These are the estimated resources required for the entire operation. Further resource checks will be made during installation to verify that these initial estimates are sufficient. ... **** End of installp PREVIEW. No apply operation has actually occurred. **** Validating live update input data.
Computing the estimated time for the live update operation: ---- LPAR: AIXmig Blackout_time(s): 18 Global_time(s): 701
Checking mirror vg device size: ---- Required device size: 7744 MB Given device size: 10239 MB PASSED: device size is sufficient. ... PASSED: Firmware level is supported. PASSED: vNIC resources available. PASSED: Consolidated system trace buffers size is within the limit of 64 MB. PASSED: SMT number is valid. PASSED: No process attached to vty0. PASSED: Audit is not enabled in stream mode. PASSED: No exclusive rsets (sysxrset) found. PASSED: Managed System state is operating. INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete. root@AIXmig / #
I took note of the current partition ID, as this would change after the Live Update was completed.
root@AIXmig / # uname -L 20 AIXmig
I started the Live Update process.
root@AIXmig / # geninstall -k -d /tmp/cg/lku_ptfs update_all
+--- Pre-installation Verification... +--- Verifying selections...done Verifying requisites...done Results...
SUCCESSES --------- Filesets listed in this section passed pre-installation verification and will be installed.
Selected Filesets ----------------- bos.liveupdate.rte 7.2. bos.mp64 7.2. bos.net.tcp.ftp 7.2. bos.rte.install 7.2. bos.rte.libcfg 7.2.1.1 # libcfg Library bos.sysmgt.hmc 7.2. devi
<< End of Success Section >>
+--- BUILDDATE Verification ... +--- Verifying build dates...done FILESET STATISTICS ------------------ 7 Selected to be installed, of which: 7 Passed pre-installation verification ---- 7 Total to be installed
0503-409 installp: bosboot verification starting... installp: bosboot verification completed. +--- +---
installp: APPLYING software for: devi
. . . . . << Copyright notice for devi Licensed Materials - Property of IBM
5765CD200 Copyright International Business Machines Corp. 2008, 2016.
All rights reserved. US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. . . . . . << End of copyright notice for devi
Filesets processed: 1 of 7 (Total time: 4 secs).
installp: APPLYING software for: bos.sysmgt.hmc 7.2.1.1 ... installp: bosboot process completed. +--- +---
Installation Summary ---- Name ---- devi bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. bos. devi bos. bos. bos. bos. bos. bos. Validating live update input data.
Computing the estimated time for the live update operation: ---- LPAR: AIXmig Blackout_time(s): 17 Global_time(s): 759 ... PASSED: Managed System state is operating. INFO: Any system dumps present in the current dump logical volumes will not be available after live update is complete.
Non-interruptable live update operation begins in 10 seconds.
Broadcast message from root@AIXmig (pts/0) at 10:38:35 ...
Live AIX update in progress.
Initializing live update on original LPAR.
Validating original LPAR environment.
Beginning live update operation on original LPAR.
Requesting resources required for live update. .... Notifying applications of impending live update.
Creating rootvg for boot of surrogate. .... Starting the surrogate LPAR. .... Creating mirror of original LPAR's rootvg. .... Moving workload to surrogate LPAR. ................ Blackout Time started.
Blackout Time end.
Workload is running on surrogate LPAR. .... Shutting down the Original LPAR. .... The live update operation succeeded.
Broadcast message from root@AIXmig (pts/0) at 10:51:35 ...
Live AIX update completed.
The process completed successfully and all of the filesets were updated to 7.2.1.1.
root@AIXmig / # lslpp -l bos.liveupdate.rte bos.mp64 bos.net.tcp.ftp bos.rte.install bos.rte.libcfg bos.sysmgt.hmc devi File ---- Path: /usr/lib/objrepos bos. bos. bos. bos. bos. bos. devi
I used the genld command to list the processes that had an old version of a library loaded.
root@AIXmig / # genld -lu Proc_pid: 4915634 Proc_name: hostmibd 9000000004f5ce0 16c90 /usr
Proc_pid: 5177766 Proc_name: nimsh d0609100 15f27 /usr
Proc_pid: 5308838 Proc_name: aso 9000000004f5ce0 16c90 /usr
Proc_pid: 5767616 Proc_name: rpc.lockd d0609100 15f27 /usr
Proc_pid: 6947326 Proc_name: IBM.HostRMd d0609100 15f27 /usr
Proc_pid: 7078374 Proc_name: clcomd d0609100 15f27 /usr
Proc_pid: 7799098 Proc_name: sshd d0609100 15f27 /usr
Proc_pid: 7864602 Proc_name: sshd d0609100 15f27 /usr
Proc_pid: 8257802 Proc_name: rmcd d0609100 15f27 /usr
Proc_pid: 8585628 Proc_name: IBM.MgmtDomainRMd d0609100 15f27 /usr
Proc_pid: 8782132 Proc_name: IBM.ConfigRMd d0609100 15f27 /usr
Proc_pid: 9765170 Proc_name: topasrec d0609100 15f27 /usr
root@AIXmig / #
As expected, the partition ID had changed.
root@AIXmig / # uname -L 31 AIXmig
The update was completed successfully.
root@AIXmig / # errpt IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION 12295E0B 1020105116 I S LVUPDATE Live AIX update completed successfully CB4A951F 1020104816 I S SRC SOFTWARE PROGRAM ERROR CB4A951F 1020104816 I S SRC SOFTWARE PROGRAM ERROR CB4A951F 1020104816 I S SRC SOFTWARE PROGRAM ERROR CB4A951F 1020104816 I S SRC SOFTWARE PROGRAM ERROR 9DBCFDEE 1020104816 T O errdemon ERROR LOGGING TURNED ON 9A74C7AB 1020103816 I S LVUPDATE Live AIX update started root@AIXmig / #
If I needed to back out, to the previous level, I could change the boot list to point to the alternate rootvg (hdisk6), and restart the system.
root@AIXmig / # lspv hdisk0 00f9 hdisk1 00f9 hdisk2 00f9 hdisk3 00f9 hdisk4 00f9 hdisk5 00f9 hdisk6 00f9
root@AIXmig / # bootlist -m normal -o hdisk0 blv=hd5 pathid=0 hdisk0 blv=hd5 pathid=1
root@AIXmig / # bootlist -m normal hdisk6 hdisk6 blv=hd5 pathid=0 hdisk6 blv=hd5 pathid=1
root@AIXmig / # bootlist -m normal -o hdisk6 blv=hd5 pathid=0 hdisk6 blv=hd5 pathid=1
root@AIXmig / # at now shutdown –Fr Job root.1477276394.a will be run at Mon Oct 24 13:33:14 AEDT 2016.
This is very cool technology. Gone are the days of needing to plan reboots shortly after applying a new TL or SP to a critical AIX system. You simply "live update" your system, without disrupting your workloads or your users. This is a win for AIX administrators everywhere!
Please refer to the AIX 7.2 Knowledge Center for more information on Live Update. http http |
_files/photo.jpg){kind=small}
Hi Shailesh, thanks for the information you sent me. The live update log file was very helpful. The preview was failing with "1430-095 FAILED: some disks are not supported". This is a match for APAR IJ06810 LIVE UPDATE DOESN'T ALLOW EMC INVISTA DISKS, http://www-01.ibm.com/support/docview.wss?uid=isg1IJ06810. Thanks for your email this morning. It's good to know that once you installed the ifix, the problem was resolved.
_files/photo_003.png){kind=small}
Hi Eggibbo,
/var/adm/ras/liveupdate/logs/lvupdlog not helping much, I have sent
you this log on your specified email-id kindly let know if you find any
clue in that.
Hi Shailesh, I'm sorry to hear you are having some trouble with live update. Please review the /var/adm/ras/liveupdate/logs/lvupdlog file for any obvious error messages. Feel free to email the log file to me also (cgibson@au1.ibm.com).
Hi Chris, Today I was trying to do live update on my server, but it failing with below error ..
Requesting resources required for live update.
................................1430-024 An error occurred when requesting resources.
1430-045 The live update operation failed. Cleanup action is started.
....1430-115 Warning: Cleanup from the Live Update failure may not have been successful. Run clvupdate to finish the cleanup.
In preview it passed all the phases .. please help to get rid of this error.
Thanks for your comment Travis. You are 100% correct. Essentially, you can patch your system just as you always have. But instead of rebooting you would run: geninstall -k. This will initiate the "Live Update" operation and resume the workload on the surrogate partition. Your application workloads are not stopped, yet they can take advantage of the new fixes immediately after the Live Update operation, without a reboot. Administrators can use the Live Update feature to install AIX interim fixes, service packs and technology levels. You can also use PowerVC to simplify the process! Here: https://www.ibm.com/developerworks/community/blogs/cgaix/entry/Simplify_AIX_Live_Updates_with_PowerVC?lang=en
_files/photo_002.jpg){kind=small}
If I
were going to use NIM to push the upgrade, I would do everything the
same, except instead of a reboot to make the TL upgrade take affect, I
would instead run the "geninstall -k -d /tmp/cg/lku_ptfs update_all" on
the client LPAR after the push was finished?
From what I read, it
looks like the upgrade process is pretty much the same except for that
command replaces the reboot.... well, also you're stuck with a new
partition ID and a "lvup_rootvg" instead of a "rootvg" disk in the
end... at least till you reboot or do another Live Update.
Thanks bhargaku. I'll write up another post about how to troubleshoot live update, which will include some clean up steps, etc. Keep watching my blog for updates, soon.
_files/photo.png){kind=small}
Chris, nice article. Can also include cleanup steps if lku failed/hang at different stages(starting surrogate,create mirror,blackout..etc).
_files/photo_002.png){kind=small}
Chris,Thank you.When I rebooted server,Its resolved.
how can ı remove lvup_rootvg.When I try to remove, I got this error below.
# lspv
hdisk0 00f82d7a45a529bd rootvg active
hdisk2 00f82d7a8afd0d1c lvup_rootvg
# exportvg lvup_rootvg
0516-1998 exportvg: Unable to export volume group lvup_rootvg
due to previous live update.
Immediately
after a live update, the running system is essentially booted from the
"lvup_rootvg " disk (which is the surrogate boot disk). So you will not
be able to remove it. It will be released either when you perform
another live update or when you reboot the system.
https://www.ibm.com/support/knowledgecenter/en//ssw_aix_72/com.ibm.aix.install/lvupdate_configuration.htm
