_files/genppageinit.gif){kind=small}
_files/ibm-comm-wordmark.png){kind=small}
_files/blank.gif){kind=spacer}
_files/iconWarning.gif "IC Sunset")
The
developerWorks Connections platform will be sunset on December 31,
2019. On January 1, 2020, this blog will no longer be available. More details available on our FAQ.
Updating your PowerHA cluster with cl_ezupdatePowerHA 7.2.1 (June 2017) introduced the cl_ezupdate utility. You can use the cl_ezupdate command to update the software for the entire cluster or a subset of nodes in the cluster, often without disrupting workloads.
You can use this tool to apply and reject updates for PowerHA service packs and technology levels and interim fixes. This process is performed on the entire cluster or on a specific subset of nodes in the cluster. You can also apply updates in preview mode. When you use preview mode, all the prerequisites for installation process are checked, but the cluster updates are not installed on the system.
The tool with also allow you to apply and reject updates for AIX service packs or interim fixes. But you cannot use it to update the cluster to newer AIX technology levels.
Make sure you read the "Limitations" section of the knowledge centre to understand the current limitations and restrictions for the cl_ezupdate command.
Upgrade PowerHA SystemMirror using the cl_ezupdate command
cl_ezupdate command http
cl_ezupdate Command Usage Information
Purpose: Designed to manage the entire PowerHA cluster update without interrupting the application activity.
Usage: cl_ezupdate [-v] -h cl_ezupdate [-v] -Q {cluster|node|nim} [-N <node1,node2,...>] cl_ezupdate [-v] {-Q {lpp|all} |-P|-A|-C|-R} [-N <node1,node2,...>] -s <repository> [-F]
Description: Query informations about cluster state and available updates, install updates in preview mode or apply, commit or reject updates.
Flags: -h Displays the help for this program. -v Sets the verbose mode of help. -s Specifies the update source. It could be a directory, then it should start with "/" character. It could be a LPP source if the update is done through NIM server. -N Specify the node names. By default the scope is the entire cluster. -Q Query cluster status and or available updates. The scope of query request is: clus -P Do not install any update, just try to install it in preview mode. -A Apply the updates located on the repository. -C Commit the latest update version. -R Roll back to the previous version. -F Force mode. only combined with -A option. Should be use if the install of SP is not possible because an Ifix is locking an installable file set.
Output file: /var
Examples: To display informations about NIM servers type: cl_ezupdate -Q nim
To check and display contents of lpp source, type: cl_ezupdate -Q lpp -s /tmp
To install in apply mode lpp located on NIM server, type: cl_ezupdate -A -s HA_v720_lpp
------------
In the following example, I will update my two node PowerHA cluster from PowerHA 7.2.1 SP1 to SP2. Both nodes are running AIX 7.1 TL4 SP4.
PowerHA fix information For PowerHA 7.2.1 Service Pack 2 http
I'll use an LPP source on my NIM master to provide the updates to the cl_ezupdate command.
The LPP source, on the NIM master, is named pha721sp2.
# lsnim -t lpp_source | grep pha72 pha7
# lsnim -l pha721sp2 pha721sp2: class = resources type = lpp_source arch = power Rstate = ready for use prev_state = unavailable for use location = /exp alloc_count = 0 server = master
Both PowerHA nodes have already been configured as NIM clients to the NIM master. I can run nimclient -l, from each node, to confirm they’re able to communicate with the master.
[root@lpar1]/# nimclient -l mast boot nim_ 10_1 710l AIX7 hsc0 750_ ....
Use the clcmd command to confirm the HA and AIX levels on both nodes in the cluster.
[root@lpar1]/# clcmd oslevel -s
---- NODE lpar ---- 7100-04-04-1717
---- NODE lpar ---- 7100-04-04-1717
[root@lpar1]/# clcmd halevel -s
---- NODE lpar ---- 7.2.1 SP1
---- NODE lpar ---- 7.2.1 SP1
From one of the of the nodes, I can run cl_ezupdate –Q to query the NIM master LPP source resources. I'm looking for pha721sp2.
[root@lpar1]/# cl_ezupdate -Q nim Checking for root authority... Running as root. Checking for AIX level... The installed AIX version is supported. Checking for PowerHA SystemMirror version. The installed PowerHA SystemMirror version is supported. Checking for clcomd communication on all nodes... clcomd on each node can both send and receive messages. INFO: The cluster: STGMELB is in state: STABLE INFO: The node: lpar1 is in state: NORMAL INFO: The node: lpar9 is in state: NORMAL Checking for NIM servers... Available lpp_source on NIM server: 750lpar4 from node: lpar9 : 710l AIX7 powe powe aix7 cglp tncp xxxl pha7 java AIX5 AIX7 Available lpp_source on NIM server: 750lpar4 from node: lpar1 : 710l AIX7 powe powe aix7 cglp tncp xxxl pha7 java AIX5 AIX7 [root@lpar1]/#
Preview the update/install first. Just to be sure there’s no issue with the filesets in the NIM resource.
[root@lpar1]/# cl_ezupdate -P -v -s pha721sp2 Checking for root authority... Running as root. Checking for AIX level... The installed AIX version is supported. Checking for PowerHA SystemMirror version. The installed PowerHA SystemMirror version is supported. Checking for clcomd communication on all nodes... clcomd on each node can both send and receive messages. INFO: The cluster: STGMELB is in state: STABLE INFO: The node: lpar1 is in state: NORMAL INFO: The node: lpar9 is in state: NORMAL Checking for NIM servers... Checking for lpps and Ifixes from source: pha721sp2... Build lists of filesets that can be apply reject or commit on node lpar9 Fileset list to apply on node lpar9: clus There is nothing to commit or reject on node: lpar9 from source: pha721sp2 Build lists of filesets that can be apply reject or commit on node lpar1 Fileset list to apply on node lpar1: clus There is nothing to commit or reject on node: lpar1 from source: pha721sp2 Installing fileset updates in preview mode on node: lpar9... Succeeded to install preview updates on node: lpar9. Installing fileset updates in preview mode on node: lpar1... Succeeded to install preview updates on node: lpar1.
To apply SP2, to both nodes in the cluster, I run cl_ezupdate –A. The process updates the second first, as it does not currently house the HA resource group. The node joins the cluster again, after the updates have been applied. Once the second node is updated successfully, the first node is updated next. Note that the cluster resources remain available. The cluster is UNMANAGED during the update, so there’s no disruption to the HA applications.
Applying Updates
[root@lpar1]/# cl_ezupdate -A -v -s pha721sp2 Checking for root authority... Running as root. Checking for AIX level... The installed AIX version is supported. Checking for PowerHA SystemMirror version. The installed PowerHA SystemMirror version is supported. Checking for clcomd communication on all nodes... clcomd on each node can both send and receive messages. INFO: The cluster: STGMELB is in state: STABLE INFO: The node: lpar1 is in state: NORMAL INFO: The node: lpar9 is in state: NORMAL Checking for NIM servers... Checking for lpps and Ifixes from source: pha721sp2... Build lists of filesets that can be apply reject or commit on node lpar9 Fileset list to apply on node lpar9: clus There is nothing to commit or reject on node: lpar9 from source: pha721sp2 Build lists of filesets that can be apply reject or commit on node lpar1 Fileset list to apply on node lpar1: clus There is nothing to commit or reject on node: lpar1 from source: pha721sp2 Installing fileset updates in preview mode on node: lpar9... Succeeded to install preview updates on node: lpar9. Installing fileset updates in preview mode on node: lpar1... Succeeded to install preview updates on node: lpar1. Stopping the node lpar9... Stopping PowerHA cluster services on node: lpar9 in offline mode... lpar9: 0513-044 The clinfoES Subsystem was requested to stop. lpar9: 0513-044 The clevmgrdES Subsystem was requested to stop. . "lpar9" is now offline.
lpar9: Jan 17 2018 14:3 Applying updates on node: lpar9... Succeeded to apply updates on node: lpar9. Starting the node: lpar9... Starting cluster manager daemon: clstrmgrES... Starting PowerHA cluster services on node: lpar9 in manual mode... Verifying cluster configuration prior to starting cluster services
WARNING: Cluster verification detected that some cluster components are inactive. Please use the matrix below to verify the status of inactive components:Node: lpar WARNING: No backup repository disk is UP and not already part of a VG for nodes : lpar9: start_cluster: Starting PowerHA SystemMirror lpar9: 3604622 - 0:00 syslogd lpar9: Setting routerevalidate to 1 lpar9: 0513-059 The clevmgrdES Subsystem has been started. Subsystem PID is 13959390. ... "lpar9" is now online.
Starting Cluster Services on node: lpar9 This may take a few minutes. Please wait... lpar9: Jan 17 2018 14:44:36Starting execution of /usr lpar9: with parameters: -boot -N -b -P cl_rc_cluster -A lpar9: lpar9: Jan 17 2018 14:44:36usage: cl_echo messageid (default) messageJan 17 2018 14:44:36usage: cl_echo messageid (default) messageJan 17 2018 14:44:37 lpar9: /usr lpar9: lpar9: Jan 17 2018 14:44:39 lpar9: Completed execution of /usr lpar9: with parameters: -boot -N -b -P cl_rc_cluster -A. lpar9: Exit status = 0 lpar9: Stopping the node lpar1... Stopping PowerHA cluster services on node: lpar1 in unmanage mode...
Broadcast message from root@lpar1 (tty) at 14:45:04 ...
PowerHA SystemMirror on lpar1 shutting down. Please exit any cluster applications... lpar1: 0513-044 The clevmgrdES Subsystem was requested to stop. . "lpar1" is now unmanaged.
lpar1: Jan 17 2018 14:4 Applying updates on node: lpar1... Succeeded to apply updates on node: lpar1. Starting the node: lpar1... Starting cluster manager daemon: clstrmgrES... Starting PowerHA cluster services on node: lpar1 in auto mode... Verifying cluster configuration prior to starting cluster services
Verifying Cluster Configuration Prior to Starting Cluster Services.
Verifying node(s): lpar1 against the running node lpar9
WARNING: No backup repository disk is UP and not already part of a VG for nodes : Successfully verified node(s): lpar1 lpar1: start_cluster: Starting PowerHA SystemMirror ....... "lpar1" is now online.
Starting Cluster Services on node: lpar1 This may take a few minutes. Please wait... lpar1: Jan 17 2018 14:51:49Starting execution of /usr lpar1: with parameters: -boot -N -b -P cl_rc_cluster -A lpar1: lpar1: Jan 17 2018 14:51:49usage: cl_echo messageid (default) messageJan 17 2018 14:51:49usage: cl_echo messageid (default) mess [root@lpar1]/#
Both nodes are now running PowerHa 7.2.1 SP2.
[root@lpar1]/# clcmd halevel -s
---- NODE lpar ---- 7.2.1 SP2
---- NODE lpar ---- 7.2.1 SP2
During the update, the cluster state is UNMANAGED, as shown below in the cldump output.
[roo
Obtaining information via SNMP from Node: lpar1...
____ Cluster Name: STGMELB Cluster State: UP Cluster Substate: STABLE ____
Node Name: lpar1 State: UP
Network Name: net_ether_01 State: UP
Address: 10.1.50.199 Label: lpar1svc State: UP Address: 10.1.50.31 Label: lpar1 State: UP
Node Name: lpar
Network Name: net_ether_01 State: UP
Address: 10.1.50.39 Label: lpar9 State: UP
Cluster Name: STGMELB
Resource Group Name: RG1 Startup Policy: Online On Home Node Only Fallover Policy: Fallover To Next Priority Node In The List Fallback Policy: Never Fallback Site Policy: ignore Node ---- lpar lpar
After the update, the cluster is stable and all resources are exactly where they should be.
[root@lpar1]/# cldump
Obtaining information via SNMP from Node: lpar1...
____ Cluster Name: STGMELB Cluster State: UP Cluster Substate: STABLE ____
Node Name: lpar1 State: UP
Network Name: net_ether_01 State: UP
Address: 10.1.50.199 Label: lpar1svc State: UP Address: 10.1.50.31 Label: lpar1 State: UP
Node Name: lpar
Network Name: net_ether_01 State: UP
Address: 10.1.50.39 Label: lpar9 State: UP
Cluster Name: STGMELB
Resource Group Name: RG1 Startup Policy: Online On Home Node Only Fallover Policy: Fallover To Next Priority Node In The List Fallback Policy: Never Fallback Site Policy: ignore Node ---- lpar lpar
The cl_ezupdate process logs all its action to log files in /var
[roo total 2336 -rw-r--r-- 1 root system 288218 Jan 17 14:35 EZUpdate.log.4 -rw-r--r-- 1 root system 288205 Jan 17 14:38 EZUpdate.log.3 -rw-r--r-- 1 root system 329460 Jan 17 14:52 EZUpdate.log.2 -rw-r--r-- 1 root system 488 Jan 17 15:08 EZUpdate.log.1 -rw-r--r-- 1 root system 278000 Jan 17 15:10 EZUpdate.log
You’ll notice, during the update, that the cl_ezupdate command is running the nimclient command to pull down and install the updates on all nodes.
root 15990974 16187598 0 14:39:22 - 0:00 /usr/sbin/nimclient -o cust -a installp_flags=agXw -a accept_licenses=yes -a lpp_
Rejecting Updates
You can also reject updates. I tested this by simply rejecting all the updates that were just applied (in the previous step). The cl_ezupdate –R command will reject the updates, on both nodes (by default. You can choose a subset of nodes if you wish).
[root@lpar1]/# cl_ezupdate -R -s pha721sp2 Checking for root authority... Running as root. Checking for AIX level... The installed AIX version is supported. Checking for PowerHA SystemMirror version. The installed PowerHA SystemMirror version is supported. Checking for clcomd communication on all nodes... clcomd on each node can both send and receive messages. INFO: The cluster: STGMELB is in state: STABLE INFO: The node: lpar1 is in state: NORMAL INFO: The node: lpar9 is in state: NORMAL Checking for NIM servers... Checking for lpps and Ifixes from source: pha721sp2... Build lists of filesets that can be apply reject or commit on node lpar9 There is nothing to install on node: lpar9 from source: pha721sp2 Fileset list to reject or commit on node lpar9 = clus Build lists of filesets that can be apply reject or commit on node lpar1 There is nothing to install on node: lpar1 from source: pha721sp2 Fileset list to reject or commit on node lpar1 = clus Stopping the node lpar9... Stopping PowerHA cluster services on node: lpar9 in offline mode... lpar9: 0513-044 The clinfoES Subsystem was requested to stop. lpar9: 0513-044 The clevmgrdES Subsystem was requested to stop. "lpar9" is now offline.
lpar9: Jan 17 2018 15:1 Rejecting applied updates on node: lpar9... Succeeded to reject updates on node: lpar9. Starting the node: lpar9... ... "lpar9" is now online.
Stopping the node lpar1... Stopping PowerHA cluster services on node: lpar1 in unmanage mode... . "lpar1" is now unmanaged.
lpar1: Jan 17 2018 15:1 Rejecting applied updates on node: lpar1... Succeeded to reject updates on node: lpar1. ...... "lpar1" is now online.
The updates have been rejected successfully and both nodes have returned to PowerHA 7.2.1 SP1.
[root@lpar1]/# clcmd halevel -s
---- NODE lpar ---- 7.2.1 SP1
---- NODE lpar ---- 7.2.1 SP1
I did notice that in both cases, applying or rejecting, that the clinfoES daemon was not restarted when the cluster returned from unmanaged to online mode. I simply restarted these services, manually. Could be a bug. I’ll need to dig deeper.
[root@lpar1]/# clcmd startsrc -s clinfoES
---- NODE lpar ---- 0513-059 The clinfoES Subsystem has been started. Subsystem PID is 16252936.
---- NODE lpar ---- 0513-059 The clinfoES Subsystem has been started. Subsystem PID is 13762662.
[root@lpar1]/# clstat -o
clstat - PowerHA SystemMirror Cluster Status Monitor ----
Cluster: STGMELB (1529383297) Wed Jan 17 15:20:13 2018 State: UP Nodes: 2 SubState: STABLE
Node: lpar1 State: UP Interface: lpar1 (0) Address: 10.1.50.31 Interface: lpar1svc (0) Address: 10.1.50.199 Resource Group: RG1
Node: lpar9 State: UP Interface: lpar9 (0) Address: 10.1.50.39
If you run into any issues with cl_ezupdate, check the log files for obvious problems.
[roo
<< End of Success Section >>
+--- BUILDDATE Verification ... +--- Verifying build dates...done FILESET STATISTICS ------------------ 12 Selected to be installed, of which: 12 Passed pre-installation verification ---- 12 Total to be installed
RESOURCES --------- Estimated system resource requirements for filesets being installed: (All sizes are in 512-byte blocks) File
/
/usr ---- TOTA
NOTE: "Needed Space" values are calculated from data available prior to installation. These are the estimated resources required for the entire operation. Further resource checks will be made during installation to verify that these initial estimates are sufficient.
**** End of installp PREVIEW. No apply operation has actually occurred. **** INF: Succeeded to install preview updates on node: lpar1. INF: _.cl INF: _.cl INF: _.cl INF: clea ### 2018_01_17 14:35 - Leaving script: cl_ezupdate -P -V -S pha721sp2 rc=0
|
_files/photo.png){kind=small}
Release 7.2.2 added the ability to have an alt_disk_copy of rootvg and restore rootvg if errors are encountered.
