Home » , , , , , , » RAC : Apply PSU Patch To Standard Cluster 12c

RAC : Apply PSU Patch To Standard Cluster 12c

Written By Srikrishna Murthy Annam on Wednesday, March 12, 2014 | 7:26 AM


In this article, we will see how to apply PSU patch to the standard cluster. I have a RAC 12c standard cluster installed with 3 nodes.

Environment Details : 
RAC Version Cluster Type
12c  12.1.0.1.0
Standard Cluster
Cluster Nodes
askmrac1/2/3
DB Running on
All Cluster Nodes
Task
Applying OCT2013 PSU Patch



Environment Configuration :
Type
Path
Owner
Version
Shared
Grid Infra Home
/u01/app/12.1.0/grid_1
Grid
12.1.0.1
False
Database Home
/u01/app/oracle/product/12.1.0/dbhome_1
Oracle
12.1.0.1
Flase


Step By Step
  1. 1. Download and Unzip the Latest OPatch to all cluster nodes
  2. 2. Validate and Record Pre-Patch information
  3. 3. Create OCM response file if it does not exist
  4. 4. Download and Unzip the JAN2014 PSU patch for GI 12c ie 12.1.0.1.2
  5. 5. One-off patch Conflicts detection and Resolution
  6. 6. Patch application
  7. 7. Verification of Patch application
  8. 8. Issues and Resolutions




1. Download and Unzip the Latest OPatch to all cluster nodes:
You must use the OPatch utility version12.1.0.1.1 or later to apply this patch. Oracle recommends that you use the latest released OPatch for 12.1 releases, which is available for download from My Oracle Support patch 6880880.

It is always a best practice to keep both GRID_HOME and DATABASE HOME at same opatch level. So change the opatch in DATABASE HOME also.

Repeat this step to all the nodes in the cluster.


2. Validate and Record Pre-Patch information :
Validate using the following commands :
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/12.1.0/grid/
$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/app/oracle/product/12.1.0/dbhome_1

Login to each node in RAC as grid user and execute the following command.
$GRID_ORACLE_HOME/OPatch/opatch lsinventory
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Login to each node in RAC as oracle user and execute the following command.
$ORACLE_HOME/OPatch/opatch lsinventory
$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Connect to each instance and record registry information.
SQL> select comp_name,version,status from dba_registry;

3. Create OCM Response File If It Does Not Exist : 
Create ocm response file using the following command and provide appropriate values for the prompts.
$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp
Verify the created file using,
$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp –verbose ocm.rsp
NOTE: The Opatch utility will prompt for your OCM (Oracle Configuration Manager) response file when it is run. Without which we cant proceed further.

Copy this response file ocm.rsp to all the nodes in the cluster to the same location. Or you can create a new response file on each node using the same method above.

4. Download and Unzip the JAN2014 PSU patch : (as grid user)
Patch 17272829 is the OCT2013 PSU patch. It is downloaded and unzipped to the locaiton "/mnt/software/RAC/1201_PSU/"

5. One-off patch Conflicts detection and Resolution :
Determine whether any currently installed one-off patches conflict with the PSU patch.
$GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/17272829  analyze

But i don't have any patches applied to my home yet. So i can ignore this step. But if you have any conflicts identified in GI home or in DB home, follow MOS ID 1061295.1 to resolve the conflicts.

6. Patch Application : 
Now patch can be applied using the following syntax.
# opatchauto apply <PATH_TO_PATCH_DIRECTORY> -ocmrf <ocm response file>
On all Cluster nodes ( askmrac1/2/3)
[root@askmrac1 ~]# export PATH=/u01/app/12.1.0/grid/OPatch:$PATH
[root@askmrac1 ~]# which opatchauto
/u01/app/12.1.0/grid/OPatch/opatchauto
[root@askmrac1 ~]# opatchauto apply /mnt/software/RAC/1201_PSU/17272829 -ocmrf /tmp/askm/ocm.rspOPatch Automation Tool
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version        : 12.1.0.1.0
Running from       : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-08_04-17-23_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /mnt/software/RAC/1201_PSU/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/mnt/software/RAC/1201_PSU/17272829/17027533" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Patch "/mnt/software/RAC/1201_PSU/17272829/17077442" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Stopping CRS ... Successful
Applying patch(es) to "/u01/app/12.1.0/grid_1" ...
Patch "/mnt/software/RAC/1201_PSU/17272829/17027533" successfully applied to "/u01/app/12.1.0/grid_1".
Patch "/mnt/software/RAC/1201_PSU/17272829/17077442" successfully applied to "/u01/app/12.1.0/grid_1".
Patch "/mnt/software/RAC/1201_PSU/17272829/17303297" successfully applied to "/u01/app/12.1.0/grid_1".
Starting CRS ... Successful
Starting RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
[WARNING] SQL changes, if any, could not be applied on the following database(s): ORCL ... Please refer to the log file for more details.
Apply Summary:
Following patch(es) are successfully installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
opatchauto ran into some warnings during patch installation (Please see log file for details):
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
opatchauto succeeded.
[root@askmrac1 ~]#

Repeat the above step to all the cluster nodes in the standard cluster.

7. Verification of Patch application : 
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/12.1.0/grid/
$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/app/oracle/product/12.1.0/dbhome_1

Login to each node in RAC as grid user and execute the following command.
$GRID_ORACLE_HOME/OPatch/opatch lsinventory
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Login to each node in RAC as oracle user and execute the following command.
$ORACLE_HOME/OPatch/opatch lsinventory
$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Connect to each instance and verify registry information.
SQL> select comp_name,version,status from dba_registry;

[root@askmrac1 ~]# su - grid
[grid@askmrac1 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/12.1.0/grid_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.1.0/grid_1/oraInst.loc
OPatch version    : 12.1.0.1.2
OUI version       : 12.1.0.1.0
Log file location : /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-08_04-46-22AM_1.log
Lsinventory Output file location : /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/lsinv/lsinventory2014-03-08_04-46-22AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c                                       12.1.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (3) :
Patch  17303297     : applied on Sat Mar 08 04:35:19 EST 2014
Unique Patch ID:  16881795
Patch description:  "ACFS Patch Set Update 12.1.0.1.1"
   Created on 14 Oct 2013, 07:25:50 hrs US/Central
   Bugs fixed:
     14487556, 16398970, 16552813, 16930184, 16420645, 16170117, 16436434
     16476044, 16458315, 16463033, 16095100, 16545876, 16429953, 14826673
     16001893, 16482869, 16371746, 16435343, 14476443, 16294308, 16671486
     16386110, 15978267, 16085530, 16347837, 16814544, 16022372, 16167084
     14510092, 16450287, 16399406
Patch  17077442     : applied on Sat Mar 08 04:31:12 EST 2014
Unique Patch ID:  16881794
Patch description:  "Oracle Clusterware Patch Set Update 12.1.0.1.1"
   Created on 12 Oct 2013, 06:33:53 hrs US/Central
   Bugs fixed:
     16505840, 16505255, 16390989, 16399322, 16505617, 16505717, 17486244
     16168869, 16444109, 16505361, 13866165, 16505763, 16208257, 16904822
     17299876, 16246222, 16505214, 16505540, 15936039, 16580269, 16838292
     16505449, 16801843, 16309853, 16505395, 17507349, 17475155, 16493242
     17039197, 16196609, 17463260, 16505667, 15970176, 16488665, 16670327
Patch  17027533     : applied on Sat Mar 08 04:27:54 EST 2014
Unique Patch ID:  16677152
Patch description:  "Database Patch Set Update : 12.1.0.1.1 (17027533)"
   Created on 27 Sep 2013, 05:30:33 hrs PST8PDT
   Bugs fixed:
     17034172, 16694728, 16448848, 16863422, 16634384, 16465158, 16320173
     16313881, 16910734, 16816103, 16911800, 16715647, 16825779, 16707927
     16392068, 14197853, 16712618, 17273253, 16902138, 16524071, 16856570
     16465149, 16705020, 16689109, 16372203, 16864864, 16849982, 16946613
     16837842, 16964279, 16459685, 16978185, 16845022, 16195633, 14536110
     16964686, 16787973, 16850996, 16674842, 16838328, 16178562, 15996344
     16503473, 16842274, 16935643, 17000176, 14355775, 16362358, 16994576
     16485876, 16919176, 16928832, 16864359, 16617325, 16921340, 16679874
     16788832, 16483559, 16733884, 16784167, 16286774, 15986012, 16660558
     16674666, 16191248, 16697600, 16993424, 16946990, 16589507, 16173738
     16784143, 16772060, 16991789, 17346196, 16495802, 16859937, 16590848
     16910001, 16603924, 16427054, 16730813, 16227068, 16663303, 16784901
     16836849, 16186165, 16457621, 16007562, 16170787, 16663465, 16524968
     16543323, 17027533, 16675710, 17005047, 16795944, 16668226, 16070351
     16212405, 16523150, 16698577, 16621274, 16930325, 17330580, 16443657


Patch level status of Cluster nodes :
 Patching Level                  Nodes
 --------------                  -----
 1650217826                      askmrac1
 0                               askmrac3,askmrac2
--------------------------------------------------------------------------------
OPatch succeeded.
[grid@askmrac1 ~]$

[root@askmrac1 ~]# su - oracle
[oracle@askmrac1 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.2
OUI version       : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-08_04-46-46AM_1.log
Lsinventory Output file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2014-03-08_04-46-46AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Database 12c                                                  12.1.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (2) :
Patch  17077442     : applied on Fri Mar 07 12:48:25 EST 2014
Unique Patch ID:  16881794
Patch description:  "Oracle Clusterware Patch Set Update 12.1.0.1.1"
   Created on 12 Oct 2013, 06:33:53 hrs US/Central
   Bugs fixed:
     16505840, 16505255, 16390989, 16399322, 16505617, 16505717, 17486244
     16168869, 16444109, 16505361, 13866165, 16505763, 16208257, 16904822
     17299876, 16246222, 16505214, 16505540, 15936039, 16580269, 16838292
     16505449, 16801843, 16309853, 16505395, 17507349, 17475155, 16493242
     17039197, 16196609, 17463260, 16505667, 15970176, 16488665, 16670327
Patch  17027533     : applied on Fri Mar 07 12:48:00 EST 2014
Unique Patch ID:  16677152
Patch description:  "Database Patch Set Update : 12.1.0.1.1 (17027533)"
   Created on 27 Sep 2013, 05:30:33 hrs PST8PDT
   Bugs fixed:
     17034172, 16694728, 16448848, 16863422, 16634384, 16465158, 16320173
     16313881, 16910734, 16816103, 16911800, 16715647, 16825779, 16707927
     16392068, 14197853, 16712618, 17273253, 16902138, 16524071, 16856570
     16465149, 16705020, 16689109, 16372203, 16864864, 16849982, 16946613
     16837842, 16964279, 16459685, 16978185, 16845022, 16195633, 14536110
     16964686, 16787973, 16850996, 16674842, 16838328, 16178562, 15996344
     16503473, 16842274, 16935643, 17000176, 14355775, 16362358, 16994576
     16485876, 16919176, 16928832, 16864359, 16617325, 16921340, 16679874
     16788832, 16483559, 16733884, 16784167, 16286774, 15986012, 16660558
     16674666, 16191248, 16697600, 16993424, 16946990, 16589507, 16173738
     16784143, 16772060, 16991789, 17346196, 16495802, 16859937, 16590848
     16910001, 16603924, 16427054, 16730813, 16227068, 16663303, 16784901
     16836849, 16186165, 16457621, 16007562, 16170787, 16663465, 16524968
     16543323, 17027533, 16675710, 17005047, 16795944, 16668226, 16070351
     16212405, 16523150, 16698577, 16621274, 16930325, 17330580, 16443657


Rac system comprising of multiple nodes
  Local node = askmrac1
  Remote node = askmrac2
  Remote node = askmrac3
--------------------------------------------------------------------------------
OPatch succeeded.
[oracle@askmrac1 ~]$
8. Issues and Resolution : 
Issue 1:
Don't have enough free space in the mount point where we have database home created. The opatch is looking for  min 5528.488MB free space in the oracle home mount point.
[root@askmrac1 askm]# export PATH=$PATH:/u01/app/12.1.0/grid_1/OPatch
[root@askmrac1 askm]# which opatchauto
/u01/app/12.1.0/grid_1/OPatch/opatchauto
[root@askmrac1 askm]# opatchauto apply /tmp/askm/17272829 -ocmrf /tmp/askm/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version        : 12.1.0.1.0
Running from       : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_09-46-11_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /tmp/askm/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Command "/u01/app/oracle/product/12.1.0/dbhome_1/OPatch/opatch napply -phBaseFile /tmp/OraDB12Home1_patchList -local  -invPtrLoc /u01/app/12.1.0/grid_1/oraInst.loc -oh /u01/app/oracle/product/12.1.0/dbhome_1 -silent -ocmrf /tmp/askm/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file Location for the failed command: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-07_09-51-07AM_1.log
[WARNING] The local database instance 'ORCL1' from '/u01/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any,  will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_09-46-11_deploy.debug.log".
Apply Summary:
Following patch(es) failed to be installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
opatchauto failed with error code 2.
[root@askmrac1 askm]#
Verification :
Verify the log file :

/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_09-46-11_deploy.log
COMMAND EXECUTION FAILURE :
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.1.0/grid_1/oraInst.loc
OPatch version    : 12.1.0.1.2
OUI version       : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-07_09-51-07AM_1.log
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(5528.488MB) is not available.
Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-07_09-51-07AM_1.log
OPatch failed with error code 73
Resolution : 
Make sure you have atleast 6GB of free space in database oracle home.


Issue 2:
Not able to stop the cluster stack even with force option during the patch application.
[root@askmrac1 askm]# opatchauto apply /tmp/askm/17272829 -ocmrf /tmp/askm/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version        : 12.1.0.1.0
Running from       : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_10-38-20_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /tmp/askm/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/tmp/askm/17272829/17027533" successfully applied to "/u01/app/oracle/product/12.1.0/dbhome_1".
Patch "/tmp/askm/17272829/17077442" successfully applied to "/u01/app/oracle/product/12.1.0/dbhome_1".
Stopping CRS ... Failed
Command "/usr/bin/perl /u01/app/12.1.0/grid_1/crs/install/rootcrs.pl -prepatch" execution failed:
Died at /u01/app/12.1.0/grid_1/crs/install/crspatch.pm line 609.

[WARNING] The local database instance 'ORCL1' from '/u01/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any,  will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_10-38-20_deploy.debug.log".
Apply Summary:
Following patch(es) are successfully installed:
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
Following patch(es) failed to be installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
opatchauto failed with error code 2.
[root@askmrac1 askm]#
Verification :
From Log file :

/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_10-38-20_deploy.log
COMMAND EXECUTION FAILURE :
Using configuration parameter file: /u01/app/12.1.0/grid_1/crs/install/crsconfig_params
Oracle Clusterware active version on the cluster is [12.1.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].
2014/03/07 10:49:18 CLSRSC-349: The Oracle Clusterware stack failed to stop
^[[0m
ERROR:
Died at /u01/app/12.1.0/grid_1/crs/install/crspatch.pm line 609.
From the logfile:
/u01/app/12.1.0/grid_1/cfgtoollogs/crsconfig/crspatch_askmrac1_2014-03-07_10-46-59AM.log
>  CRS-1151: The cluster was successfully set to rolling patch mode.
>End Command output
2014-03-07 10:47:17: Successfully set the cluster in rolling patch mode
2014-03-07 10:47:17: Stopping Oracle Clusterware stack ...
2014-03-07 10:47:17: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl stop crs -f
2014-03-07 10:49:18: Command output:
>  CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmrac1'
>  CRS-2673: Attempting to stop 'ora.crsd' on 'askmrac1'
>  CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmrac1'
>  CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'askmrac1'
>  CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmrac1'
>  CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmrac1'
>  CRS-2673: Attempting to stop 'ora.oc4j' on 'askmrac1'
>  CRS-2673: Attempting to stop 'ora.cvu' on 'askmrac1'
>  CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'askmrac1'
>  CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmrac1' succeeded
>  CRS-2673: Attempting to stop 'ora.askmrac1.vip' on 'askmrac1'
>  CRS-2677: Stop of 'ora.MGMTLSNR' on 'askmrac1' succeeded
>  CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'askmrac3'
>  CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmrac1' succeeded
>  CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmrac1'
>  CRS-2677: Stop of 'ora.askmrac1.vip' on 'askmrac1' succeeded
>  CRS-2672: Attempting to start 'ora.askmrac1.vip' on 'askmrac3'
>  CRS-2677: Stop of 'ora.scan1.vip' on 'askmrac1' succeeded
>  CRS-2672: Attempting to start 'ora.scan1.vip' on 'askmrac3'
>  CRS-2676: Start of 'ora.askmrac1.vip' on 'askmrac3' succeeded
>  CRS-2676: Start of 'ora.scan1.vip' on 'askmrac3' succeeded
>  CRS-2677: Stop of 'ora.cvu' on 'askmrac1' succeeded
>  CRS-2672: Attempting to start 'ora.cvu' on 'askmrac2'
>  CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'askmrac3'
>  CRS-2676: Start of 'ora.cvu' on 'askmrac2' succeeded
>  CRS-2676: Start of 'ora.MGMTLSNR' on 'askmrac3' succeeded
>  CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'askmrac3' succeeded
>  CRS-2677: Stop of 'ora.oc4j' on 'askmrac1' succeeded
>  CRS-2672: Attempting to start 'ora.oc4j' on 'askmrac2'
>  CRS-2675: Stop of 'ora.OCR_VOTE.dg' on 'askmrac1' failed
>  CRS-2679: Attempting to clean 'ora.OCR_VOTE.dg' on 'askmrac1'
>  CRS-2678: 'ora.OCR_VOTE.dg' on 'askmrac1' has experienced an unrecoverable failure
>  CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'askmrac1'
>  CRS-2675: Stop of 'ora.OCR_VOTE.dg' on 'askmrac1' failed
>  CRS-2679: Attempting to clean 'ora.OCR_VOTE.dg' on 'askmrac1'
>  CRS-2676: Start of 'ora.oc4j' on 'askmrac2' succeeded
>  CRS-2678: 'ora.OCR_VOTE.dg' on 'askmrac1' has experienced an unrecoverable failure
>  CRS-2799: Failed to shut down resource 'ora.OCR_VOTE.dg' on 'askmrac1'
>  CRS-2799: Failed to shut down resource 'ora.askmrac1.ASM1.asm' on 'askmrac1'
>  CRS-2799: Failed to shut down resource 'ora.asm' on 'askmrac1'
>  CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'askmrac1' has failed
>  CRS-2675: Stop of 'ora.crsd' on 'askmrac1' failed
>  CRS-2799: Failed to shut down resource 'ora.crsd' on 'askmrac1'
>  CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'askmrac1' has failed
>  CRS-4687: Shutdown command has completed with errors.
>  CRS-4000: Command Stop failed, or completed with errors.
Resolution : 
The issue seems to be intermittent and environment specific. After some research, i found that there is some problem with the disk group where i have my OCR and Voting files. Restarting the node resolved the issue.

Issue 3:
Don't have enough free space in the mount point where we have grid infra home created. The opatch is looking for  min 10578.277MB free space in the grid home mount point.
[root@askmrac1 askm]# opatchauto apply /tmp/askm/17272829 -ocmrf /tmp/askm/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version        : 12.1.0.1.0
Running from       : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_14-38-30_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /tmp/askm/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/tmp/askm/17272829/17027533" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Patch "/tmp/askm/17272829/17077442" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Stopping CRS ... Successful
Applying patch(es) to "/u01/app/12.1.0/grid_1" ...
Command "/u01/app/12.1.0/grid_1/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_patchList -local  -invPtrLoc /u01/app/12.1.0/grid_1/oraInst.loc -oh /u01/app/12.1.0/grid_1 -silent -ocmrf /tmp/askm/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file Location for the failed command: /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-07_14-43-51PM_1.log
[WARNING] The local database instance 'ORCL1' from '/u01/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any,  will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_14-38-30_deploy.debug.log".
Apply Summary:
opatchauto ran into some warnings during patch installation (Please see log file for details):
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
Following patch(es) failed to be installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
opatchauto failed with error code 2.
[root@askmrac1 askm]# 

Verification :
From the logfile :

/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_14-38-30_deploy.log
COMMAND EXECUTION FAILURE :
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/12.1.0/grid_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/12.1.0/grid_1/oraInst.loc
OPatch version    : 12.1.0.1.2
OUI version       : 12.1.0.1.0
Log file location : /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-07_14-43-51PM_1.log
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(10578.277MB) is not available.
Log file location: /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-07_14-43-51PM_1.log
OPatch failed with error code 73
ERROR:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Resolution : 
Make sure you have enough free space ( ie min 11GB) in the oracle home where you have installed grid home.

Issue 4:
Permission Issue on the patch source files where it is unzipped.
[root@askmrac1 ~]# opatchauto apply /mnt/software/RAC/1201_PSU/17272829 -ocmrf /tmp/askm/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version        : 12.1.0.1.0
Running from       : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-08_03-28-09_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /mnt/software/RAC/1201_PSU/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/mnt/software/RAC/1201_PSU/17272829/17027533" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Patch "/mnt/software/RAC/1201_PSU/17272829/17077442" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Stopping CRS ... Successful
Applying patch(es) to "/u01/app/12.1.0/grid_1" ...
Command "/u01/app/12.1.0/grid_1/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_patchList -local  -invPtrLoc /u01/app/12.1.0/grid_1/oraInst.loc -oh /u01/app/12.1.0/grid_1 -silent -ocmrf /tmp/askm/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckApplicable" failed.
Log file Location for the failed command: /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-08_03-33-17AM_1.log
[WARNING] The local database instance 'ORCL1' from '/u01/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any,  will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-08_03-28-09_deploy.debug.log".
Apply Summary:
opatchauto ran into some warnings during patch installation (Please see log file for details):
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
Following patch(es) failed to be installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
opatchauto failed with error code 2.
[root@askmrac1 ~]#

Verification :
From the logfile :

/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-08_03-28-09_deploy.log
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckApplicable" failed.
The details are:
Patch 17077442:
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/bin/appvipcfg.pl" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'appvipcfg.pl' to '/u01/app/12.1.0/grid_1/bin/appvipcfg.pl'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/bin/oclumon.bin" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'oclumon.bin' to '/u01/app/12.1.0/grid_1/bin/oclumon.bin'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/bin/ologgerd" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'ologgerd' to '/u01/app/12.1.0/grid_1/bin/ologgerd'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/bin/osysmond.bin" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'osysmond.bin' to '/u01/app/12.1.0/grid_1/bin/osysmond.bin'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/crs/demo/coldfailover/act_db.pl" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'act_db.pl' to '/u01/app/12.1.0/grid_1/crs/demo/coldfailover/act_db.pl'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/crs/demo/coldfailover/act_listener.pl" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'act_listener.pl' to '/u01/app/12.1.0/grid_1/crs/demo/coldfailover/act_listener.pl'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/crs/demo/coldfailover/act_resgroup.pl" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'act_resgroup.pl' to '/u01/app/12.1.0/grid_1/crs/demo/coldfailover/act_resgroup.pl'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/crs/demo/demoActionScript" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'demoActionScript' to '/u01/app/12.1.0/grid_1/crs/demo/demoActionScript'

Log file location: /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-08_03-33-17AM_1.log
OPatch failed with error code 73
Resolution : 
Given 777 permission to the directory where we have unzipped PSU patch.

Logfile Location For PSU Patch : 
/u01/app/12.1.0/grid_1/cfgtoollogs/opatch/
/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/

Important Note :
Whenever you have some issue while applying the patch and there were some services shutdown as part of patch application and these services still remain down after the occurance of the error. So when you start applying the patch again after resolving the issue, make sure you have started all the cluster stack services and database services on that node.
Use the following commands to start services ..
crsctl start has  or crsctl start cluster ( Don't user -all option. It is required to start services on local node only)
srvctl start instance -d ORCL -i ORCL1

Hope This Helps
SRI

Share this article :

Related Articles By Category



Post a Comment

Thank you for visiting our site and leaving your valuable comment.

 
Support :
Copyright © 2013. askMLabs - All Rights Reserved
Proudly powered by Blogger