RAC : Apply PSU Patch To Standard Cluster 12c

In this article, we will see how to apply PSU patch to the standard cluster. I have a RAC 12c standard cluster installed with 3 nodes.

Environment Details :

RAC Version Cluster Type	12c 12.1.0.1.0 Standard Cluster
Cluster Nodes	askmrac1/2/3
DB Running on	All Cluster Nodes
Task	Applying OCT2013 PSU Patch

Environment Configuration :

Type	Path	Owner	Version	Shared
Grid Infra Home	/u01/app/12.1.0/grid_1	Grid	12.1.0.1	False
Database Home	/u01/app/oracle/product/12.1.0/dbhome_1	Oracle	12.1.0.1	Flase

Step By Step

1. Download and Unzip the Latest OPatch to all cluster nodes
2. Validate and Record Pre-Patch information
3. Create OCM response file if it does not exist
4. Download and Unzip the JAN2014 PSU patch for GI 12c ie 12.1.0.1.2
5. One-off patch Conflicts detection and Resolution
6. Patch application
7. Verification of Patch application
8. Issues and Resolutions

1. Download and Unzip the Latest OPatch to all cluster nodes:

You must use the OPatch utility version12.1.0.1.1 or later to apply this patch. Oracle recommends that you use the latest released OPatch for 12.1 releases, which is available for download from My Oracle Support patch 6880880.

It is always a best practice to keep both GRID_HOME and DATABASE HOME at same opatch level. So change the opatch in DATABASE HOME also.

Repeat this step to all the nodes in the cluster.

2. Validate and Record Pre-Patch information :
Validate using the following commands :
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/12.1.0/grid/
$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/app/oracle/product/12.1.0/dbhome_1

Login to each node in RAC as grid user and execute the following command.
$GRID_ORACLE_HOME/OPatch/opatch lsinventory
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Login to each node in RAC as oracle user and execute the following command.
$ORACLE_HOME/OPatch/opatch lsinventory
$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Connect to each instance and record registry information.
SQL> select comp_name,version,status from dba_registry;

3. Create OCM Response File If It Does Not Exist :
Create ocm response file using the following command and provide appropriate values for the prompts.

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp

Verify the created file using,

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp –verbose ocm.rsp

NOTE: The Opatch utility will prompt for your OCM (Oracle Configuration Manager) response file when it is run. Without which we cant proceed further.

Copy this response file ocm.rsp to all the nodes in the cluster to the same location. Or you can create a new response file on each node using the same method above.

4. Download and Unzip the JAN2014 PSU patch : (as grid user)

Patch 17272829 is the OCT2013 PSU patch. It is downloaded and unzipped to the locaiton "/mnt/software/RAC/1201_PSU/"

5. One-off patch Conflicts detection and Resolution :

Determine whether any currently installed one-off patches conflict with the PSU patch.

$GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/17272829 analyze

But i don't have any patches applied to my home yet. So i can ignore this step. But if you have any conflicts identified in GI home or in DB home, follow MOS ID 1061295.1 to resolve the conflicts.

6. Patch Application :

Now patch can be applied using the following syntax.

# opatchauto apply <PATH_TO_PATCH_DIRECTORY> -ocmrf <ocm response file>

On all Cluster nodes ( askmrac1/2/3)

[root@askmrac1 ~]# export PATH=/u01/app/12.1.0/grid/OPatch:$PATH
[root@askmrac1 ~]# which opatchauto
/u01/app/12.1.0/grid/OPatch/opatchauto
[root@askmrac1 ~]# opatchauto apply /mnt/software/RAC/1201_PSU/17272829 -ocmrf /tmp/askm/ocm.rspOPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-08_04-17-23_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /mnt/software/RAC/1201_PSU/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/mnt/software/RAC/1201_PSU/17272829/17027533" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Patch "/mnt/software/RAC/1201_PSU/17272829/17077442" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Stopping CRS ... Successful
Applying patch(es) to "/u01/app/12.1.0/grid_1" ...
Patch "/mnt/software/RAC/1201_PSU/17272829/17027533" successfully applied to "/u01/app/12.1.0/grid_1".
Patch "/mnt/software/RAC/1201_PSU/17272829/17077442" successfully applied to "/u01/app/12.1.0/grid_1".
Patch "/mnt/software/RAC/1201_PSU/17272829/17303297" successfully applied to "/u01/app/12.1.0/grid_1".
Starting CRS ... Successful
Starting RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
[WARNING] SQL changes, if any, could not be applied on the following database(s): ORCL ... Please refer to the log file for more details.
Apply Summary:
Following patch(es) are successfully installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
opatchauto ran into some warnings during patch installation (Please see log file for details):
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
opatchauto succeeded.
[root@askmrac1 ~]#

Repeat the above step to all the cluster nodes in the standard cluster.

7. Verification of Patch application :

$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/12.1.0/grid/
$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/app/oracle/product/12.1.0/dbhome_1

Login to each node in RAC as grid user and execute the following command.
$GRID_ORACLE_HOME/OPatch/opatch lsinventory
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Login to each node in RAC as oracle user and execute the following command.
$ORACLE_HOME/OPatch/opatch lsinventory
$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Connect to each instance and verify registry information.
SQL> select comp_name,version,status from dba_registry;

[root@askmrac1 ~]# su - grid
[grid@askmrac1 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/12.1.0/grid_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/12.1.0/grid_1/oraInst.loc
OPatch version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Log file location : /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-08_04-46-22AM_1.log
Lsinventory Output file location : /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/lsinv/lsinventory2014-03-08_04-46-22AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.1.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (3) :
Patch 17303297 : applied on Sat Mar 08 04:35:19 EST 2014
Unique Patch ID: 16881795
Patch description: "ACFS Patch Set Update 12.1.0.1.1"
Created on 14 Oct 2013, 07:25:50 hrs US/Central
Bugs fixed:
14487556, 16398970, 16552813, 16930184, 16420645, 16170117, 16436434
16476044, 16458315, 16463033, 16095100, 16545876, 16429953, 14826673
16001893, 16482869, 16371746, 16435343, 14476443, 16294308, 16671486
16386110, 15978267, 16085530, 16347837, 16814544, 16022372, 16167084
14510092, 16450287, 16399406
Patch 17077442 : applied on Sat Mar 08 04:31:12 EST 2014
Unique Patch ID: 16881794
Patch description: "Oracle Clusterware Patch Set Update 12.1.0.1.1"
Created on 12 Oct 2013, 06:33:53 hrs US/Central
Bugs fixed:
16505840, 16505255, 16390989, 16399322, 16505617, 16505717, 17486244
16168869, 16444109, 16505361, 13866165, 16505763, 16208257, 16904822
17299876, 16246222, 16505214, 16505540, 15936039, 16580269, 16838292
16505449, 16801843, 16309853, 16505395, 17507349, 17475155, 16493242
17039197, 16196609, 17463260, 16505667, 15970176, 16488665, 16670327
Patch 17027533 : applied on Sat Mar 08 04:27:54 EST 2014
Unique Patch ID: 16677152
Patch description: "Database Patch Set Update : 12.1.0.1.1 (17027533)"
Created on 27 Sep 2013, 05:30:33 hrs PST8PDT
Bugs fixed:
17034172, 16694728, 16448848, 16863422, 16634384, 16465158, 16320173
16313881, 16910734, 16816103, 16911800, 16715647, 16825779, 16707927
16392068, 14197853, 16712618, 17273253, 16902138, 16524071, 16856570
16465149, 16705020, 16689109, 16372203, 16864864, 16849982, 16946613
16837842, 16964279, 16459685, 16978185, 16845022, 16195633, 14536110
16964686, 16787973, 16850996, 16674842, 16838328, 16178562, 15996344
16503473, 16842274, 16935643, 17000176, 14355775, 16362358, 16994576
16485876, 16919176, 16928832, 16864359, 16617325, 16921340, 16679874
16788832, 16483559, 16733884, 16784167, 16286774, 15986012, 16660558
16674666, 16191248, 16697600, 16993424, 16946990, 16589507, 16173738
16784143, 16772060, 16991789, 17346196, 16495802, 16859937, 16590848
16910001, 16603924, 16427054, 16730813, 16227068, 16663303, 16784901
16836849, 16186165, 16457621, 16007562, 16170787, 16663465, 16524968
16543323, 17027533, 16675710, 17005047, 16795944, 16668226, 16070351
16212405, 16523150, 16698577, 16621274, 16930325, 17330580, 16443657

Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
1650217826 askmrac1
0 askmrac3,askmrac2
--------------------------------------------------------------------------------
OPatch succeeded.
[grid@askmrac1 ~]$

[root@askmrac1 ~]# su - oracle
[oracle@askmrac1 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-08_04-46-46AM_1.log
Lsinventory Output file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2014-03-08_04-46-46AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Database 12c 12.1.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (2) :
Patch 17077442 : applied on Fri Mar 07 12:48:25 EST 2014
Unique Patch ID: 16881794
Patch description: "Oracle Clusterware Patch Set Update 12.1.0.1.1"
Created on 12 Oct 2013, 06:33:53 hrs US/Central
Bugs fixed:
16505840, 16505255, 16390989, 16399322, 16505617, 16505717, 17486244
16168869, 16444109, 16505361, 13866165, 16505763, 16208257, 16904822
17299876, 16246222, 16505214, 16505540, 15936039, 16580269, 16838292
16505449, 16801843, 16309853, 16505395, 17507349, 17475155, 16493242
17039197, 16196609, 17463260, 16505667, 15970176, 16488665, 16670327
Patch 17027533 : applied on Fri Mar 07 12:48:00 EST 2014
Unique Patch ID: 16677152
Patch description: "Database Patch Set Update : 12.1.0.1.1 (17027533)"
Created on 27 Sep 2013, 05:30:33 hrs PST8PDT
Bugs fixed:
17034172, 16694728, 16448848, 16863422, 16634384, 16465158, 16320173
16313881, 16910734, 16816103, 16911800, 16715647, 16825779, 16707927
16392068, 14197853, 16712618, 17273253, 16902138, 16524071, 16856570
16465149, 16705020, 16689109, 16372203, 16864864, 16849982, 16946613
16837842, 16964279, 16459685, 16978185, 16845022, 16195633, 14536110
16964686, 16787973, 16850996, 16674842, 16838328, 16178562, 15996344
16503473, 16842274, 16935643, 17000176, 14355775, 16362358, 16994576
16485876, 16919176, 16928832, 16864359, 16617325, 16921340, 16679874
16788832, 16483559, 16733884, 16784167, 16286774, 15986012, 16660558
16674666, 16191248, 16697600, 16993424, 16946990, 16589507, 16173738
16784143, 16772060, 16991789, 17346196, 16495802, 16859937, 16590848
16910001, 16603924, 16427054, 16730813, 16227068, 16663303, 16784901
16836849, 16186165, 16457621, 16007562, 16170787, 16663465, 16524968
16543323, 17027533, 16675710, 17005047, 16795944, 16668226, 16070351
16212405, 16523150, 16698577, 16621274, 16930325, 17330580, 16443657

Rac system comprising of multiple nodes
Local node = askmrac1
Remote node = askmrac2
Remote node = askmrac3
--------------------------------------------------------------------------------
OPatch succeeded.
[oracle@askmrac1 ~]$

8. Issues and Resolution :

Issue 1:
Don't have enough free space in the mount point where we have database home created. The opatch is looking for min 5528.488MB free space in the oracle home mount point.

[root@askmrac1 askm]# export PATH=$PATH:/u01/app/12.1.0/grid_1/OPatch
[root@askmrac1 askm]# which opatchauto
/u01/app/12.1.0/grid_1/OPatch/opatchauto
[root@askmrac1 askm]# opatchauto apply /tmp/askm/17272829 -ocmrf /tmp/askm/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_09-46-11_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /tmp/askm/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Command "/u01/app/oracle/product/12.1.0/dbhome_1/OPatch/opatch napply -phBaseFile /tmp/OraDB12Home1_patchList -local -invPtrLoc /u01/app/12.1.0/grid_1/oraInst.loc -oh /u01/app/oracle/product/12.1.0/dbhome_1 -silent -ocmrf /tmp/askm/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file Location for the failed command: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-07_09-51-07AM_1.log
[WARNING] The local database instance 'ORCL1' from '/u01/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any, will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_09-46-11_deploy.debug.log".
Apply Summary:
Following patch(es) failed to be installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
opatchauto failed with error code 2.
[root@askmrac1 askm]#

Verification :
Verify the log file :

/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_09-46-11_deploy.log

COMMAND EXECUTION FAILURE :
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/12.1.0/grid_1/oraInst.loc
OPatch version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-07_09-51-07AM_1.log
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(5528.488MB) is not available.
Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2014-03-07_09-51-07AM_1.log
OPatch failed with error code 73

Resolution :

Make sure you have atleast 6GB of free space in database oracle home.

Issue 2:
Not able to stop the cluster stack even with force option during the patch application.

[root@askmrac1 askm]# opatchauto apply /tmp/askm/17272829 -ocmrf /tmp/askm/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_10-38-20_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /tmp/askm/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/tmp/askm/17272829/17027533" successfully applied to "/u01/app/oracle/product/12.1.0/dbhome_1".
Patch "/tmp/askm/17272829/17077442" successfully applied to "/u01/app/oracle/product/12.1.0/dbhome_1".
Stopping CRS ... Failed
Command "/usr/bin/perl /u01/app/12.1.0/grid_1/crs/install/rootcrs.pl -prepatch" execution failed:
Died at /u01/app/12.1.0/grid_1/crs/install/crspatch.pm line 609.

[WARNING] The local database instance 'ORCL1' from '/u01/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any, will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_10-38-20_deploy.debug.log".
Apply Summary:
Following patch(es) are successfully installed:
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
Following patch(es) failed to be installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
opatchauto failed with error code 2.
[root@askmrac1 askm]#

Verification :

From Log file :

/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_10-38-20_deploy.log

COMMAND EXECUTION FAILURE :
Using configuration parameter file: /u01/app/12.1.0/grid_1/crs/install/crsconfig_params
Oracle Clusterware active version on the cluster is [12.1.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].
2014/03/07 10:49:18 CLSRSC-349: The Oracle Clusterware stack failed to stop
^[[0m
ERROR:
Died at /u01/app/12.1.0/grid_1/crs/install/crspatch.pm line 609.

From the logfile:
/u01/app/12.1.0/grid_1/cfgtoollogs/crsconfig/crspatch_askmrac1_2014-03-07_10-46-59AM.log

> CRS-1151: The cluster was successfully set to rolling patch mode.
>End Command output
2014-03-07 10:47:17: Successfully set the cluster in rolling patch mode
2014-03-07 10:47:17: Stopping Oracle Clusterware stack ...
2014-03-07 10:47:17: Executing cmd: /u01/app/12.1.0/grid_1/bin/crsctl stop crs -f
2014-03-07 10:49:18: Command output:
> CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmrac1'
> CRS-2673: Attempting to stop 'ora.crsd' on 'askmrac1'
> CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmrac1'
> CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'askmrac1'
> CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmrac1'
> CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmrac1'
> CRS-2673: Attempting to stop 'ora.oc4j' on 'askmrac1'
> CRS-2673: Attempting to stop 'ora.cvu' on 'askmrac1'
> CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'askmrac1'
> CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmrac1' succeeded
> CRS-2673: Attempting to stop 'ora.askmrac1.vip' on 'askmrac1'
> CRS-2677: Stop of 'ora.MGMTLSNR' on 'askmrac1' succeeded
> CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'askmrac3'
> CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmrac1' succeeded
> CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmrac1'
> CRS-2677: Stop of 'ora.askmrac1.vip' on 'askmrac1' succeeded
> CRS-2672: Attempting to start 'ora.askmrac1.vip' on 'askmrac3'
> CRS-2677: Stop of 'ora.scan1.vip' on 'askmrac1' succeeded
> CRS-2672: Attempting to start 'ora.scan1.vip' on 'askmrac3'
> CRS-2676: Start of 'ora.askmrac1.vip' on 'askmrac3' succeeded
> CRS-2676: Start of 'ora.scan1.vip' on 'askmrac3' succeeded
> CRS-2677: Stop of 'ora.cvu' on 'askmrac1' succeeded
> CRS-2672: Attempting to start 'ora.cvu' on 'askmrac2'
> CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'askmrac3'
> CRS-2676: Start of 'ora.cvu' on 'askmrac2' succeeded
> CRS-2676: Start of 'ora.MGMTLSNR' on 'askmrac3' succeeded
> CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'askmrac3' succeeded
> CRS-2677: Stop of 'ora.oc4j' on 'askmrac1' succeeded
> CRS-2672: Attempting to start 'ora.oc4j' on 'askmrac2'
> CRS-2675: Stop of 'ora.OCR_VOTE.dg' on 'askmrac1' failed
> CRS-2679: Attempting to clean 'ora.OCR_VOTE.dg' on 'askmrac1'
> CRS-2678: 'ora.OCR_VOTE.dg' on 'askmrac1' has experienced an unrecoverable failure
> CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'askmrac1'
> CRS-2675: Stop of 'ora.OCR_VOTE.dg' on 'askmrac1' failed
> CRS-2679: Attempting to clean 'ora.OCR_VOTE.dg' on 'askmrac1'
> CRS-2676: Start of 'ora.oc4j' on 'askmrac2' succeeded
> CRS-2678: 'ora.OCR_VOTE.dg' on 'askmrac1' has experienced an unrecoverable failure
> CRS-2799: Failed to shut down resource 'ora.OCR_VOTE.dg' on 'askmrac1'
> CRS-2799: Failed to shut down resource 'ora.askmrac1.ASM1.asm' on 'askmrac1'
> CRS-2799: Failed to shut down resource 'ora.asm' on 'askmrac1'
> CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'askmrac1' has failed
> CRS-2675: Stop of 'ora.crsd' on 'askmrac1' failed
> CRS-2799: Failed to shut down resource 'ora.crsd' on 'askmrac1'
> CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'askmrac1' has failed
> CRS-4687: Shutdown command has completed with errors.
> CRS-4000: Command Stop failed, or completed with errors.

Resolution :
The issue seems to be intermittent and environment specific. After some research, i found that there is some problem with the disk group where i have my OCR and Voting files. Restarting the node resolved the issue.

Issue 3:
Don't have enough free space in the mount point where we have grid infra home created. The opatch is looking for min 10578.277MB free space in the grid home mount point.

[root@askmrac1 askm]# opatchauto apply /tmp/askm/17272829 -ocmrf /tmp/askm/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_14-38-30_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /tmp/askm/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/tmp/askm/17272829/17027533" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Patch "/tmp/askm/17272829/17077442" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Stopping CRS ... Successful
Applying patch(es) to "/u01/app/12.1.0/grid_1" ...
Command "/u01/app/12.1.0/grid_1/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_patchList -local -invPtrLoc /u01/app/12.1.0/grid_1/oraInst.loc -oh /u01/app/12.1.0/grid_1 -silent -ocmrf /tmp/askm/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.
Log file Location for the failed command: /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-07_14-43-51PM_1.log
[WARNING] The local database instance 'ORCL1' from '/u01/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any, will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_14-38-30_deploy.debug.log".
Apply Summary:
opatchauto ran into some warnings during patch installation (Please see log file for details):
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
Following patch(es) failed to be installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
opatchauto failed with error code 2.
[root@askmrac1 askm]#

Verification :

From the logfile :

/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-07_14-38-30_deploy.log

COMMAND EXECUTION FAILURE :
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/12.1.0/grid_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/12.1.0/grid_1/oraInst.loc
OPatch version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Log file location : /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-07_14-43-51PM_1.log
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemSpace" failed.
The details are:
Required amount of space(10578.277MB) is not available.
Log file location: /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-07_14-43-51PM_1.log
OPatch failed with error code 73
ERROR:
UtilSession failed:
Prerequisite check "CheckSystemSpace" failed.

Resolution :
Make sure you have enough free space ( ie min 11GB) in the oracle home where you have installed grid home.

Issue 4:
Permission Issue on the patch source files where it is unzipped.

[root@askmrac1 ~]# opatchauto apply /mnt/software/RAC/1201_PSU/17272829 -ocmrf /tmp/askm/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /u01/app/12.1.0/grid_1
opatchauto log file: /u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-08_03-28-09_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid_1
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /mnt/software/RAC/1201_PSU/17272829
Grid Infrastructure Patch(es): 17027533 17077442 17303297
RAC Patch(es): 17027533 17077442
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/mnt/software/RAC/1201_PSU/17272829/17027533" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Patch "/mnt/software/RAC/1201_PSU/17272829/17077442" is already installed on "/u01/app/oracle/product/12.1.0/dbhome_1". Please rollback the existing identical patch first.
Stopping CRS ... Successful
Applying patch(es) to "/u01/app/12.1.0/grid_1" ...
Command "/u01/app/12.1.0/grid_1/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_patchList -local -invPtrLoc /u01/app/12.1.0/grid_1/oraInst.loc -oh /u01/app/12.1.0/grid_1 -silent -ocmrf /tmp/askm/ocm.rsp" execution failed:
UtilSession failed:
Prerequisite check "CheckApplicable" failed.
Log file Location for the failed command: /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-08_03-33-17AM_1.log
[WARNING] The local database instance 'ORCL1' from '/u01/app/oracle/product/12.1.0/dbhome_1' is not running. SQL changes, if any, will not be applied. Please refer to the log file for more details.
For more details, please refer to the log file "/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-08_03-28-09_deploy.debug.log".
Apply Summary:
opatchauto ran into some warnings during patch installation (Please see log file for details):
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17027533, 17077442
Following patch(es) failed to be installed:
GI Home: /u01/app/12.1.0/grid_1: 17027533, 17077442, 17303297
opatchauto failed with error code 2.
[root@askmrac1 ~]#

Verification :

From the logfile :

/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/17272829/opatch_gi_2014-03-08_03-28-09_deploy.log

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckApplicable" failed.
The details are:
Patch 17077442:
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/bin/appvipcfg.pl" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'appvipcfg.pl' to '/u01/app/12.1.0/grid_1/bin/appvipcfg.pl'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/bin/oclumon.bin" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'oclumon.bin' to '/u01/app/12.1.0/grid_1/bin/oclumon.bin'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/bin/ologgerd" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'ologgerd' to '/u01/app/12.1.0/grid_1/bin/ologgerd'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/bin/osysmond.bin" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'osysmond.bin' to '/u01/app/12.1.0/grid_1/bin/osysmond.bin'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/crs/demo/coldfailover/act_db.pl" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'act_db.pl' to '/u01/app/12.1.0/grid_1/crs/demo/coldfailover/act_db.pl'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/crs/demo/coldfailover/act_listener.pl" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'act_listener.pl' to '/u01/app/12.1.0/grid_1/crs/demo/coldfailover/act_listener.pl'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/crs/demo/coldfailover/act_resgroup.pl" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'act_resgroup.pl' to '/u01/app/12.1.0/grid_1/crs/demo/coldfailover/act_resgroup.pl'
Copy Action: Source File "/mnt/software/RAC/1201_PSU/17272829/17077442/files/crs/demo/demoActionScript" does not exists or is not readable
'oracle.crs, 12.1.0.1.0': Cannot copy file from 'demoActionScript' to '/u01/app/12.1.0/grid_1/crs/demo/demoActionScript'
Log file location: /u01/app/12.1.0/grid_1/cfgtoollogs/opatch/opatch2014-03-08_03-33-17AM_1.log
OPatch failed with error code 73

Resolution :
Given 777 permission to the directory where we have unzipped PSU patch.

Logfile Location For PSU Patch :
/u01/app/12.1.0/grid_1/cfgtoollogs/opatch/
/u01/app/12.1.0/grid_1/cfgtoollogs/opatchauto/

Important Note :
Whenever you have some issue while applying the patch and there were some services shutdown as part of patch application and these services still remain down after the occurance of the error. So when you start applying the patch again after resolving the issue, make sure you have started all the cluster stack services and database services on that node.
Use the following commands to start services ..
crsctl start has or crsctl start cluster ( Don't user -all option. It is required to start services on local node only)
srvctl start instance -d ORCL -i ORCL1

Hope This Helps
SRI

+ comments + 2 comments

INFYCLE TECHNOLOGIES

December 14, 2021 at 5:34 AM

Don’t follow your role model. Be the Role model person for others. But it's so simple by getting Hadoop training in Chennai. Because it is an assurance course to bounce back from a double salary. For joining call 7502633633.

Anonymous

November 19, 2022 at 2:48 AM

For instance, you've have} only a 1 – 36 or 1 – 37 odds of successful. Thankfully, in Roulette, the table provides numerous other betting options. Live dealer on line casino video games, but you’ll additionally discover a wealth 카지노 of knowledge in our blog posts to assist get you began. Straight Up — Betting on a single number including the ‘0’. On American roulette wheels, this includes the option to guess on the ‘00’.

askMLabs