askMLabs

Showing posts with label leaf node. Show all posts

2:25 PM

RAC 12c : JAN2014 PSU To 4 Node Flex Cluster

Written By askMLabs on Sunday, March 16, 2014 | 2:25 PM

In this article, we will see how to apply PSU patch to the flex cluster. I have a RAC 12c flex cluster installed with 4 nodes. 3nodes being hub nodes and 1 node being leaf node.
Environment Details :

RAC Version Cluster Type	12c 12.1.0.1.0 Flex Cluster
Hub Nodes	rac12cnode1/2/3
Leaf Nodes	rac12cnode4
DB Running on	All Hub Nodes
Task	Applying JAN2014 PSU Patch

Environment Configuration :

Type	Path	Owner	Version	Shared
Grid Infra Home	/u01/app/12.1.0/grid	Grid	12.1.0.1	False
Database Home	/u01/app/oracle/product/12.1.0/dbhome_1	Oracle	12.1.0.1	Flase

[root@rac12cnode1 ~]# su - grid
[grid@rac12cnode1 ~]$ olsnodes -s -t
rac12cnode1 Active Unpinned
rac12cnode2 Active Unpinned
rac12cnode3 Active Unpinned
rac12cnode4 Active Unpinned
[grid@rac12cnode1 ~]$ crsctl get node role config -all
Node 'rac12cnode1' configured role is 'hub'
Node 'rac12cnode2' configured role is 'hub'
Node 'rac12cnode3' configured role is 'hub'
Node 'rac12cnode4' configured role is 'leaf'
[grid@rac12cnode1 ~]$ crsctl get node role status -all
Node 'rac12cnode1' active role is 'hub'
Node 'rac12cnode2' active role is 'hub'
Node 'rac12cnode3' active role is 'hub'
Node 'rac12cnode4' active role is 'leaf'
[grid@rac12cnode1 ~]$

Step By Step

1. Download and Unzip the Latest OPatch to all cluster nodes
2. Validate and Record Pre-Patch information
3. Create OCM response file if it does not exist
4. Download and Unzip the JAN2014 PSU patch for GI 12c ie 12.1.0.1.2
5. One-off patch Conflicts detection and Resolution
6. Patch application
7. Verification of Patch application
8. Issues and Resolutions

1. Download and Unzip the Latest OPatch to all cluster nodes:

You must use the OPatch utility version12.1.0.1.1 or later to apply this patch. Oracle recommends that you use the latest released OPatch for 12.1 releases, which is available for download from My Oracle Support patch 6880880.

[grid@rac12cnode1 1201_PSU]$ cd $ORACLE_HOME[grid@rac12cnode1 grid]$ ls -ld OPatchdrwxr-xr-x. 7 grid oinstall 4096 Mar 7 08:36 OPatch
[grid@rac12cnode1 grid]$ which opatch/u01/app/12.1.0/grid/OPatch/opatch
[grid@rac12cnode1 grid]$ opatch versionOPatch Version: 12.1.0.1.0
OPatch succeeded.
[grid@rac12cnode1 grid]$ pwd
/u01/app/12.1.0/grid
[grid@rac12cnode1 grid]$ mv OPatch OPatch_bakmv: cannot move `OPatch' to `OPatch_bak': Permission denied
[grid@rac12cnode1 grid]$ exit
logout
[root@rac12cnode1 ~]# cd /u01/app/12.1.0/grid/[root@rac12cnode1 grid]# mv OPatch OPatch_bak[root@rac12cnode1 grid]# unzip /mnt/software/RAC/1201_PSU/p6880880_121010_Linux-x86-64.zipArchive: /mnt/software/RAC/1201_PSU/p6880880_121010_Linux-x86-64.zip
..
..
[root@rac12cnode1 grid]# ls -ld OPatchdrwxr-xr-x. 7 root root 4096 Oct 9 20:25 OPatch
[root@rac12cnode1 grid]# chown -R grid:oinstall OPatch[root@rac12cnode1 grid]# ls -ld OPatchdrwxr-xr-x. 7 grid oinstall 4096 Oct 9 20:25 OPatch
[root@rac12cnode1 grid]# pwd
/u01/app/12.1.0/grid
[root@rac12cnode1 grid]# cd OPatch[root@rac12cnode1 OPatch]# ls
datapatch docs jlib opatch opatch.bat opatch.pl operr operr_readme.txt README.txt
datapatch.bat emdpatch.pl ocm opatchauto opatch.ini opatchprereqs operr.bat oplan version.txt
[root@rac12cnode1 OPatch]# ./opatch versionOPatch Version: 12.1.0.1.2
OPatch succeeded.
[root@rac12cnode1 OPatch]#

It is always a best practice to keep both GRID_HOME and DATABASE HOME at same opatch level. So change the opatch in DATABASE HOME also.

[root@rac12cnode1 OPatch]# cd /u01/app/oracle/product/12.1.0/[root@rac12cnode1 12.1.0]# ls
dbhome_1
[root@rac12cnode1 12.1.0]# cd dbhome_1/[root@rac12cnode1 dbhome_1]# ls -ld OPatch
drwxr-xr-x. 7 oracle oinstall 4096 May 27 2013 OPatch
[root@rac12cnode1 dbhome_1]# mv OPatch OPatch_bak[root@rac12cnode1 dbhome_1]# unzip /mnt/software/RAC/1201_PSU/p6880880_121010_Linux-x86-64.zipArchive: /mnt/software/RAC/1201_PSU/p6880880_121010_Linux-x86-64.zip
..
..
[root@rac12cnode1 dbhome_1]# chown -R oracle:oinstall OPatch[root@rac12cnode1 dbhome_1]# cd Opatch
-bash: cd: Opatch: No such file or directory
[root@rac12cnode1 dbhome_1]# ./opatch version
-bash: ./opatch: No such file or directory
[root@rac12cnode1 dbhome_1]# cd OPatch
[root@rac12cnode1 OPatch]# ./opatch versionOPatch Version: 12.1.0.1.2
OPatch succeeded.
[root@rac12cnode1 OPatch]#

Repeat the above step to all the nodes in the cluster.

[root@rac12cnode2 OPatch]# ./opatch version
OPatch Version: 12.1.0.1.2
OPatch succeeded.
[root@rac12cnode2 OPatch]#
[root@rac12cnode3 OPatch]# ./opatch version
OPatch Version: 12.1.0.1.2
OPatch succeeded.
[root@rac12cnode3 OPatch]#
[root@rac12cnode4 OPatch]# ./opatch version
OPatch Version: 12.1.0.1.2
OPatch succeeded.
[root@rac12cnode4 OPatch]#

2. Validate and Record Pre-Patch information :
Validate using the following commands :
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/12.1.0/grid/
$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/app/oracle/product/12.1.0/dbhome_1

Login to each node in RAC as grid user and execute the following command.
$GRID_ORACLE_HOME/OPatch/opatch lsinventory
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Login to each node in RAC as oracle user and execute the following command.
$ORACLE_HOME/OPatch/opatch lsinventory
$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Connect to each instance and record registry information.
SQL> select comp_name,version,status from dba_registry;

[grid@rac12cnode1 bin]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.2
Copyright (c) 2013, Oracle Corporation. All rights reserved.

Oracle Home : /u01/app/12.1.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/12.1.0/grid/oraInst.loc
OPatch version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Log file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2014-03-13_02-41-32AM_1.log
Lsinventory Output file location : /u01/app/12.1.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2014-03-13_02-41-32AM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Grid Infrastructure 12c 12.1.0.1.0
There are 1 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Patch level status of Cluster nodes :
Patching Level Nodes
-------------- -----
0 rac12cnode4,rac12cnode1,rac12cnode2,rac12cnode3
--------------------------------------------------------------------------------
OPatch succeeded.
[grid@rac12cnode1 bin]$

3. Create OCM Response File If It Does Not Exist :
Create ocm response file using the following command and provide appropriate values for the prompts.

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp

Verify the created file using,

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp –verbose ocm.rsp

NOTE: The Opatch utility will prompt for your OCM (Oracle Configuration Manager) response file when it is run. Without which we cant proceed further.

[root@rac12cnode1 ~]# su - grid
[grid@rac12cnode1 ~]$ cd $ORACLE_HOME/OPatch/ocm/bin
[grid@rac12cnode1 bin]$ ls
emocmrsp
[grid@rac12cnode1 bin]$ ./emocmrsp
OCM Installation Response Generator 10.3.7.0.0 - Production
Copyright (c) 2005, 2012, Oracle and/or its affiliates. All rights reserved.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
The OCM configuration response file (ocm.rsp) was successfully created.
[grid@rac12cnode1 bin]$ ls
emocmrsp ocm.rsp
[grid@rac12cnode1 bin]$ ls -lrt
total 16
-rwxr-----. 1 grid oinstall 9063 Nov 27 2009 emocmrsp
-rw-r--r--. 1 grid oinstall 621 Mar 13 02:38 ocm.rsp
[grid@rac12cnode1 bin]$ chmod 777 ocm.rsp
[grid@rac12cnode1 bin]$

Copy this response file ocm.rsp to all the nodes in the cluster to the same location. Or you can create a new response file on each node using the same method above.

4. Download and Unzip the JAN2014 PSU patch : (as grid user)

Patch 17735306 is the JAN2014 PSU patch. It is downloaded and unzipped to the locaiton "/mnt/software/RAC/1201_PSU/JAN2014_PSU/"

5. One-off patch Conflicts detection and Resolution :

Determine whether any currently installed one-off patches conflict with the PSU patch.

$GRID_HOME/OPatch/opatchauto apply <UNZIPPED_PATCH_LOCATION>/17735306 analyze

But i don't have any patches applied to my home yet. So i can ignore this step. But if you have any conflicts identified in GI home or in DB home, follow MOS ID 1061295.1 to resolve the conflicts.

6. Patch Application :

While applying the patch to a flex cluster, please note the following points .

a. First node must be a HUB node.The last node can be either hub/leaf node.
b. Make sure GI Stack is up on the first hub node and at least one other hub node.
c. Make sure stack is up on all the leaf nodes
d. The opatchauto command will restart the stack on the local node and restart the Databases on the local node.

Now patch can be applied using the following syntax ..

# opatchauto apply <PATH_TO_PATCH_DIRECTORY> -ocmrf <ocm response file>

On all HUB nodes ( rac12cnode1/2/3)

[root@rac12cnode1 ~]# export PATH=/u01/app/12.1.0/grid/OPatch:$PATH[root@rac12cnode1 ~]# which opatchauto/u01/app/12.1.0/grid/OPatch/opatchauto
[root@rac12cnode1 ~]# opatchauto apply /mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306 -ocmrf /u01/app/12.1.0/grid/OPatch/ocm/bin/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /u01/app/12.1.0/grid
opatchauto log file: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/17735306/opatch_gi_2014-03-13_02-55-54_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid
RAC home(s):
/u01/app/oracle/product/12.1.0/dbhome_1
Configuration Validation: Successful
Patch Location: /mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306
Grid Infrastructure Patch(es): 17077442 17303297 17552800
RAC Patch(es): 17077442 17552800
Patch Validation: Successful
Stopping RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
Following database(s) were stopped and will be restarted later during the session: orcl
Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...
Patch "/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442" successfully applied to "/u01/app/oracle/product/12.1.0/dbhome_1".
Patch "/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17552800" successfully applied to "/u01/app/oracle/product/12.1.0/dbhome_1".
Stopping CRS ... Successful
Applying patch(es) to "/u01/app/12.1.0/grid" ...
Patch "/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442" successfully applied to "/u01/app/12.1.0/grid".
Patch "/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17303297" successfully applied to "/u01/app/12.1.0/grid".
Patch "/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17552800" successfully applied to "/u01/app/12.1.0/grid".
Starting CRS ... Successful
Starting RAC (/u01/app/oracle/product/12.1.0/dbhome_1) ... Successful
[WARNING] SQL changes, if any, could not be applied on the following database(s): ORCL ... Please refer to the log file for more details.
Apply Summary:
Following patch(es) are successfully installed:
GI Home: /u01/app/12.1.0/grid: 17077442, 17303297, 17552800
RAC Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17077442, 17552800
opatchauto succeeded.
[root@rac12cnode1 ~]#

While applying the patch, the services running on node 1 are relocated to other nodes in the cluster. While applying the patch to node 1 ( rac12cnode1) , i have the following status for the clusterware services..

[grid@rac12cnode4 ~]$ /u01/app/12.1.0/grid/bin/srvctl status database -d orcl
Instance ORCL1 is not running on node rac12cnode1
Instance ORCL2 is running on node rac12cnode2
Instance ORCL3 is running on node rac12cnode3
[grid@rac12cnode4 ~]$ /u01/app/12.1.0/grid/bin/crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.DATA.dg
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.LISTENER_LEAF.lsnr
OFFLINE OFFLINE rac12cnode4 STABLE
ora.net1.network
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.ons
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.proxy_advm
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac12cnode2 STABLE
ora.asm
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac12cnode2 STABLE
3 ONLINE ONLINE rac12cnode3 STABLE
ora.cvu
1 ONLINE ONLINE rac12cnode2 STABLE
ora.gns
1 ONLINE ONLINE rac12cnode3 STABLE
ora.gns.vip
1 ONLINE ONLINE rac12cnode3 STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.orcl.db
1 OFFLINE OFFLINE STABLE
2 ONLINE ONLINE rac12cnode2 Open,STABLE
3 ONLINE ONLINE rac12cnode3 Open,STABLE
ora.rac12cnode1.vip
1 ONLINE INTERMEDIATE rac12cnode3 FAILED OVER,STABLE
ora.rac12cnode2.vip
1 ONLINE ONLINE rac12cnode2 STABLE
ora.rac12cnode3.vip
1 ONLINE ONLINE rac12cnode3 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac12cnode2 STABLE
--------------------------------------------------------------------------------
[grid@rac12cnode4 ~]$

Repeat the above step to all the hub nodes in the flex cluster.

On Leaf Node :

Leaf node is not connected to storage directly and there is no oracle database instance running on this node. So when we run the patch application, it will apply the patch only to grid home. You can see that difference from the below session log from node 4( rac12cnode4) which is leaf node in my flex cluster.

[root@rac12cnode4 ~]# export PATH=/u01/app/12.1.0/grid/OPatch:$PATH[root@rac12cnode4 ~]# which opatchauto
/u01/app/12.1.0/grid/OPatch/opatchauto
[root@rac12cnode4 ~]# opatchauto apply /mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306 -ocmrf /u01/app/12.1.0/grid/OPatch/ocm/bin/ocm.rsp
OPatch Automation Tool
Copyright (c) 2013, Oracle Corporation. All rights reserved.
OPatchauto version : 12.1.0.1.2
OUI version : 12.1.0.1.0
Running from : /u01/app/12.1.0/grid
opatchauto log file: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/17735306/opatch_gi_2014-03-13_10-00-11_deploy.log
Parameter Validation: Successful
Grid Infrastructure home:
/u01/app/12.1.0/grid
Configuration Validation: Successful
Patch Location: /mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306
Grid Infrastructure Patch(es): 17077442 17303297 17552800
RAC Patch(es): 17077442 17552800
Patch Validation: Successful
Stopping CRS ... Successful
Applying patch(es) to "/u01/app/12.1.0/grid" ...
Patch "/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442" successfully applied to "/u01/app/12.1.0/grid".
Patch "/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17303297" successfully applied to "/u01/app/12.1.0/grid".
Patch "/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17552800" successfully applied to "/u01/app/12.1.0/grid".
Starting CRS ... Successful
Apply Summary:
Following patch(es) are successfully installed:
GI Home: /u01/app/12.1.0/grid: 17077442, 17303297, 17552800
opatchauto succeeded.
[root@rac12cnode4 ~]#

NOTE : please verify the difference in applying the patch to hub node and leaf node from the above output.

7. Verification of Patch application :

$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/app/12.1.0/grid/
$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/app/oracle/product/12.1.0/dbhome_1

Login to each node in RAC as grid user and execute the following command.
$GRID_ORACLE_HOME/OPatch/opatch lsinventory
$GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Login to each node in RAC as oracle user and execute the following command.
$ORACLE_HOME/OPatch/opatch lsinventory
$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'PSU'

Connect to each instance and verify registry information.
SQL> select comp_name,version,status from dba_registry;

8. Issues and Resolution :

Issue 1:

While applying the patch i got one error saying "OUI-67124:Copy failed from '/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442/files/bin/olsnodes.bin' to '/u01/app/12.1.0/grid/bin/olsnodes.bin'..."

Content From Logs:

ERROR:

UtilSession failed: ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442/files/bin/olsnodes.bin' to '/u01/app/12.1.0/grid/bin/olsnodes.bin'...

2014-03-13_05-10-11 :

Failed to run this command :

/u01/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGrid12c_patchList -local -invPtrLoc /u01/app/12.1.0/grid/oraInst.loc -oh /u01/app/12.1.0/grid -silent -ocmrf /u01/app/12.1.0/grid/OPatch/ocm/bin/ocm.rsp

oracle.opatch.gi.RunExecutionSteps.runGenericShellCommands(RunExecutionSteps.java:724)

oracle.opatch.gi.RunExecutionSteps.processAllSteps(RunExecutionSteps.java:183)

oracle.opatch.gi.GIPatching.processPatchingSteps(GIPatching.java:747)

oracle.opatch.gi.OPatchautoExecution.main(OPatchautoExecution.java:101)

Command "/u01/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGrid12c_patchList -local -invPtrLoc /u01/app/12.1.0/grid/oraInst.loc -oh /u01/app/12.1.0/grid -silent -ocmrf /u01/app/12.1.0/grid/OPatch/ocm/bin/ocm.rsp" execution failed:

Log file Location for the failed command: /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2014-03-13_04-41-37AM_1.log

[Mar 13, 2014 4:53:26 AM] The following actions have failed:

[Mar 13, 2014 4:53:26 AM] OUI-67124:Copy failed from '/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442/files/bin/olsnodes.bin' to '/u01/app/12.1.0/grid/bin/olsnodes.bin'...

[Mar 13, 2014 4:53:26 AM] Do you want to proceed? [y|n]

[Mar 13, 2014 4:53:29 AM] N (auto-answered by -silent)

[Mar 13, 2014 4:53:29 AM] User Responded with: N

[Mar 13, 2014 4:53:29 AM] OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442/files/bin/olsnodes.bin' to '/u01/app/12.1.0/grid/bin/olsnodes.bin'...

[Mar 13, 2014 4:53:29 AM] Restoring "/u01/app/12.1.0/grid" to the state prior to running NApply...

[Mar 13, 2014 5:10:09 AM] Checking if OPatch needs to invoke 'make' to restore some binaries...

[Mar 13, 2014 5:10:09 AM] OPatch was able to restore your system. Look at log file and timestamp of each file to make sure your system is in the state prior to applying the patch.

[Mar 13, 2014 5:10:09 AM] OUI-67124:

NApply restored the home. Please check your ORACLE_HOME to make sure:

- files are restored properly.

- binaries are re-linked correctly.

(use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under

"/u01/app/12.1.0/grid/.patch_storage/NApply/2014-03-13_04-41-37AM"

[Mar 13, 2014 5:10:10 AM] OUI-67073:UtilSession failed: ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442/files/bin/olsnodes.bin' to '/u01/app/12.1.0/grid/bin/olsnodes.bin'...

[Mar 13, 2014 5:10:10 AM] --------------------------------------------------------------------------------

[Mar 13, 2014 5:10:10 AM] The following warnings have occurred during OPatch execution:

[Mar 13, 2014 5:10:10 AM] 1) OUI-67124:Copy failed from '/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442/files/bin/olsnodes.bin' to '/u01/app/12.1.0/grid/bin/olsnodes.bin'...

[Mar 13, 2014 5:10:10 AM] 2) OUI-67124:ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442/files/bin/olsnodes.bin' to '/u01/app/12.1.0/grid/bin/olsnodes.bin'...

[Mar 13, 2014 5:10:10 AM] 3) OUI-67124:

NApply restored the home. Please check your ORACLE_HOME to make sure:

- files are restored properly.

- binaries are re-linked correctly.

(use restore.[sh,bat] and make.txt (Unix only) as a reference. They are located under

"/u01/app/12.1.0/grid/.patch_storage/NApply/2014-03-13_04-41-37AM"

[Mar 13, 2014 5:10:10 AM] --------------------------------------------------------------------------------

[Mar 13, 2014 5:10:10 AM] Finishing UtilSession at Thu Mar 13 05:10:10 EDT 2014

[Mar 13, 2014 5:10:10 AM] Log file location: /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2014-03-13_04-41-37AM_1.log

[Mar 13, 2014 5:10:10 AM] Stack Description: java.lang.RuntimeException: ApplySession failed in system modification phase... 'ApplySession::apply failed: Copy failed from '/mnt/software/RAC/1201_PSU/JAN2014_PSU/17735306/17077442/files/bin/olsnodes.bin' to '/u01/app/12.1.0/grid/bin/olsnodes.bin'...

Resolution :

Verified permission and there is no issue accessing and copying file manually to target location. Then re-executed without any modificaiton. Then it went fine. May be it is my environment specific as i have PSU patch located on NFS file system. But if you have also got the same issue, it might be something common to many environment and intemittent issue.

Logfile Location For PSU Patch :
/u01/app/12.1.0/grid/cfgtoollogs/opatch/
/u01/app/12.1.0/grid/cfgtoollogs/opatchauto/

Hope This Helps
SRI

11:53 AM

RAC 12c Flex Cluster Installation

11:57 PM

RAC : Clone Flex Cluster To Extend Cluster By Adding Hub and Lead Nodes

Written By askMLabs on Saturday, March 15, 2014 | 11:57 PM

In this article , we will see how to clone RAC environment to extend the existing cluster. Clone can also be used to prepare a new cluster environment. But in this article, we are going to see extending the existing using clone method. A cluster can also be extended using addnode method. But cloning is a different method to extend the cluster.

Environment Details :

RAC Version Cluster Type	12c 12.1.0.1.0 Flex Cluster
Hub Nodes	rac12cnode1/2
Leaf Nodes	No
DB Running on	All Hub Nodes
Task	Clone cluster to extend from 2 node to 4 node

[grid@rac12cnode1 ~]$ olsnodes -s -t
rac12cnode1 Active Unpinned
rac12cnode2 Active Unpinned
[grid@rac12cnode1 ~]$ crsctl get cluster mode status
Cluster is running in "flex" mode
[grid@rac12cnode1 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled
[grid@rac12cnode1 ~]$ crsctl get node role config -all
Node 'rac12cnode1' configured role is 'hub'
Node 'rac12cnode2' configured role is 'hub'
[grid@rac12cnode1 ~]$

Step By Step :

1. Prepare the new cluster nodes
2. Prepare the existing cluster nodes
3. Deploy the Grid Infrastructure to new cluster nodes
4. Run clone.pl on all the cluster nodes.
5. Run orainstRoot.sh script on all target cluster nodes.
6. Execute addnode.sh in silent mode to add hub node and leaf node
7. Copy config files from source system to all target cluster nodes
8. Run root.sh on all the target nodes to configure cluster.
9. Verify the cloned cluster with cluvfy

1. Prepare the new cluster nodes:

Our task is to add two nodes to the existing flex cluster in clone method. We need to first prepare the nodes which are to be added to the cluster.

I have the following points from documentation, but you can also follow my other articles/videos on rac to prepare nodes for cluster addition.

On each destination node, perform the following preinstallation steps:

Specify the kernel parameters
Configure block devices for Oracle Clusterware devices
Ensure that you have set the block device permissions correctly
Use short, nondomain-qualified names for all of the names in the /etc/hosts file
Test whether the interconnect interfaces are reachable using the ping command
Verify that the VIP addresses are not active at the start of the cloning process by using the ping command (the ping command of the VIP address must fail)
On AIX systems, and on Solaris x86-64-bit systems running vendor clusterware, if you add a node to the cluster, then you must run the rootpre.sh script (located at the mount point it you install Oracle Clusterware from a DVD or in the directory where you unzip the tar file if you download the software) on the node before you add the node to the cluster
Run CVU to verify your hardware and operating system environment

Complete all above steps , so that the nodes are ready for adding / clone to cluster node.

Verify if the nodes are ready to add to the cluster using the following commands

[grid@rac12cnode1 ~]$ nslookup rac12cnode3
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode3.localdomain
Address: 192.168.1.133
[grid@rac12cnode1 ~]$ nslookup rac12cnode4
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode4.localdomain
Address: 192.168.1.134
[grid@rac12cnode1 ~]$ clear
[grid@rac12cnode1 ~]$ cluvfy stage -pre nodeadd -n rac12cnode3,rac12cnode4 -fixup -verbose
Performing pre-checks for node addition......

2. Prepare the existing cluster nodes:

In this step, we need to create a copy of the existing oracle grid infrastructure home and remove the unnecessary files from the copied home. You can perform this step while the clusterware is up and running.

Create an exclusion list to be excluded while creating the tar backup.

[root@rac12cnode1 askm]# cat excl_list.txt
/u01/app/12.1.0/grid/rac12cnode1
/u01/app/12.1.0/grid/log/host_name
/u01/app/12.1.0/grid/gpnp/host_name
/u01/app/12.1.0/grid/crs/init/*
/u01/app/12.1.0/grid/cdata/*
/u01/app/12.1.0/grid/crf/*
/u01/app/12.1.0/grid/network/admin/*.ora
/u01/app/12.1.0/grid/root.sh*
*.ouibak
*.ouibak1
[root@rac12cnode1 askm]#

Create compressed copy of the oracle grid infrastructure home using tar utility. Execute the following command on any existing node in the cluster :

[root@rac12cnode1 12.1.0]# tar -czf gridHome.tar.gz -X /tmp/askm/excl_list.txt /u01/app/12.1.0/grid
tar: Removing leading `/' from member names
tar: /u01/app/12.1.0/grid/log/rac12cnode1/gipcd/gipcd.log: file changed as we read it
tar: /u01/app/12.1.0/grid/log/rac12cnode1/agent/crsd/oraagent_grid/oraagent_grid.log: file changed as we read it
tar: /u01/app/12.1.0/grid/log/rac12cnode1/ctssd/octssd.log: file changed as we read it
tar: /u01/app/12.1.0/grid/log/rac12cnode1/cssd/ocssd.log: file changed as we read it
tar: /u01/app/12.1.0/grid/rdbms/audit: file changed as we read it
[root@rac12cnode1 12.1.0]# ls -lrt
total 3100700
drwxr-xr-x. 74 root oinstall 4096 Mar 13 03:41 grid
-rw-r--r--. 1 root root 3172006616 Mar 14 04:53 gridHome.tar.gz
[root@rac12cnode1 12.1.0]#

3. Deploy the Grid Infrastructure to new cluster nodes:

Now copy the compressed backup of the oracle grid infrastructure home created in step 2 above to all the target nodes ie rac12cnode3/4.

[root@rac12cnode1 12.1.0]# ls -lrt
total 3100700
drwxr-xr-x. 74 root oinstall 4096 Mar 13 03:41 grid
-rw-r--r--. 1 root root 3172006616 Mar 14 04:53 gridHome.tar.gz
[root@rac12cnode1 12.1.0]# scp gridHome.tar.gz root@rac12cnode3:/u01/app/12.1.0
root@rac12cnode3's password:
gridHome.tar.gz 100% 3025MB 9.7MB/s 05:12
[root@rac12cnode1 12.1.0]# ls -lrt gridHome.tar.gz
-rw-r--r--. 1 root root 3172006616 Mar 14 04:53 gridHome.tar.gz
[root@rac12cnode1 12.1.0]# du -sh gridHome.tar.gz
3.0G gridHome.tar.gz
[root@rac12cnode1 12.1.0]# scp gridHome.tar.gz root@rac12cnode4:/u01/app/12.1.0
root@rac12cnode4's password:
gridHome.tar.gz 100% 3025MB 9.3MB/s 05:25
[root@rac12cnode1 12.1.0]#

Extract the compressed backup to node 3 and node 4.

[root@rac12cnode4 ~]# cd /u01/app/12.1.0/
[root@rac12cnode4 12.1.0]# ls
[root@rac12cnode4 12.1.0]# ls -lrt
total 3100696
-rw-r--r--. 1 root root 3172006616 Mar 14 06:33 gridHome.tar.gz
[root@rac12cnode4 12.1.0]# du -sh *
3.0G gridHome.tar.gz
[root@rac12cnode4 12.1.0]# tar xvzf gridHome.tar.gz -C /
..
..
[root@rac12cnode4 12.1.0]#

[root@rac12cnode3 ~]# cd /u01/app/12.1.0/
[root@rac12cnode3 12.1.0]# ls -lrt
total 3100696
-rw-r--r--. 1 root root 3172006616 Mar 14 05:16 gridHome.tar.gz
[root@rac12cnode3 12.1.0]# tar xvzf gridHome.tar.gz -C /
..
..
[root@rac12cnode3 12.1.0]#

NOTE : Review the copied homes on target nodes , If you still have any files need to be deleted from the copied home on nodes 3 and 4.

I have created following shell script to delete any unwanted files from the new homes created on node 3 and 4.

[root@rac12cnode3 log]# cat /tmp/askm/file_delete.sh
cd /u01/app/12.1.0/grid/
rm -rf /u01/app/12.1.0/grid/log/rac12cnode1
rm -rf /u01/app/12.1.0/grid/gpnp/rac12cnode1
find gpnp -type f -exec rm -f {} \;
rm -rf /u01/app/12.1.0/grid/cfgtoollogs/*
rm -rf /u01/app/12.1.0/grid/crs/init/*
rm -rf /u01/app/12.1.0/grid/cdata/*
rm -rf /u01/app/12.1.0/grid/crf/*
rm -rf /u01/app/12.1.0/grid/network/admin/*.ora
rm -rf /u01/app/12.1.0/grid/crs/install/crsconfig_params
find . -name '*.ouibak' -exec rm {} \;
find . -name '*.ouibak.1' -exec rm {} \;
rm -rf /u01/app/12.1.0/grid/root.sh*
rm -rf /u01/app/12.1.0/grid/rdbms/audit/*
rm -rf /u01/app/12.1.0/grid/rdbms/log/*
rm -rf /u01/app/12.1.0/grid/inventory/backup/*
[root@rac12cnode3 log]#

Now execute the file /tmp/askm/file_delete.sh on nodes 3 and 4.

4. Run clone.pl on all the cluster nodes.

Run the clone.pl script located in the Grid_home/clone/bin directory on Node 3 and Node 4.

On Node 3:(HUB Node ):

[grid@rac12cnode3 ~]$ cd $ORACLE_HOME/clone/bin
[grid@rac12cnode3 bin]$ perl clone.pl ORACLE_HOME=/u01/app/12.1.0/grid ORACLE_HOME_NAME=OraGrid12c ORACLE_BASE=/u01/app/grid "'CLUSTER_NODES={rac12cnode1, rac12cnode2,rac12cnode3,rac12cnode4}'" "'LOCAL_NODE=rac12cnode3'" CRS=TRUE INVENTORY_LOCATION=/u01/app/oraInventory
./runInstaller -clone -waitForCompletion "ORACLE_HOME=/u01/app/12.1.0/grid" "ORACLE_HOME_NAME=OraGrid12c" "ORACLE_BASE=/u01/app/grid" "'CLUSTER_NODES={rac12cnode1, rac12cnode2,rac12cnode3,rac12cnode4}'" "'LOCAL_NODE=rac12cnode3'" "CRS=TRUE" "INVENTORY_LOCATION=/u01/app/oraInventory" -silent -paramFile /u01/app/12.1.0/grid/clone/clone_oraparam.ini
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 500 MB. Actual 7237 MB Passed
Checking swap space: must be greater than 500 MB. Actual 2042 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-03-14_08-10-41AM. Please wait ...You can find the log of this install session at:
/u01/app/oraInventory/logs/cloneActions2014-03-14_08-10-41AM.log
.................................................. 5% Done.
.................................................. 10% Done.
.................................................. 15% Done.
.................................................. 20% Done.
.................................................. 25% Done.
.................................................. 30% Done.
.................................................. 35% Done.
.................................................. 40% Done.
.................................................. 45% Done.
.................................................. 50% Done.
.................................................. 55% Done.
.................................................. 60% Done.
.................................................. 65% Done.
.................................................. 70% Done.
.................................................. 75% Done.
.................................................. 80% Done.
.................................................. 85% Done.
.................................................. 90% Done.
.................................................. 95% Done.
Copy files in progress.
Copy files successful.
Link binaries in progress.
Link binaries successful.
Setup files in progress.
Setup files successful.
Setup Inventory in progress.
Setup Inventory successful.
Finish Setup successful.
The cloning of OraGrid12c was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2014-03-14_08-10-41AM.log' for more details.
As a root user, execute the following script(s):
1. /u01/app/12.1.0/grid/root.sh

.................................................. 100% Done.
[grid@rac12cnode3 bin]$

On Node 4: (Leaf Node)

[grid@rac12cnode4 ~]$ cd /u01/app/12.1.0/grid/clone/bin
[grid@rac12cnode4 bin]$ pwd
/u01/app/12.1.0/grid/clone/bin
[grid@rac12cnode4 bin]$ perl clone.pl ORACLE_HOME=/u01/app/12.1.0/grid ORACLE_HOME_NAME=OraGrid12c ORACLE_BASE=/u01/app/grid "'CLUSTER_NODES={rac12cnode1, rac12cnode2,rac12cnode3,rac12cnode4}'" "'LOCAL_NODE=rac12cnode4'" CRS=TRUE INVENTORY_LOCATION=/u01/app/oraInventory
./runInstaller -clone -waitForCompletion "ORACLE_HOME=/u01/app/12.1.0/grid" "ORACLE_HOME_NAME=OraGrid12c" "ORACLE_BASE=/u01/app/grid" "'CLUSTER_NODES={rac12cnode1, rac12cnode2,rac12cnode3,rac12cnode4}'" "'LOCAL_NODE=rac12cnode4'" "CRS=TRUE" "INVENTORY_LOCATION=/u01/app/oraInventory" -silent -paramFile /u01/app/12.1.0/grid/clone/clone_oraparam.ini
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 500 MB. Actual 7239 MB Passed
Checking swap space: must be greater than 500 MB. Actual 2041 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-03-14_08-15-13AM. Please wait ...You can find the log of this install session at:
/u01/app/oraInventory/logs/cloneActions2014-03-14_08-15-13AM.log
.................................................. 5% Done.
.................................................. 10% Done.
.................................................. 15% Done.
.................................................. 20% Done.
.................................................. 25% Done.
.................................................. 30% Done.
.................................................. 35% Done.
.................................................. 40% Done.
.................................................. 45% Done.
.................................................. 50% Done.
.................................................. 55% Done.
.................................................. 60% Done.
.................................................. 65% Done.
.................................................. 70% Done.
.................................................. 75% Done.
.................................................. 80% Done.
.................................................. 85% Done.
.................................................. 90% Done.
.................................................. 95% Done.
Copy files in progress.
Copy files successful.
Link binaries in progress.
Link binaries successful.
Setup files in progress.
Setup files successful.
Setup Inventory in progress.
Setup Inventory successful.
Finish Setup successful.
The cloning of OraGrid12c was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2014-03-14_08-15-13AM.log' for more details.
As a root user, execute the following script(s):
1. /u01/app/12.1.0/grid/root.sh

.................................................. 100% Done.
[grid@rac12cnode4 bin]$

IMP IMP : Please don't run root.sh at this point of time.

5. Run orainstRoot.sh script on all target cluster nodes:(as root user)

This script populates the /etc/oraInst.loc directory with the location of the central inventory.

[root@rac12cnode3 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac12cnode3 ~]#

[root@rac12cnode4 oraInventory]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac12cnode4 oraInventory]#

6. Execute addnode.sh in silent mode to add hub node and leaf node:

Run the addnode.sh script from $GRID_HOME/addnode/

[grid@rac12cnode1 addnode]$ ./addnode.sh -silent -noCopy ORACLE_HOME=/u01/app/12.1.0/grid "CLUSTER_NEW_NODES={rac12cnode3,rac12cnode4}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac12cnode3-vip}" "CLUSTER_NEW_NODE_ROLES={HUB,LEAF}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 7444 MB Passed
Checking swap space: must be greater than 150 MB. Actual 407 MB Passed
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2014-03-15_11-16-57AM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2014-03-15_11-16-57AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 40% Done.
As a root user, execute the following script(s):
1. /u01/app/12.1.0/grid/root.sh
Execute /u01/app/12.1.0/grid/root.sh on the following nodes:
[rac12cnode3, rac12cnode4]
The scripts can be executed in parallel on all the nodes. If there are any policy managed databases managed by cluster, proceed with the addnode procedure without executing the root.sh script. Ensure that root.sh script is executed after all the policy managed databases managed by clusterware are extended to the new nodes.
.................................................. 60% Done.
Update Inventory in progress.
.................................................. 100% Done.
Update Inventory successful.
Successfully Setup Software.
[grid@rac12cnode1 addnode]$

In the preceding syntax example, Node 4 is designated as a Leaf Node and does not require that a VIP be included.

7. Copy config files from source system to all target cluster nodes:

Copy the following files from Node 1, on which you ran addnode.sh, to Node3 and Node4.

Grid_home/crs/install/crsconfig_addparams
Grid_home/crs/install/crsconfig_params
Grid_home/gpnp

[root@rac12cnode1 grid]# scp /u01/app/12.1.0/grid/crs/install/crsconfig_addparams /u01/app/12.1.0/grid/crs/install/crsconfig_params /u01/app/12.1.0/grid/gpnp.tar.gz 192.168.1.133:/tmp/askm
root@192.168.1.133's password:
crsconfig_addparams 100% 1089 1.1KB/s 00:00
crsconfig_params 100% 5509 5.4KB/s 00:00
gpnp.tar.gz 100% 90KB 90.1KB/s 00:00
[root@rac12cnode1 grid]# scp /u01/app/12.1.0/grid/crs/install/crsconfig_addparams /u01/app/12.1.0/grid/crs/install/crsconfig_params /u01/app/12.1.0/grid/gpnp.tar.gz 192.168.1.134:/tmp/askm
root@192.168.1.134's password:
crsconfig_addparams 100% 1089 1.1KB/s 00:00
crsconfig_params 100% 5509 5.4KB/s 00:00
gpnp.tar.gz 100% 90KB 90.1KB/s 00:00
[root@rac12cnode1 grid]#

[root@rac12cnode4 ~]# mv /u01/app/12.1.0/grid/crs/install/crsconfig_addparams /u01/app/12.1.0/grid/crs/install/crsconfig_addparams_bak
[root@rac12cnode4 ~]# mv /u01/app/12.1.0/grid/crs/install/crsconfig_params /u01/app/12.1.0/grid/crs/install/crsconfig_params_bak
[root@rac12cnode4 ~]# mv /tmp/askm/crsconfig_addparams /u01/app/12.1.0/grid/crs/install/crsconfig_addparams
[root@rac12cnode4 ~]# mv /tmp/askm/crsconfig_params /u01/app/12.1.0/grid/crs/install/crsconfig_params
[root@rac12cnode4 ~]# chown grid:oinstall /u01/app/12.1.0/grid/crs/install/crsconfig_addparams
[root@rac12cnode4 ~]# chown grid:oinstall /u01/app/12.1.0/grid/crs/install/crsconfig_params
[root@rac12cnode4 ~]# cd /u01/app/12.1.0/grid
[root@rac12cnode4 grid]# mv gpnp gpnp_bak
[root@rac12cnode4 grid]# pwd
/u01/app/12.1.0/grid
[root@rac12cnode4 grid]# tar xzf /tmp/askm/gpnp.tar.gz
[root@rac12cnode4 grid]# ls -ld gpnp
drwxr-x---. 8 grid oinstall 4096 Mar 7 09:06 gpnp
[root@rac12cnode4 grid]#

Complete the this step on node 3 and node 4.

8. Run root.sh on all the target nodes to configure cluster:

On Node 3 and Node 4, run the Grid_home/root.sh script.

[root@rac12cnode3 grid]# ./root.sh
Check /u01/app/12.1.0/grid/install/root_rac12cnode3_2014-03-15_11-55-20.log for the output of root script
[root@rac12cnode3 grid]# cat /u01/app/12.1.0/grid/install/root_rac12cnode3_2014-03-15_11-55-20.log
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/roothas.pl

To configure Grid Infrastructure for a Cluster execute the following command as grid user:
/u01/app/12.1.0/grid/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.
[root@rac12cnode3 grid]#

If you get this output after executing the root.sh , then your cluster is not configured correctly. You need to modify the script "/u01/app/12.1.0/grid/crs/config/rootconfig.sh" to uncomment the some part of the script. Please refer to my video if you need more details on this part.

[root@rac12cnode4 ~]# diff /u01/app/12.1.0/grid/crs/config/rootconfig.sh /u01/app/12.1.0/grid/crs/config/rootconfig.sh_bak_askm
33,37c33,37
< if [ "$ADDNODE" = "true" ]; then
< SW_ONLY=false
< HA_CONFIG=false
< fi
---
> #if [ "$ADDNODE" = "true" ]; then
> # SW_ONLY=false
> # HA_CONFIG=false
> #fi
[root@rac12cnode4 ~]#

Now run the root.sh again. This time it will execute successfully and configures the cluster.

[root@rac12cnode3 grid]# ./root.sh
Check /u01/app/12.1.0/grid/install/root_rac12cnode3_2014-03-15_11-57-18.log for the output of root script
[root@rac12cnode3 grid]#

[root@rac12cnode4 grid]# ./root.sh
Check /u01/app/12.1.0/grid/install/root_rac12cnode4_2014-03-15_12-24-46.log for the output of root script
[root@rac12cnode4 grid]#

The output for above executions for confirmation that they configured clusterware correctly ....

[root@rac12cnode3 ~]# tail -f /u01/app/12.1.0/grid/install/root_rac12cnode3_2014-03-15_11-57-18.log
ORACLE_HOME= /u01/app/12.1.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
OLR initialization - successful
2014/03/15 11:59:52 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac12cnode3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac12cnode3'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac12cnode3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac12cnode3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac12cnode3'
CRS-2672: Attempting to start 'ora.evmd' on 'rac12cnode3'
CRS-2676: Start of 'ora.mdnsd' on 'rac12cnode3' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac12cnode3'
CRS-2676: Start of 'ora.gpnpd' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac12cnode3'
CRS-2676: Start of 'ora.gipcd' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac12cnode3'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac12cnode3'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac12cnode3'
CRS-2676: Start of 'ora.diskmon' on 'rac12cnode3' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rac12cnode3'
CRS-2676: Start of 'ora.cssd' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac12cnode3'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac12cnode3'
CRS-2676: Start of 'ora.ctssd' on 'rac12cnode3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac12cnode3'
CRS-2676: Start of 'ora.asm' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac12cnode3'
CRS-2676: Start of 'ora.storage' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac12cnode3'
CRS-2676: Start of 'ora.crsd' on 'rac12cnode3' succeeded
CRS-6017: Processing resource auto-start for servers: rac12cnode3
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac12cnode3'
CRS-2672: Attempting to start 'ora.ons' on 'rac12cnode3'
CRS-2676: Start of 'ora.ons' on 'rac12cnode3' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac12cnode3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac12cnode3'
CRS-2676: Start of 'ora.asm' on 'rac12cnode3' succeeded
CRS-2664: Resource 'ora.DATA.dg' is already running on 'rac12cnode1'
CRS-2664: Resource 'ora.proxy_advm' is already running on 'rac12cnode1'
CRS-2664: Resource 'ora.proxy_advm' is already running on 'rac12cnode2'
CRS-2664: Resource 'ora.DATA.dg' is already running on 'rac12cnode2'
CRS-2664: Resource 'ora.proxy_advm' is already running on 'rac12cnode1'
CRS-2664: Resource 'ora.proxy_advm' is already running on 'rac12cnode2'
CRS-6016: Resource auto-start has completed for server rac12cnode3
CRS-2672: Attempting to start 'ora.proxy_advm' on 'rac12cnode3'
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/03/15 12:10:16 CLSRSC-343: Successfully started Oracle clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2014/03/15 12:11:15 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac12cnode3 ~]#

[root@rac12cnode4 ~]# tail -f /u01/app/12.1.0/grid/install/root_rac12cnode4_2014-03-15_12-24-46.log
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
OLR initialization - successful
2014/03/15 12:27:14 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac12cnode4'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac12cnode4'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac12cnode4' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac12cnode4' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac12cnode4'
CRS-2672: Attempting to start 'ora.evmd' on 'rac12cnode4'
CRS-2676: Start of 'ora.evmd' on 'rac12cnode4' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac12cnode4'
CRS-2676: Start of 'ora.gpnpd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac12cnode4'
CRS-2676: Start of 'ora.gipcd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac12cnode4'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac12cnode4'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac12cnode4'
CRS-2676: Start of 'ora.diskmon' on 'rac12cnode4' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rac12cnode4'
CRS-2676: Start of 'ora.cssd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac12cnode4'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac12cnode4'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac12cnode4' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac12cnode4'
CRS-2676: Start of 'ora.storage' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac12cnode4'
CRS-2676: Start of 'ora.crsd' on 'rac12cnode4' succeeded
CRS-6017: Processing resource auto-start for servers: rac12cnode4
CRS-6016: Resource auto-start has completed for server rac12cnode4
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/03/15 12:31:56 CLSRSC-343: Successfully started Oracle clusterware stack
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2014/03/15 12:32:05 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac12cnode4 ~]#

9. Verify the cloned cluster with cluvfy:

[root@rac12cnode1 grid]# su - grid
[grid@rac12cnode1 ~]$ olsnodes -s -t
rac12cnode1 Active Unpinned
rac12cnode2 Active Unpinned
rac12cnode3 Active Unpinned
rac12cnode4 Active Unpinned
[grid@rac12cnode1 ~]$ crsctl get cluster mode status
Cluster is running in "flex" mode
[grid@rac12cnode1 ~]$ srvctl config gns
GNS is enabled.
[grid@rac12cnode1 ~]$ crsctl get node role config
Node 'rac12cnode1' configured role is 'hub'
[grid@rac12cnode1 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled
[grid@rac12cnode1 ~]$ srvctl status asm -detail
ASM is running on rac12cnode1,rac12cnode2,rac12cnode3
ASM is enabled.
[grid@rac12cnode1 ~]$ crsctl get node role config -all
Node 'rac12cnode1' configured role is 'hub'
Node 'rac12cnode2' configured role is 'hub'
Node 'rac12cnode3' configured role is 'hub'
Node 'rac12cnode4' configured role is 'leaf'
[grid@rac12cnode1 ~]$ clear
[grid@rac12cnode1 ~]$ cluvfy stage -post nodeadd -n rac12cnode3,rac12cnode4 -verbose
...
....
Post-check for node addition was successful.
[grid@rac12cnode1 ~]$

Hope this helps

SRI

10:31 AM

RAC : Flex Cluster Useful Commands

Written By askMLabs on Wednesday, March 12, 2014 | 10:31 AM

[grid@rac12cnode1 ~]$ which crsctl
/u01/app/12.1.0/grid/bin/crsctl
[grid@rac12cnode1 ~]$ crsctl get cluster mode status
Cluster is running in "flex" mode <== Note that the cluster is in flex mode
[grid@rac12cnode1 ~]$ olsnodes
rac12cnode1
rac12cnode2
rac12cnode3
rac12cnode4
[grid@rac12cnode1 ~]$ srvctl config gns
GNS is enabled. <= GNS is enabled in flex cluster
[grid@rac12cnode1 ~]$ srvctl config gns -detail
GNS is enabled.
GNS is listening for DNS server requests on port 53
GNS is using port 5,353 to connect to mDNS
GNS status: OK
GNS version: 12.1.0.1.0
Globally unique identifier of the cluster where GNS is running: 68a8c394a9e27f58ff447892d2693872
Name of the cluster where GNS is running: crs64bit12c
Cluster type: server.
GNS log level: 1.
GNS listening addresses: tcp://192.168.1.140:24363.
[grid@rac12cnode1 ~]$ oifcfg getif
eth0 192.168.1.0 global public
eth1 10.10.10.0 global cluster_interconnect
eth2 10.11.0.0 global asm <== Separate private network for storage in flex cluster
[grid@rac12cnode1 ~]$ crsctl get node role config
Node 'rac12cnode1' configured role is 'hub' <== This node is a hub node
[grid@rac12cnode1 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled
[grid@rac12cnode1 ~]$ asmcmd showclusterstate
Normal
[grid@rac12cnode1 ~]$ srvctl status asm -detail
ASM is running on rac12cnode1,rac12cnode2,rac12cnode3 <== Note that ASM is only running on hub nodes
ASM is enabled.
[grid@rac12cnode1 ~]$ olsnodes
rac12cnode1
rac12cnode2
rac12cnode3
rac12cnode4
[grid@rac12cnode1 ~]$ olsnodes -s
rac12cnode1 Active
rac12cnode2 Active
rac12cnode3 Active
rac12cnode4 Active <== Note that the leaf node is also active participant in cluster
[grid@rac12cnode1 ~]$ srvctl config asm
ASM home: /u01/app/12.1.0/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: 3 <== ASM instance count is equal to the hub nodes
Cluster ASM listener: ASMNET1LSNR_ASM <== Note this new listener in 12c
[grid@rac12cnode1 ~]$ crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr <== New in 12c
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.DATA.dg
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.LISTENER_LEAF.lsnr <== New in 12c
OFFLINE OFFLINE rac12cnode4 STABLE
ora.net1.network
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.ons
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.proxy_advm <== New in 12c
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac12cnode2 STABLE
ora.asm
1 ONLINE ONLINE rac12cnode1 STABLE
2 ONLINE ONLINE rac12cnode2 STABLE
3 ONLINE ONLINE rac12cnode3 STABLE
ora.cvu
1 ONLINE ONLINE rac12cnode2 STABLE
ora.gns
1 ONLINE ONLINE rac12cnode2 STABLE
ora.gns.vip
1 ONLINE ONLINE rac12cnode2 STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.orcl.db <== Note that DB is running only on hub nodes, not on leaf nodes
1 ONLINE ONLINE rac12cnode1 Open,STABLE
2 ONLINE ONLINE rac12cnode2 Open,STABLE
3 ONLINE ONLINE rac12cnode3 Open,STABLE
ora.rac12cnode1.vip
1 ONLINE ONLINE rac12cnode1 STABLE
ora.rac12cnode2.vip
1 ONLINE ONLINE rac12cnode2 STABLE
ora.rac12cnode3.vip
1 ONLINE ONLINE rac12cnode3 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac12cnode2 STABLE
--------------------------------------------------------------------------------
[grid@rac12cnode1 ~]$
[grid@rac12cnode1 ~]$ crsctl get cluster mode config
Cluster is configured as "flex"
[grid@rac12cnode1 ~]$ crsctl get cluster mode status
Cluster is running in "flex" mode
[grid@rac12cnode1 ~]$
[grid@rac12cnode1 ~]$ crsctl get node role config -all
Node 'rac12cnode1' configured role is 'hub'
Node 'rac12cnode2' configured role is 'hub'
Node 'rac12cnode3' configured role is 'hub'
Node 'rac12cnode4' configured role is 'leaf'
[grid@rac12cnode1 ~]$ crsctl get node role status -all
Node 'rac12cnode1' active role is 'hub'
Node 'rac12cnode2' active role is 'hub'
Node 'rac12cnode3' active role is 'hub'
Node 'rac12cnode4' active role is 'leaf'
[grid@rac12cnode1 ~]$ crsctl get node role config -node rac12cnode3
Node 'rac12cnode3' configured role is 'hub'
[grid@rac12cnode1 ~]$ crsctl get node role status -node rac12cnode3
Node 'rac12cnode3' active role is 'hub'
[grid@rac12cnode1 ~]$ crsctl get node role status -node rac12cnode4
Node 'rac12cnode4' active role is 'leaf'
[grid@rac12cnode1 ~]$ crsctl get node role config -node rac12cnode4
Node 'rac12cnode4' configured role is 'leaf'
[grid@rac12cnode1 ~]$

The above commands are just some of the new command options introduced in 12c.

Hope This Helps
SRI

8:20 AM

RAC : Leaf Node Addition To Flex Cluster - 12101

Written By askMLabs on Tuesday, March 11, 2014 | 8:20 AM

In this article, we will see the step by step procedure to add a leaf node to flex cluster. Flex cluster is introduced in database version 12.1.0.1. Leaf node in a flex cluster is not attached directly to the storage and there will be no database running on this node. This is part of the cluster running all high availability cluster services. Initially during the beta version of the 12c database, oracle told that even the leaf node can host the database instance, but the release 12.1.0.1 does not still support to have the database running on leaf node.

So as of this time, when i am writing this article, ( 12.1.0.1 ) version, the leaf node is not supported to run the database instances, but it can be used to give high availability to any other application servicies like application server, e-business suite etc...

Having said the above theory about leaf node, now the deletion and addition procedure of leaf node to RAC flex cluster is different from the actual procedure that we do till 11gR2 versions.

Before proceeding with the procedure, let me explain a little about the environment where i am going to perform the leaf node addition. I have a 3 node cluster environment with 12c flex cluster enabled. I have 3 hub nodes and 0 leaf node. I am going to add leaf node to the 3 node flex cluster. rac12cnode1/2/3 are my hub nodes and rac12cnode4 is leaf node to be added to the cluster.

We need to prepare the node to add it to the cluster. I am mentioning some important points here for your reference to complete before starting the actual node addition.

1.Make physical connections to the node
2.Install OS on the node. (Make sure it is the same version as it is on all other cluster nodes)
3.Create necessary user accounts. We are using role separation , so create grid and oracle users
4.Complete all network configurations. Update your DNS or /etc/hosts file to properly resolve the new node in the cluster. Please note that the leaf node may or may not have vip. It is not compulsory.
5.Make sure time synchronization is done. It is by CTSS or NTPD.
6.Create any directory structure needed in the new server.
7.Complete SSH setup ( use the automated script availabe in $GRID_HOME/oui/bin/runSSHSetup.sh)
8.Set kernerl parameters as per the document.
9.It is not mandatory that shared storage should be attached to the leaf node in the cluster.
10.Complete all the required packages installation.
11.Verify if you have enough free space for grid infra management repository.(It is optional step. Required only If you have cluster health monitor configured in your environment)

( Refer to my rac installation articles to complete all the above steps )

1) Flex cluster status ...(execute the following commands from the owner for clusterware owner, ie grid)

olsnodes -s -t ( To know the cluster nodes in my environment)
crsctl get cluster mode config ( To know cluster mode ie. flex or standard )
crsctl get cluster mode status ( To know the status of the cluster )
crsctl get node role config -all ( To know each nodes role ie hub or leaf )
crsctl get node role status -all ( To know the staus of each nodes role )
crsctl status res -t ( To know the complete cluster status )
srvctl status asm -detail ( Note from this command that asm will be running on hub nodes, not on leaf node)
srvctl status database -d orcl ( Note : database is running only on hub nodes, not on leaf node )
srvctl config gns ( gns is enabled in my environment )
oifcfg getif ( Note that i am using a separate network for asm in my environment )

Session Log:

[grid@rac12cnode1 ~]$ iduid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1020(asmadmin),1021(asmdba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[grid@rac12cnode1 ~]$ olsnodes
rac12cnode1
rac12cnode2
rac12cnode3
[grid@rac12cnode1 ~]$ crsctl get cluster mode status
Cluster is running in "flex" mode
[grid@rac12cnode1 ~]$ crsctl get node role config -all
Node 'rac12cnode1' configured role is 'hub'
Node 'rac12cnode2' configured role is 'hub'
Node 'rac12cnode3' configured role is 'hub'
[grid@rac12cnode1 ~]$ crsctl get node role status -all
Node 'rac12cnode1' active role is 'hub'
Node 'rac12cnode2' active role is 'hub'
Node 'rac12cnode3' active role is 'hub'
[grid@rac12cnode1 ~]$ asmcmd showclustermode
ASM cluster : Flex mode enabled
[grid@rac12cnode1 ~]$ srvctl config asm
ASM home: /u01/app/12.1.0/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM
[grid@rac12cnode1 ~]$ srvctl status asm -detail
ASM is running on rac12cnode1,rac12cnode2,rac12cnode3
ASM is enabled.
[grid@rac12cnode1 ~]$

[grid@rac12cnode1 ~]$ oclumon manage -get repsize
CRS-9003-Cluster Health Monitor is not supported in this configuration.[grid@rac12cnode1 ~]$

2) Verify the nodes readyness to add to the cluster with cluvfy utility ...
cluvfy stage -post hwos -n rac12cnode4
cluvfy comp peer -refnode rac12cnode1 -n rac12cnode4 -orainv oinstall -osdba dba -verbose
cluvfy stage -pre nodeadd -n rac12cnode4 -fixup -verbose
Session Log in my environment for the above commands

[grid@rac12cnode1 ~]$ cluvfy stage -post hwos -n rac12cnode4
Performing post-checks for hardware and operating system setup
Checking node reachability...
Node reachability check passed from node "rac12cnode1"

Checking user equivalence...
User equivalence check passed for user "grid"
Checking node connectivity...
Checking hosts config file...
Verification of the hosts config file successful
Check: Node connectivity using interfaces on subnet "10.11.0.0"
Node connectivity passed for subnet "10.11.0.0" with node(s) rac12cnode4
TCP connectivity check passed for subnet "10.11.0.0"

Check: Node connectivity using interfaces on subnet "10.10.10.0"
Node connectivity passed for subnet "10.10.10.0" with node(s) rac12cnode4
TCP connectivity check passed for subnet "10.10.10.0"

Check: Node connectivity using interfaces on subnet "192.168.1.0"
Node connectivity passed for subnet "192.168.1.0" with node(s) rac12cnode4
TCP connectivity check passed for subnet "192.168.1.0"

Node connectivity check passed
Checking multicast communication...
Checking subnet "10.10.10.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "10.10.10.0" for multicast communication with multicast group "224.0.0.251" passed.
Checking subnet "10.11.0.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "10.11.0.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Check for multiple users with UID value 0 passed
Time zone consistency check passed
Checking shared storage accessibility...
WARNING:
rac12cnode4:PRVF-7017 : Package cvuqdisk not installed
rac12cnode4
No shared storage found

Shared storage check failed on nodes "rac12cnode4"
Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed

Post-check for hardware and operating system setup was unsuccessful on all the nodes.
[grid@rac12cnode1 ~]$

[grid@rac12cnode1 ~]$ cluvfy comp peer -refnode rac12cnode1 -n rac12cnode4 -orainv oinstall -osdba dba -verbose
Verifying peer compatibility
Checking peer compatibility...
Compatibility check: Physical memory [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 2.9326GB (3075076.0KB) 2.9326GB (3075076.0KB) matched
Physical memory check passed
Compatibility check: Available memory [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 2.7669GB (2901284.0KB) 908.1406MB (929936.0KB) matched
Available memory check passed
Compatibility check: Swap space [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 2GB (2097148.0KB) 2GB (2097148.0KB) matched
Swap space check passed
Compatibility check: Free disk space for "/usr" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 7.5781GB (7946240.0KB) 7.6396GB (8010752.0KB) mismatched
Free disk space check failed
Compatibility check: Free disk space for "/var" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 7.5781GB (7946240.0KB) 7.6396GB (8010752.0KB) mismatched
Free disk space check failed
Compatibility check: Free disk space for "/etc" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 7.5781GB (7946240.0KB) 7.6396GB (8010752.0KB) mismatched
Free disk space check failed
Compatibility check: Free disk space for "/sbin" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 7.5781GB (7946240.0KB) 7.6396GB (8010752.0KB) mismatched
Free disk space check failed
Compatibility check: Free disk space for "/tmp" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 7.5781GB (7946240.0KB) 7.6396GB (8010752.0KB) mismatched
Free disk space check failed
Compatibility check: Free disk space for "/u01/app/12.1.0/grid" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 24.4902GB (2.5679872E7KB) 17.9102GB (1.878016E7KB) matched
Free disk space check passed
Compatibility check: User existence for "grid" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 grid(1100) grid(1100) matched
User existence for "grid" check passed
Compatibility check: Group existence for "oinstall" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 oinstall(1000) oinstall(1000) matched
Group existence for "oinstall" check passed
Compatibility check: Group existence for "dba" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 dba(1031) dba(1031) matched
Group existence for "dba" check passed
Compatibility check: Group membership for "grid" in "oinstall (Primary)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 yes yes matched
Group membership for "grid" in "oinstall (Primary)" check passed
Compatibility check: Group membership for "grid" in "dba" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 no no matched
Group membership for "grid" in "dba" check passed
Compatibility check: Run level [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 3 3 matched
Run level check passed
Compatibility check: System architecture [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 x86_64 x86_64 matched
System architecture check passed
Compatibility check: Kernel version [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 2.6.39-400.109.1.el6uek.x86_64 2.6.39-400.109.1.el6uek.x86_64 matched
Kernel version check passed
Compatibility check: Kernel param "semmsl" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 250 250 matched
Kernel param "semmsl" check passed
Compatibility check: Kernel param "semmns" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 32000 32000 matched
Kernel param "semmns" check passed
Compatibility check: Kernel param "semopm" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 100 100 matched
Kernel param "semopm" check passed
Compatibility check: Kernel param "semmni" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 128 128 matched
Kernel param "semmni" check passed
Compatibility check: Kernel param "shmmax" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 4398046511104 4398046511104 matched
Kernel param "shmmax" check passed
Compatibility check: Kernel param "shmmni" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 4096 4096 matched
Kernel param "shmmni" check passed
Compatibility check: Kernel param "shmall" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 0 0 matched
Kernel param "shmall" check passed
Compatibility check: Kernel param "file-max" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 6815744 6815744 matched
Kernel param "file-max" check passed
Compatibility check: Kernel param "ip_local_port_range" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 9000 65500 9000 65500 matched
Kernel param "ip_local_port_range" check passed
Compatibility check: Kernel param "rmem_default" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 262144 262144 matched
Kernel param "rmem_default" check passed
Compatibility check: Kernel param "rmem_max" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 4194304 4194304 matched
Kernel param "rmem_max" check passed
Compatibility check: Kernel param "wmem_default" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 262144 262144 matched
Kernel param "wmem_default" check passed
Compatibility check: Kernel param "wmem_max" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 1048576 1048576 matched
Kernel param "wmem_max" check passed
Compatibility check: Kernel param "aio-max-nr" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 1048576 1048576 matched
Kernel param "aio-max-nr" check passed
Compatibility check: Package existence for "binutils" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2-5.36.el6 matched
Package existence for "binutils" check passed
Compatibility check: Package existence for "compat-libcap1" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 compat-libcap1-1.10-1 compat-libcap1-1.10-1 matched
Package existence for "compat-libcap1" check passed
Compatibility check: Package existence for "compat-libstdc++-33 (x86_64)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 compat-libstdc++-33-3.2.3-69.el6 (x86_64) compat-libstdc++-33-3.2.3-69.el6 (x86_64) matched
Package existence for "compat-libstdc++-33 (x86_64)" check passed
Compatibility check: Package existence for "libgcc (x86_64)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 libgcc-4.4.7-3.el6 (x86_64) libgcc-4.4.7-3.el6 (x86_64) matched
Package existence for "libgcc (x86_64)" check passed
Compatibility check: Package existence for "libstdc++ (x86_64)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 libstdc++-4.4.7-3.el6 (x86_64) libstdc++-4.4.7-3.el6 (x86_64) matched
Package existence for "libstdc++ (x86_64)" check passed
Compatibility check: Package existence for "libstdc++-devel (x86_64)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 libstdc++-devel-4.4.7-3.el6 (x86_64) libstdc++-devel-4.4.7-3.el6 (x86_64) matched
Package existence for "libstdc++-devel (x86_64)" check passed
Compatibility check: Package existence for "sysstat" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 sysstat-9.0.4-20.el6 sysstat-9.0.4-20.el6 matched
Package existence for "sysstat" check passed
Compatibility check: Package existence for "gcc" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 gcc-4.4.7-3.el6 gcc-4.4.7-3.el6 matched
Package existence for "gcc" check passed
Compatibility check: Package existence for "gcc-c++" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 gcc-c++-4.4.7-3.el6 gcc-c++-4.4.7-3.el6 matched
Package existence for "gcc-c++" check passed
Compatibility check: Package existence for "ksh" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 ksh-20100621-19.el6_4.4 ksh-20100621-19.el6_4.4 matched
Package existence for "ksh" check passed
Compatibility check: Package existence for "make" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 make-3.81-20.el6 make-3.81-20.el6 matched
Package existence for "make" check passed
Compatibility check: Package existence for "glibc (x86_64)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 glibc-2.12-1.107.el6_4.2 (x86_64) glibc-2.12-1.107.el6_4.2 (x86_64) matched
Package existence for "glibc (x86_64)" check passed
Compatibility check: Package existence for "glibc-devel (x86_64)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 glibc-devel-2.12-1.107.el6_4.2 (x86_64) glibc-devel-2.12-1.107.el6_4.2 (x86_64) matched
Package existence for "glibc-devel (x86_64)" check passed
Compatibility check: Package existence for "libaio (x86_64)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 libaio-0.3.107-10.el6 (x86_64) libaio-0.3.107-10.el6 (x86_64) matched
Package existence for "libaio (x86_64)" check passed
Compatibility check: Package existence for "libaio-devel (x86_64)" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 libaio-devel-0.3.107-10.el6 (x86_64) libaio-devel-0.3.107-10.el6 (x86_64) matched
Package existence for "libaio-devel (x86_64)" check passed
Compatibility check: Package existence for "nfs-utils" [reference node: rac12cnode1]
Node Name Status Ref. node status Comment
------------ ------------------------ ------------------------ ----------
rac12cnode4 nfs-utils-1.2.3-36.el6 nfs-utils-1.2.3-36.el6 matched
Package existence for "nfs-utils" check passed
Verification of peer compatibility was unsuccessful.
Checks did not pass for the following node(s):
rac12cnode4
[grid@rac12cnode1 ~]$

[grid@rac12cnode1 ~]$ cluvfy stage -pre nodeadd -n rac12cnode4 -fixup -verbose
Performing pre-checks for node addition
Checking node reachability...
Check: Node reachability from node "rac12cnode1"
Destination Node Reachable?
------------------------------------ ------------------------
rac12cnode4 yes
Result: Node reachability check passed from node "rac12cnode1"

Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
rac12cnode4 passed
Result: User equivalence check passed for user "grid"
Check: Package existence for "cvuqdisk"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode2 cvuqdisk-1.0.9-1 cvuqdisk-1.0.9-1 passed
rac12cnode1 cvuqdisk-1.0.9-1 cvuqdisk-1.0.9-1 passed
rac12cnode4 missing cvuqdisk-1.0.9-1 failed
rac12cnode3 cvuqdisk-1.0.9-1 cvuqdisk-1.0.9-1 passed
Result: Package existence check failed for "cvuqdisk"
Checking CRS integrity...
The Oracle Clusterware is healthy on node "rac12cnode1"
CRS integrity check passed
Clusterware version consistency passed.
Checking shared resources...
Checking CRS home location...
Location check passed for: "/u01/app/12.1.0/grid"
Result: Shared resources check for node addition passed

Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
rac12cnode1 passed
rac12cnode2 passed
rac12cnode3 passed
rac12cnode4 passed
Verification of the hosts config file successful

Interface information for node "rac12cnode1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.131 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:22 1500
eth0 192.168.1.135 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:22 1500
eth1 10.10.10.131 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:09 1500
eth1 169.254.251.99 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:09 1500
eth2 10.11.0.135 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0B 1500

Interface information for node "rac12cnode2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.132 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.140 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.139 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.136 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth1 10.10.10.132 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0C 1500
eth1 169.254.169.69 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0C 1500
eth2 10.11.0.136 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:11 1500

Interface information for node "rac12cnode3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.133 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:24 1500
eth0 192.168.1.137 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:24 1500
eth1 10.10.10.133 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:12 1500
eth1 169.254.177.41 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:12 1500
eth2 10.11.0.137 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:13 1500

Interface information for node "rac12cnode4"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.134 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:1E 1500
eth1 10.10.10.134 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:14 1500
eth2 10.11.0.138 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:15 1500

Check: Node connectivity using interfaces on subnet "192.168.1.0"
Check: Node connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode2[192.168.1.139] rac12cnode2[192.168.1.132] yes
rac12cnode2[192.168.1.139] rac12cnode3[192.168.1.133] yes
rac12cnode2[192.168.1.139] rac12cnode1[192.168.1.135] yes
rac12cnode2[192.168.1.139] rac12cnode2[192.168.1.140] yes
rac12cnode2[192.168.1.139] rac12cnode1[192.168.1.131] yes
rac12cnode2[192.168.1.139] rac12cnode3[192.168.1.137] yes
rac12cnode2[192.168.1.139] rac12cnode2[192.168.1.136] yes
rac12cnode2[192.168.1.139] rac12cnode4[192.168.1.134] yes
rac12cnode2[192.168.1.132] rac12cnode3[192.168.1.133] yes
rac12cnode2[192.168.1.132] rac12cnode1[192.168.1.135] yes
rac12cnode2[192.168.1.132] rac12cnode2[192.168.1.140] yes
rac12cnode2[192.168.1.132] rac12cnode1[192.168.1.131] yes
rac12cnode2[192.168.1.132] rac12cnode3[192.168.1.137] yes
rac12cnode2[192.168.1.132] rac12cnode2[192.168.1.136] yes
rac12cnode2[192.168.1.132] rac12cnode4[192.168.1.134] yes
rac12cnode3[192.168.1.133] rac12cnode1[192.168.1.135] yes
rac12cnode3[192.168.1.133] rac12cnode2[192.168.1.140] yes
rac12cnode3[192.168.1.133] rac12cnode1[192.168.1.131] yes
rac12cnode3[192.168.1.133] rac12cnode3[192.168.1.137] yes
rac12cnode3[192.168.1.133] rac12cnode2[192.168.1.136] yes
rac12cnode3[192.168.1.133] rac12cnode4[192.168.1.134] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.140] yes
rac12cnode1[192.168.1.135] rac12cnode1[192.168.1.131] yes
rac12cnode1[192.168.1.135] rac12cnode3[192.168.1.137] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.136] yes
rac12cnode1[192.168.1.135] rac12cnode4[192.168.1.134] yes
rac12cnode2[192.168.1.140] rac12cnode1[192.168.1.131] yes
rac12cnode2[192.168.1.140] rac12cnode3[192.168.1.137] yes
rac12cnode2[192.168.1.140] rac12cnode2[192.168.1.136] yes
rac12cnode2[192.168.1.140] rac12cnode4[192.168.1.134] yes
rac12cnode1[192.168.1.131] rac12cnode3[192.168.1.137] yes
rac12cnode1[192.168.1.131] rac12cnode2[192.168.1.136] yes
rac12cnode1[192.168.1.131] rac12cnode4[192.168.1.134] yes
rac12cnode3[192.168.1.137] rac12cnode2[192.168.1.136] yes
rac12cnode3[192.168.1.137] rac12cnode4[192.168.1.134] yes
rac12cnode2[192.168.1.136] rac12cnode4[192.168.1.134] yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rac12cnode2,rac12cnode3,rac12cnode1,rac12cnode4

Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode2:192.168.1.139 rac12cnode2:192.168.1.132 passed
rac12cnode2:192.168.1.139 rac12cnode3:192.168.1.133 passed
rac12cnode2:192.168.1.139 rac12cnode1:192.168.1.135 passed
rac12cnode2:192.168.1.139 rac12cnode2:192.168.1.140 passed
rac12cnode2:192.168.1.139 rac12cnode1:192.168.1.131 passed
rac12cnode2:192.168.1.139 rac12cnode3:192.168.1.137 passed
rac12cnode2:192.168.1.139 rac12cnode2:192.168.1.136 passed
rac12cnode2:192.168.1.139 rac12cnode4:192.168.1.134 passed
Result: TCP connectivity check passed for subnet "192.168.1.0"

Check: Node connectivity using interfaces on subnet "10.10.10.0"
Check: Node connectivity of subnet "10.10.10.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode1[10.10.10.131] rac12cnode4[10.10.10.134] yes
rac12cnode1[10.10.10.131] rac12cnode2[10.10.10.132] yes
rac12cnode1[10.10.10.131] rac12cnode3[10.10.10.133] yes
rac12cnode4[10.10.10.134] rac12cnode2[10.10.10.132] yes
rac12cnode4[10.10.10.134] rac12cnode3[10.10.10.133] yes
rac12cnode2[10.10.10.132] rac12cnode3[10.10.10.133] yes
Result: Node connectivity passed for subnet "10.10.10.0" with node(s) rac12cnode1,rac12cnode4,rac12cnode2,rac12cnode3

Check: TCP connectivity of subnet "10.10.10.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode1:10.10.10.131 rac12cnode4:10.10.10.134 passed
rac12cnode1:10.10.10.131 rac12cnode2:10.10.10.132 passed
rac12cnode1:10.10.10.131 rac12cnode3:10.10.10.133 passed
Result: TCP connectivity check passed for subnet "10.10.10.0"

Check: Node connectivity using interfaces on subnet "10.11.0.0"
Check: Node connectivity of subnet "10.11.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode4[10.11.0.138] rac12cnode1[10.11.0.135] yes
rac12cnode4[10.11.0.138] rac12cnode2[10.11.0.136] yes
rac12cnode4[10.11.0.138] rac12cnode3[10.11.0.137] yes
rac12cnode1[10.11.0.135] rac12cnode2[10.11.0.136] yes
rac12cnode1[10.11.0.135] rac12cnode3[10.11.0.137] yes
rac12cnode2[10.11.0.136] rac12cnode3[10.11.0.137] yes
Result: Node connectivity passed for subnet "10.11.0.0" with node(s) rac12cnode4,rac12cnode1,rac12cnode2,rac12cnode3

Check: TCP connectivity of subnet "10.11.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode4:10.11.0.138 rac12cnode1:10.11.0.135 passed
rac12cnode4:10.11.0.138 rac12cnode2:10.11.0.136 passed
rac12cnode4:10.11.0.138 rac12cnode3:10.11.0.137 passed
Result: TCP connectivity check passed for subnet "10.11.0.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed for subnet "10.10.10.0".
Subnet mask consistency check passed for subnet "10.11.0.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "10.10.10.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "10.10.10.0" for multicast communication with multicast group "224.0.0.251" passed.
Checking subnet "10.11.0.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "10.11.0.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Task ASM Integrity check started...
Checking if connectivity exists across cluster nodes on the ASM network
Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
rac12cnode4 passed
Verification of the hosts config file successful

Interface information for node "rac12cnode4"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.134 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:1E 1500
eth1 10.10.10.134 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:14 1500
eth2 10.11.0.138 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:15 1500

Check: Node connectivity using interfaces on subnet "10.11.0.0"
Check: Node connectivity of subnet "10.11.0.0"
Result: Node connectivity passed for subnet "10.11.0.0" with node(s) rac12cnode4

Check: TCP connectivity of subnet "10.11.0.0"
Result: TCP connectivity check passed for subnet "10.11.0.0"

Result: Node connectivity check passed
Network connectivity check across cluster nodes on the ASM network passed
Task ASM Integrity check passed...
Check: Total memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 2.9326GB (3075076.0KB) 4GB (4194304.0KB) failed
rac12cnode4 2.9326GB (3075076.0KB) 4GB (4194304.0KB) failed
Result: Total memory check failed
Check: Available memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 892.875MB (914304.0KB) 50MB (51200.0KB) passed
rac12cnode4 2.766GB (2900356.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed
Check: Swap space
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 2GB (2097148.0KB) 2.9326GB (3075076.0KB) failed
rac12cnode4 2GB (2097148.0KB) 2.9326GB (3075076.0KB) failed
Result: Swap space check failed
Check: Free disk space for "rac12cnode1:/usr,rac12cnode1:/var,rac12cnode1:/etc,rac12cnode1:/sbin,rac12cnode1:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr rac12cnode1 / 7.6396GB 1.0635GB passed
/var rac12cnode1 / 7.6396GB 1.0635GB passed
/etc rac12cnode1 / 7.6396GB 1.0635GB passed
/sbin rac12cnode1 / 7.6396GB 1.0635GB passed
/tmp rac12cnode1 / 7.6396GB 1.0635GB passed
Result: Free disk space check passed for "rac12cnode1:/usr,rac12cnode1:/var,rac12cnode1:/etc,rac12cnode1:/sbin,rac12cnode1:/tmp"
Check: Free disk space for "rac12cnode4:/usr,rac12cnode4:/var,rac12cnode4:/etc,rac12cnode4:/sbin,rac12cnode4:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/usr rac12cnode4 / 7.5762GB 1.0635GB passed
/var rac12cnode4 / 7.5762GB 1.0635GB passed
/etc rac12cnode4 / 7.5762GB 1.0635GB passed
/sbin rac12cnode4 / 7.5762GB 1.0635GB passed
/tmp rac12cnode4 / 7.5762GB 1.0635GB passed
Result: Free disk space check passed for "rac12cnode4:/usr,rac12cnode4:/var,rac12cnode4:/etc,rac12cnode4:/sbin,rac12cnode4:/tmp"
Check: Free disk space for "rac12cnode1:/u01/app/12.1.0/grid"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/12.1.0/grid rac12cnode1 /u01 17.8799GB 6.9GB passed
Result: Free disk space check passed for "rac12cnode1:/u01/app/12.1.0/grid"
Check: Free disk space for "rac12cnode4:/u01/app/12.1.0/grid"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/u01/app/12.1.0/grid rac12cnode4 /u01 24.4902GB 6.9GB passed
Result: Free disk space check passed for "rac12cnode4:/u01/app/12.1.0/grid"
Check: User existence for "grid"
Node Name Status Comment
------------ ------------------------ ------------------------
rac12cnode1 passed exists(1100)
rac12cnode4 passed exists(1100)
Checking for multiple users with UID value 1100
Result: Check for multiple users with UID value 1100 passed
Result: User existence check passed for "grid"
Check: Run level
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 3 3,5 passed
rac12cnode4 3 3,5 passed
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
rac12cnode1 hard 65536 65536 passed
rac12cnode4 hard 65536 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
rac12cnode1 soft 65536 1024 passed
rac12cnode4 soft 1024 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
rac12cnode1 hard 16384 16384 passed
rac12cnode4 hard 16384 16384 passed
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
rac12cnode1 soft 16384 2047 passed
rac12cnode4 soft 16384 2047 passed
Result: Soft limits check passed for "maximum user processes"
Check: System architecture
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 x86_64 x86_64 passed
rac12cnode4 x86_64 x86_64 passed
Result: System architecture check passed
Check: Kernel version
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 2.6.39-400.109.1.el6uek.x86_64 2.6.32 passed
rac12cnode4 2.6.39-400.109.1.el6uek.x86_64 2.6.32 passed
Result: Kernel version check passed
Check: Kernel parameter for "semmsl"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 250 250 250 passed
rac12cnode4 250 250 250 passed
Result: Kernel parameter check passed for "semmsl"
Check: Kernel parameter for "semmns"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 32000 32000 32000 passed
rac12cnode4 32000 32000 32000 passed
Result: Kernel parameter check passed for "semmns"
Check: Kernel parameter for "semopm"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 100 100 100 passed
rac12cnode4 100 100 100 passed
Result: Kernel parameter check passed for "semopm"
Check: Kernel parameter for "semmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 128 128 128 passed
rac12cnode4 128 128 128 passed
Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 4398046511104 4398046511104 1574438912 passed
rac12cnode4 4398046511104 4398046511104 1574438912 passed
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 4096 4096 4096 passed
rac12cnode4 4096 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmall"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 1073741824 1073741824 307507 passed
rac12cnode4 1073741824 1073741824 307507 passed
Result: Kernel parameter check passed for "shmall"
Check: Kernel parameter for "file-max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 6815744 6815744 6815744 passed
rac12cnode4 6815744 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"
Check: Kernel parameter for "ip_local_port_range"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
rac12cnode4 between 9000 & 65500 between 9000 & 65500 between 9000 & 65535 passed
Result: Kernel parameter check passed for "ip_local_port_range"
Check: Kernel parameter for "rmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 262144 262144 262144 passed
rac12cnode4 262144 262144 262144 passed
Result: Kernel parameter check passed for "rmem_default"
Check: Kernel parameter for "rmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 4194304 4194304 4194304 passed
rac12cnode4 4194304 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"
Check: Kernel parameter for "wmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 262144 262144 262144 passed
rac12cnode4 262144 262144 262144 passed
Result: Kernel parameter check passed for "wmem_default"
Check: Kernel parameter for "wmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 1048576 1048576 1048576 passed
rac12cnode4 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "wmem_max"
Check: Kernel parameter for "aio-max-nr"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac12cnode1 1048576 1048576 1048576 passed
rac12cnode4 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"
Check: Package existence for "binutils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2 passed
rac12cnode4 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2 passed
Result: Package existence check passed for "binutils"
Check: Package existence for "compat-libcap1"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
rac12cnode4 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
Result: Package existence check passed for "compat-libcap1"
Check: Package existence for "compat-libstdc++-33(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
rac12cnode4 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"
Check: Package existence for "libgcc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 libgcc(x86_64)-4.4.7-3.el6 libgcc(x86_64)-4.4.4 passed
rac12cnode4 libgcc(x86_64)-4.4.7-3.el6 libgcc(x86_64)-4.4.4 passed
Result: Package existence check passed for "libgcc(x86_64)"
Check: Package existence for "libstdc++(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 libstdc++(x86_64)-4.4.7-3.el6 libstdc++(x86_64)-4.4.4 passed
rac12cnode4 libstdc++(x86_64)-4.4.7-3.el6 libstdc++(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++(x86_64)"
Check: Package existence for "libstdc++-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 libstdc++-devel(x86_64)-4.4.7-3.el6 libstdc++-devel(x86_64)-4.4.4 passed
rac12cnode4 libstdc++-devel(x86_64)-4.4.7-3.el6 libstdc++-devel(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"
Check: Package existence for "sysstat"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
rac12cnode4 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
Result: Package existence check passed for "sysstat"
Check: Package existence for "gcc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 gcc-4.4.7-3.el6 gcc-4.4.4 passed
rac12cnode4 gcc-4.4.7-3.el6 gcc-4.4.4 passed
Result: Package existence check passed for "gcc"
Check: Package existence for "gcc-c++"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 gcc-c++-4.4.7-3.el6 gcc-c++-4.4.4 passed
rac12cnode4 gcc-c++-4.4.7-3.el6 gcc-c++-4.4.4 passed
Result: Package existence check passed for "gcc-c++"
Check: Package existence for "ksh"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 ksh-20100621-19.el6_4.4 ksh-... passed
rac12cnode4 ksh-20100621-19.el6_4.4 ksh-... passed
Result: Package existence check passed for "ksh"
Check: Package existence for "make"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 make-3.81-20.el6 make-3.81 passed
rac12cnode4 make-3.81-20.el6 make-3.81 passed
Result: Package existence check passed for "make"
Check: Package existence for "glibc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 glibc(x86_64)-2.12-1.107.el6_4.2 glibc(x86_64)-2.12 passed
rac12cnode4 glibc(x86_64)-2.12-1.107.el6_4.2 glibc(x86_64)-2.12 passed
Result: Package existence check passed for "glibc(x86_64)"
Check: Package existence for "glibc-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 glibc-devel(x86_64)-2.12-1.107.el6_4.2 glibc-devel(x86_64)-2.12 passed
rac12cnode4 glibc-devel(x86_64)-2.12-1.107.el6_4.2 glibc-devel(x86_64)-2.12 passed
Result: Package existence check passed for "glibc-devel(x86_64)"
Check: Package existence for "libaio(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
rac12cnode4 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio(x86_64)"
Check: Package existence for "libaio-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
rac12cnode4 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio-devel(x86_64)"
Check: Package existence for "nfs-utils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
rac12cnode1 nfs-utils-1.2.3-36.el6 nfs-utils-1.2.3-15 passed
rac12cnode4 nfs-utils-1.2.3-36.el6 nfs-utils-1.2.3-15 passed
Result: Package existence check passed for "nfs-utils"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID
Result: Current group ID check passed
Starting check for consistency of primary group of root user
Node Name Status
------------------------------------ ------------------------
rac12cnode1 passed
rac12cnode4 passed
Check for consistency of root user's primary group passed
Check: Group existence for "asmadmin"
Node Name Status Comment
------------ ------------------------ ------------------------
rac12cnode1 passed exists
rac12cnode4 passed exists
Result: Group existence check passed for "asmadmin"
Check: Group existence for "asmdba"
Node Name Status Comment
------------ ------------------------ ------------------------
rac12cnode1 passed exists
rac12cnode4 passed exists
Result: Group existence check passed for "asmdba"
Checking ASMLib configuration.
Node Name Status
------------------------------------ ------------------------
rac12cnode1 passed
rac12cnode4 passed
Result: Check for ASMLib configuration passed.
Checking OCR integrity...
OCR integrity check passed
Checking Oracle Cluster Voting Disk configuration...
Oracle Cluster Voting Disk configuration check passed
Check: Time zone consistency
Result: Time zone consistency check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
rac12cnode1 yes
rac12cnode4 yes
Result: Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
Checking whether NTP daemon or service is using UDP port 123 on all nodes
Check for NTP daemon or service using UDP port 123
Node Name Port Open?
------------------------------------ ------------------------
rac12cnode1 yes
rac12cnode4 yes
NTP common Time Server Check started...
NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Checking on nodes "[rac12cnode1, rac12cnode4]"...
Check: Clock time offset from NTP Time Server
Time Server: .LOCL.
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
------------ ------------------------ ------------------------
rac12cnode1 0.0 passed
rac12cnode4 0.0 passed
Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[rac12cnode1, rac12cnode4]".
Clock time offset check passed
Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking to make sure user "grid" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
rac12cnode1 passed does not exist
rac12cnode4 passed does not exist
Result: User "grid" is not part of "root" group. Check passed
Checking integrity of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
"domain" and "search" entries do not coexist in any "/etc/resolv.conf" file
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
"domain" entry does not exist in any "/etc/resolv.conf" file
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
"search" entry does not exist in any "/etc/resolv.conf" file
Checking DNS response time for an unreachable node
Node Name Status
------------------------------------ ------------------------
rac12cnode1 failed
rac12cnode4 failed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: rac12cnode1,rac12cnode4
Check for integrity of file "/etc/resolv.conf" failed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
Checking if "hosts" entry in file "/etc/nsswitch.conf" is consistent across nodes...
Checking file "/etc/nsswitch.conf" to make sure that only one "hosts" entry is defined
More than one "hosts" entry does not exist in any "/etc/nsswitch.conf" file
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed

Checking GNS integrity...
Checking if the GNS VIP belongs to same subnet as the public network...
Public network subnets "192.168.1.0, 192.168.1.0" match with the GNS VIP "192.168.1.0, 192.168.1.0"
Checking if the GNS VIP is a valid address...
GNS VIP "192.168.1.140" resolves to a valid IP address
Checking the status of GNS VIP...
Checking status of GNS resource...
Node Running? Enabled?
------------ ------------------------ ------------------------
rac12cnode1 no yes
rac12cnode2 yes yes
rac12cnode3 no yes
GNS resource configuration check passed
Checking status of GNS VIP resource...
Node Running? Enabled?
------------ ------------------------ ------------------------
rac12cnode1 no yes
rac12cnode2 yes yes
rac12cnode3 no yes
GNS VIP resource configuration check passed.
GNS integrity check passed
Checking Flex Cluster node role configuration...
Flex Cluster node role configuration check passed
******************************************************************************************
Following is the list of fixable prerequisites selected to fix in this session
******************************************************************************************
-------------- --------------- ----------------
Check failed. Failed on nodes Reboot required?
-------------- --------------- ----------------
Package: cvuqdisk-1.0.9-1 rac12cnode4 no

Execute "/tmp/CVU_12.1.0.1.0_grid/runfixup.sh" as root user on nodes "rac12cnode4" to perform the fix up operations manually
Press ENTER key to continue after execution of "/tmp/CVU_12.1.0.1.0_grid/runfixup.sh" has completed on nodes "rac12cnode4"

Fix: Package: cvuqdisk-1.0.9-1
Node Name Status
------------------------------------ ------------------------
rac12cnode4 failed
ERROR:
PRVG-9023 : Manual fix up command "/tmp/CVU_12.1.0.1.0_grid/runfixup.sh" was not issued by root user on node "rac12cnode4"
Result: "Package: cvuqdisk-1.0.9-1" could not be fixed on nodes "rac12cnode4"
Fix up operations for selected fixable prerequisites were unsuccessful on nodes "rac12cnode4"
Pre-check for node addition was unsuccessful on all the nodes.
[grid@rac12cnode1 ~]$

[root@rac12cnode4 ~]# /tmp/CVU_12.1.0.1.0_grid/runfixup.sh
All Fix-up operations were completed successfully.
[root@rac12cnode4 ~]#

I had some DNS issue which caused some test cases to fail. But i am proceeding further to see if that actually halts me to proceed with the node addition. In pre 12c versions( standard cluster), we were able to proceed without DNS and with just /etc/hosts. But in 12c, particularly in flex cluster, it is not allowing me to proceed further with the node addition unless i have proper DNS entries. I am going to show you in a short while the issue and then i will modify my DNS to have proper entries to my RAC servers.

3) Now let me start adding the leaf node to the flex cluster.
addnode.sh -silent "CLUSTER_NEW_NODES={rac12cnode4}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac12cnode4-vip}" "CLUSTER_NEW_NODE_ROLES={leaf}"
or
addnode.sh -silent "CLUSTER_NEW_NODES={rac12cnode4}" "CLUSTER_NEW_NODE_ROLES={leaf}"
NOTE : When you are adding a node as leaf node to cluster, you may or may not specify the VIP address for that node. It is not mandatory. So i choose to run without VIP details ...

[grid@rac12cnode1 addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={rac12cnode4}" "CLUSTER_NEW_NODE_ROLES={leaf}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 7460 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1072 MB Passed
[FATAL] [INS-13013] Target environment does not meet some mandatory requirements.
CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2014-03-11_12-51-25AM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2014-03-11_12-51-25AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
[grid@rac12cnode1 addnode]$

I got issue and it failed to add a node. I reviewed the log file. It is failed on some pre-reqs. So thought to execute the same command , with the variable IGNORE_PREADDNODE_CHECKS=Y as we were doing in pre 12c releases.

[grid@rac12cnode1 addnode]$ export IGNORE_PREADDNODE_CHECKS=Y
[grid@rac12cnode1 addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={rac12cnode4}" "CLUSTER_NEW_NODE_ROLES={leaf}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 7460 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1073 MB Passed
[FATAL] [INS-13013] Target environment does not meet some mandatory requirements.
CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2014-03-11_12-59-26AM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2014-03-11_12-59-26AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
[grid@rac12cnode1 addnode]$

Now i reviewed the logs and there are three errors. Out of three one is critical and two are ignorable.

INFO: ------------------List of failed Tasks------------------
INFO: *********************************************
INFO: Physical Memory: This is a prerequisite condition to test whether the system has at least 4GB (4194304.0KB) of total physical memory.
INFO: Severity:IGNORABLE
INFO: OverallStatus:VERIFICATION_FAILED
INFO: *********************************************
INFO: Swap Size: This is a prerequisite condition to test whether sufficient total swap space is available on the system.
INFO: Severity:IGNORABLE
INFO: OverallStatus:VERIFICATION_FAILED
INFO: *********************************************
INFO: Task resolv.conf Integrity: This task checks consistency of file /etc/resolv.conf file across nodes
INFO: Severity:CRITICAL
INFO: OverallStatus:VERIFICATION_FAILED
INFO: -----------------End of failed Tasks List----------------

Now here is the place where it did not allow me to go further with all mimics as we were doing in previous releases. So i have to have entries in DNS to proceed further. So i have completed DNS configuration and added all nodes to DNS server.

[grid@rac12cnode1 addnode]$ nslookup rac12cnode4
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode4.localdomain
Address: 192.168.1.134
[grid@rac12cnode1 addnode]$ nslookup 192.168.1.134
Server: 192.168.1.51
Address: 192.168.1.51#53
134.1.168.192.in-addr.arpa name = rac12cnode4.localdomain.
[grid@rac12cnode1 addnode]$ nslookup rac12cnode3
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode3.localdomain
Address: 192.168.1.133
[grid@rac12cnode1 addnode]$ nslookup rac12cnode2
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode2.localdomain
Address: 192.168.1.132
[grid@rac12cnode1 addnode]$
[root@rac12cnode4 ~]# nslookup 192.168.1.134
Server: 192.168.1.51
Address: 192.168.1.51#53
134.1.168.192.in-addr.arpa name = rac12cnode4.localdomain.
[root@rac12cnode4 ~]# nslookup 192.168.1.133
Server: 192.168.1.51
Address: 192.168.1.51#53
133.1.168.192.in-addr.arpa name = rac12cnode3.localdomain.
[root@rac12cnode4 ~]# nslookup 192.168.1.132
Server: 192.168.1.51
Address: 192.168.1.51#53
132.1.168.192.in-addr.arpa name = rac12cnode2.localdomain.
[root@rac12cnode4 ~]# nslookup 192.168.1.131
Server: 192.168.1.51
Address: 192.168.1.51#53
131.1.168.192.in-addr.arpa name = rac12cnode1.localdomain.
[root@rac12cnode4 ~]# nslookup rac12cnode4
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode4.localdomain
Address: 192.168.1.134
[root@rac12cnode4 ~]# nslookup rac12cnode3
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode3.localdomain
Address: 192.168.1.133
[root@rac12cnode4 ~]# nslookup rac12cnode2
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode2.localdomain
Address: 192.168.1.132
[root@rac12cnode4 ~]# nslookup rac12cnode1
Server: 192.168.1.51
Address: 192.168.1.51#53
Name: rac12cnode1.localdomain
Address: 192.168.1.131
[root@rac12cnode4 ~]#

Now i am able to successfully complete the execution of addnode command. But as part of this execution it asked me to execute root.sh.

[grid@rac12cnode1 addnode]$ ./addnode.sh -silent "CLUSTER_NEW_NODES={rac12cnode4}" "CLUSTER_NEW_NODE_ROLES={leaf}"
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 120 MB. Actual 7460 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1058 MB Passed
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2014-03-11_01-37-37AM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2014-03-11_01-37-37AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 9% Done.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2014-03-11_01-37-37AM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 15% Done.
Copying files to node in progress.
Copying files to node successful.
.................................................. 79% Done.
Saving cluster inventory in progress.
.................................................. 87% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/12.1.0/grid was successful.
Please check '/tmp/silentInstall.log' for more details.
As a root user, execute the following script(s):
1. /u01/app/12.1.0/grid/root.sh
Execute /u01/app/12.1.0/grid/root.sh on the following nodes:
[rac12cnode4]
The scripts can be executed in parallel on all the nodes. If there are any policy managed databases managed by cluster, proceed with the addnode procedure without executing the root.sh script. Ensure that root.sh script is executed after all the policy managed databases managed by clusterware are extended to the new nodes.
..........
Update Inventory in progress.
.................................................. 100% Done.
Update Inventory successful.
Successfully Setup Software.
[grid@rac12cnode1 addnode]$

When ran root.sh, it did not configure the cluster. Here i took some time to actually debug the issue.

[root@rac12cnode4 12.1.0]# /u01/app/12.1.0/grid/root.sh
Check /u01/app/12.1.0/grid/install/root_rac12cnode4_2014-03-11_02-19-08.log for the output of root script
[root@rac12cnode4 12.1.0]# cat /u01/app/12.1.0/grid/install/root_rac12cnode4_2014-03-11_02-19-08.log
Performing root user operation for Oracle 12c
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/12.1.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/crs/install /u01/app/12.1.0/grid/crs/install/roothas.pl

To configure Grid Infrastructure for a Cluster execute the following command as grid user:
/u01/app/12.1.0/grid/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.
[root@rac12cnode4 12.1.0]#

So it is just asking me to run config.sh to configure my cluster. But i don't know actually why is it asking me to run this script. Thinking it might be some change, i started running, config.sh , then as per my experience, i understood that i am going somewhere in wrong direction. Then i started debugging the scripts root.sh and all the scripts it runs.

Then realized the problem with the script /u01/app/12.1.0/grid/crs/config/rootconfig.sh. After careful review, i edited the oracle delivered script and uncommented the lines as shown below to allow the script to configure the cluster for leaf node.

[root@rac12cnode4 ~]# cp /u01/app/12.1.0/grid/crs/config/rootconfig.sh /u01/app/12.1.0/grid/crs/config/rootconfig.sh_bak_askm
[root@rac12cnode4 ~]# vi /u01/app/12.1.0/grid/crs/config/rootconfig.sh
[root@rac12cnode4 ~]# diff /u01/app/12.1.0/grid/crs/config/rootconfig.sh /u01/app/12.1.0/grid/crs/config/rootconfig.sh_bak_askm
33,37c33,37
< if [ "$ADDNODE" = "true" ]; then
< SW_ONLY=false
< HA_CONFIG=false
< fi
---
> #if [ "$ADDNODE" = "true" ]; then
> # SW_ONLY=false
> # HA_CONFIG=false
> #fi
[root@rac12cnode4 ~]#

Then ran root.sh again, which completed successfully this time.

[root@rac12cnode4 ~]# /u01/app/12.1.0/grid/root.sh
Check /u01/app/12.1.0/grid/install/root_rac12cnode4_2014-03-11_04-14-13.log for the output of root script
[root@rac12cnode4 ~]#

The above logfile shows following output while execution ...

[root@rac12cnode4 ~]# tail -f /u01/app/12.1.0/grid/install/root_rac12cnode4_2014-03-11_04-14-13.log
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.1.0/grid/crs/install/crsconfig_params
OLR initialization - successful
2014/03/11 04:16:23 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac12cnode4'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac12cnode4'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac12cnode4' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac12cnode4' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac12cnode4'
CRS-2672: Attempting to start 'ora.evmd' on 'rac12cnode4'
CRS-2676: Start of 'ora.evmd' on 'rac12cnode4' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac12cnode4'
CRS-2676: Start of 'ora.gpnpd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac12cnode4'
CRS-2676: Start of 'ora.gipcd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac12cnode4'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac12cnode4'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac12cnode4'
CRS-2676: Start of 'ora.diskmon' on 'rac12cnode4' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rac12cnode4'
CRS-2676: Start of 'ora.cssd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac12cnode4'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac12cnode4'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac12cnode4' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac12cnode4'
CRS-2676: Start of 'ora.storage' on 'rac12cnode4' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac12cnode4'
CRS-2676: Start of 'ora.crsd' on 'rac12cnode4' succeeded
CRS-6017: Processing resource auto-start for servers: rac12cnode4
CRS-6016: Resource auto-start has completed for server rac12cnode4
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/03/11 04:21:03 CLSRSC-343: Successfully started Oracle clusterware stack
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2014/03/11 04:21:11 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
^C
[root@rac12cnode4 ~]#

4) Post verification Steps :
On node 1:

[grid@rac12cnode1 addnode]$ olsnodes -s -trac12cnode1 Active Unpinned
rac12cnode2 Active Unpinned
rac12cnode3 Active Unpinned
rac12cnode4 Active Unpinned
[grid@rac12cnode1 addnode]$ clear
[grid@rac12cnode1 addnode]$ olsnodes -s -t
rac12cnode1 Active Unpinned
rac12cnode2 Active Unpinned
rac12cnode3 Active Unpinned
rac12cnode4 Active Unpinned
[grid@rac12cnode1 addnode]$ crsctl get cluster mode status
Cluster is running in "flex" mode
[grid@rac12cnode1 addnode]$ srvctl config gns
GNS is enabled.
[grid@rac12cnode1 addnode]$ oifcfg getif
eth0 192.168.1.0 global public
eth1 10.10.10.0 global cluster_interconnect
eth2 10.11.0.0 global asm
[grid@rac12cnode1 addnode]$ crsctl get node role config
Node 'rac12cnode1' configured role is 'hub'
[grid@rac12cnode1 addnode]$ asmcmd showclustermode
ASM cluster : Flex mode enabled
[grid@rac12cnode1 addnode]$ asmcmd showclusterstate
Normal
[grid@rac12cnode1 addnode]$ srvctl status asm -detail
ASM is running on rac12cnode1,rac12cnode2,rac12cnode3
ASM is enabled.
[grid@rac12cnode1 addnode]$ srvctl config asm
ASM home: /u01/app/12.1.0/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM
[grid@rac12cnode1 addnode]$ crsctl get node role config -all
Node 'rac12cnode1' configured role is 'hub'
Node 'rac12cnode2' configured role is 'hub'
Node 'rac12cnode3' configured role is 'hub'
Node 'rac12cnode4' configured role is 'leaf'
[grid@rac12cnode1 addnode]$ crsctl get node role status -all
Node 'rac12cnode1' active role is 'hub'
Node 'rac12cnode2' active role is 'hub'
Node 'rac12cnode3' active role is 'hub'
Node 'rac12cnode4' active role is 'leaf'
[grid@rac12cnode1 addnode]$ clear
[grid@rac12cnode1 addnode]$ crsctl status res -t | grep -i offline
OFFLINE OFFLINE rac12cnode4 STABLE
1 OFFLINE OFFLINE STABLE
[grid@rac12cnode1 addnode]$ crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.DATA.dg
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.LISTENER_LEAF.lsnr
OFFLINE OFFLINE rac12cnode4 STABLE
ora.net1.network
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.ons
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
ora.proxy_advm
ONLINE ONLINE rac12cnode1 STABLE
ONLINE ONLINE rac12cnode2 STABLE
ONLINE ONLINE rac12cnode3 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac12cnode2 STABLE
ora.asm
1 ONLINE ONLINE rac12cnode1 STABLE
2 ONLINE ONLINE rac12cnode2 STABLE
3 ONLINE ONLINE rac12cnode3 STABLE
ora.cvu
1 ONLINE ONLINE rac12cnode2 STABLE
ora.gns
1 ONLINE ONLINE rac12cnode2 STABLE
ora.gns.vip
1 ONLINE ONLINE rac12cnode2 STABLE
ora.oc4j
1 OFFLINE OFFLINE STABLE
ora.orcl.db
1 ONLINE ONLINE rac12cnode1 Open,STABLE
2 ONLINE ONLINE rac12cnode2 Open,STABLE
3 ONLINE ONLINE rac12cnode3 Open,STABLE
ora.rac12cnode1.vip
1 ONLINE ONLINE rac12cnode1 STABLE
ora.rac12cnode2.vip
1 ONLINE ONLINE rac12cnode2 STABLE
ora.rac12cnode3.vip
1 ONLINE ONLINE rac12cnode3 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac12cnode2 STABLE
--------------------------------------------------------------------------------
[grid@rac12cnode1 addnode]$ srvctl start listener -listener LISTENER_LEAF
[grid@rac12cnode1 addnode]$

On node 4(Leaf Node):

[grid@rac12cnode4 ~]$ /u01/app/12.1.0/grid/bin/crsctl get node role config -all
Node 'rac12cnode1' configured role is 'hub'
Node 'rac12cnode2' configured role is 'hub'
Node 'rac12cnode3' configured role is 'hub'
Node 'rac12cnode4' configured role is 'leaf'
[grid@rac12cnode4 ~]$ /u01/app/12.1.0/grid/bin/crsctl get node role status -all
Node 'rac12cnode1' active role is 'hub'
Node 'rac12cnode2' active role is 'hub'
Node 'rac12cnode3' active role is 'hub'
Node 'rac12cnode4' active role is 'leaf'
[grid@rac12cnode4 ~]$ ps ucx
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
grid 10841 0.0 0.0 98216 2180 ? S 03:51 0:01 sshd
grid 10842 0.0 0.0 108384 1696 pts/2 Ss 03:51 0:00 bash
grid 10864 0.0 0.0 106140 1200 pts/2 S+ 03:52 0:00 config.sh
grid 10866 0.0 0.1 127920 5208 pts/2 S+ 03:52 0:00 perl
grid 10869 0.6 4.3 1628076 134544 pts/2 Sl+ 03:52 0:16 java
grid 23614 0.0 1.0 712668 31480 ? Ssl 04:19 0:01 oraagent.bin
grid 23627 0.6 1.0 682972 32520 ? Ssl 04:19 0:06 evmd.bin
grid 23629 0.0 0.6 193248 19892 ? Ssl 04:19 0:00 mdnsd.bin
grid 23653 0.0 0.4 189816 14688 ? S 04:19 0:00 evmlogger.bin
grid 23657 0.1 0.9 539796 30556 ? Ssl 04:19 0:01 gpnpd.bin
grid 23663 4.0 1.5 617292 48008 ? Sl 04:19 0:42 gipcd.bin
grid 23708 0.5 4.2 683812 130852 ? SLl 04:20 0:05 ocssdrim.bin
grid 24167 0.0 0.0 108384 1804 pts/0 S 04:36 0:00 bash
grid 24197 0.0 0.0 110236 1120 pts/0 R+ 04:37 0:00 ps
[grid@rac12cnode4 ~]$

5) Now it is time to do final verification step : cluvfy

cluvfy stage -post nodeadd -n rac12cnode4 -verbose

[grid@rac12cnode1 ~]$ cluvfy stage -post nodeadd -n rac12cnode4 -verbose
Performing post-checks for node addition
Checking node reachability...
Check: Node reachability from node "rac12cnode1"
Destination Node Reachable?
------------------------------------ ------------------------
rac12cnode4 yes
Result: Node reachability check passed from node "rac12cnode1"

Checking user equivalence...
Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
rac12cnode4 passed
Result: User equivalence check passed for user "grid"
WARNING:
Unable to obtain VIP information from node "rac12cnode4".

Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
rac12cnode1 passed
rac12cnode2 passed
rac12cnode4 passed
rac12cnode3 passed
Verification of the hosts config file successful

Interface information for node "rac12cnode1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.131 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:22 1500
eth0 192.168.1.135 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:22 1500
eth1 10.10.10.131 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:09 1500
eth1 169.254.251.99 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:09 1500
eth2 10.11.0.135 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0B 1500

Interface information for node "rac12cnode2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.132 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.140 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.139 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.136 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth1 10.10.10.132 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0C 1500
eth1 169.254.169.69 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0C 1500
eth2 10.11.0.136 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:11 1500

Interface information for node "rac12cnode4"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.134 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:1E 1500
eth1 10.10.10.134 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:14 1500
eth2 10.11.0.138 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:15 1500

Interface information for node "rac12cnode3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.133 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:24 1500
eth0 192.168.1.137 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:24 1500
eth1 10.10.10.133 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:12 1500
eth1 169.254.177.41 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:12 1500
eth2 10.11.0.137 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:13 1500

Check: Node connectivity using interfaces on subnet "192.168.1.0"
Check: Node connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode2[192.168.1.136] rac12cnode3[192.168.1.137] yes
rac12cnode2[192.168.1.136] rac12cnode4[192.168.1.134] yes
rac12cnode2[192.168.1.136] rac12cnode1[192.168.1.135] yes
rac12cnode2[192.168.1.136] rac12cnode2[192.168.1.139] yes
rac12cnode2[192.168.1.136] rac12cnode1[192.168.1.131] yes
rac12cnode2[192.168.1.136] rac12cnode2[192.168.1.132] yes
rac12cnode2[192.168.1.136] rac12cnode2[192.168.1.140] yes
rac12cnode2[192.168.1.136] rac12cnode3[192.168.1.133] yes
rac12cnode3[192.168.1.137] rac12cnode4[192.168.1.134] yes
rac12cnode3[192.168.1.137] rac12cnode1[192.168.1.135] yes
rac12cnode3[192.168.1.137] rac12cnode2[192.168.1.139] yes
rac12cnode3[192.168.1.137] rac12cnode1[192.168.1.131] yes
rac12cnode3[192.168.1.137] rac12cnode2[192.168.1.132] yes
rac12cnode3[192.168.1.137] rac12cnode2[192.168.1.140] yes
rac12cnode3[192.168.1.137] rac12cnode3[192.168.1.133] yes
rac12cnode4[192.168.1.134] rac12cnode1[192.168.1.135] yes
rac12cnode4[192.168.1.134] rac12cnode2[192.168.1.139] yes
rac12cnode4[192.168.1.134] rac12cnode1[192.168.1.131] yes
rac12cnode4[192.168.1.134] rac12cnode2[192.168.1.132] yes
rac12cnode4[192.168.1.134] rac12cnode2[192.168.1.140] yes
rac12cnode4[192.168.1.134] rac12cnode3[192.168.1.133] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.139] yes
rac12cnode1[192.168.1.135] rac12cnode1[192.168.1.131] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.132] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.140] yes
rac12cnode1[192.168.1.135] rac12cnode3[192.168.1.133] yes
rac12cnode2[192.168.1.139] rac12cnode1[192.168.1.131] yes
rac12cnode2[192.168.1.139] rac12cnode2[192.168.1.132] yes
rac12cnode2[192.168.1.139] rac12cnode2[192.168.1.140] yes
rac12cnode2[192.168.1.139] rac12cnode3[192.168.1.133] yes
rac12cnode1[192.168.1.131] rac12cnode2[192.168.1.132] yes
rac12cnode1[192.168.1.131] rac12cnode2[192.168.1.140] yes
rac12cnode1[192.168.1.131] rac12cnode3[192.168.1.133] yes
rac12cnode2[192.168.1.132] rac12cnode2[192.168.1.140] yes
rac12cnode2[192.168.1.132] rac12cnode3[192.168.1.133] yes
rac12cnode2[192.168.1.140] rac12cnode3[192.168.1.133] yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rac12cnode2,rac12cnode3,rac12cnode4,rac12cnode1

Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode2:192.168.1.136 rac12cnode3:192.168.1.137 passed
rac12cnode2:192.168.1.136 rac12cnode4:192.168.1.134 passed
rac12cnode2:192.168.1.136 rac12cnode1:192.168.1.135 passed
rac12cnode2:192.168.1.136 rac12cnode2:192.168.1.139 passed
rac12cnode2:192.168.1.136 rac12cnode1:192.168.1.131 passed
rac12cnode2:192.168.1.136 rac12cnode2:192.168.1.132 passed
rac12cnode2:192.168.1.136 rac12cnode2:192.168.1.140 passed
rac12cnode2:192.168.1.136 rac12cnode3:192.168.1.133 passed
Result: TCP connectivity check passed for subnet "192.168.1.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.1.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Checking cluster integrity...
Node Name
------------------------------------
rac12cnode1
rac12cnode2
rac12cnode3
rac12cnode4
Cluster integrity check passed

Checking CRS integrity...
The Oracle Clusterware is healthy on node "rac12cnode1"
CRS integrity check passed
Clusterware version consistency passed.
Checking shared resources...
Checking CRS home location...
"/u01/app/12.1.0/grid" is not shared
Result: Shared resources check for node addition passed

Checking node connectivity...
Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
rac12cnode1 passed
rac12cnode2 passed
rac12cnode4 passed
rac12cnode3 passed
Verification of the hosts config file successful

Interface information for node "rac12cnode1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.131 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:22 1500
eth0 192.168.1.135 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:22 1500
eth1 10.10.10.131 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:09 1500
eth1 169.254.251.99 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:09 1500
eth2 10.11.0.135 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0B 1500

Interface information for node "rac12cnode2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.132 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.140 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.139 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth0 192.168.1.136 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:23 1500
eth1 10.10.10.132 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0C 1500
eth1 169.254.169.69 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:0C 1500
eth2 10.11.0.136 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:11 1500

Interface information for node "rac12cnode4"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.134 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:1E 1500
eth1 10.10.10.134 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:14 1500
eth2 10.11.0.138 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:15 1500

Interface information for node "rac12cnode3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.133 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:24 1500
eth0 192.168.1.137 192.168.1.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:24 1500
eth1 10.10.10.133 10.10.10.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:12 1500
eth1 169.254.177.41 169.254.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:12 1500
eth2 10.11.0.137 10.11.0.0 0.0.0.0 192.168.1.1 00:21:F6:00:00:13 1500

Check: Node connectivity using interfaces on subnet "10.10.10.0"
Check: Node connectivity of subnet "10.10.10.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode4[10.10.10.134] rac12cnode3[10.10.10.133] yes
rac12cnode4[10.10.10.134] rac12cnode2[10.10.10.132] yes
rac12cnode4[10.10.10.134] rac12cnode1[10.10.10.131] yes
rac12cnode3[10.10.10.133] rac12cnode2[10.10.10.132] yes
rac12cnode3[10.10.10.133] rac12cnode1[10.10.10.131] yes
rac12cnode2[10.10.10.132] rac12cnode1[10.10.10.131] yes
Result: Node connectivity passed for subnet "10.10.10.0" with node(s) rac12cnode4,rac12cnode3,rac12cnode2,rac12cnode1

Check: TCP connectivity of subnet "10.10.10.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode4:10.10.10.134 rac12cnode3:10.10.10.133 passed
rac12cnode4:10.10.10.134 rac12cnode2:10.10.10.132 passed
rac12cnode4:10.10.10.134 rac12cnode1:10.10.10.131 passed
Result: TCP connectivity check passed for subnet "10.10.10.0"

Check: Node connectivity using interfaces on subnet "192.168.1.0"
Check: Node connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode1[192.168.1.131] rac12cnode1[192.168.1.135] yes
rac12cnode1[192.168.1.131] rac12cnode3[192.168.1.137] yes
rac12cnode1[192.168.1.131] rac12cnode2[192.168.1.132] yes
rac12cnode1[192.168.1.131] rac12cnode3[192.168.1.133] yes
rac12cnode1[192.168.1.131] rac12cnode2[192.168.1.139] yes
rac12cnode1[192.168.1.131] rac12cnode2[192.168.1.136] yes
rac12cnode1[192.168.1.131] rac12cnode2[192.168.1.140] yes
rac12cnode1[192.168.1.131] rac12cnode4[192.168.1.134] yes
rac12cnode1[192.168.1.135] rac12cnode3[192.168.1.137] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.132] yes
rac12cnode1[192.168.1.135] rac12cnode3[192.168.1.133] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.139] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.136] yes
rac12cnode1[192.168.1.135] rac12cnode2[192.168.1.140] yes
rac12cnode1[192.168.1.135] rac12cnode4[192.168.1.134] yes
rac12cnode3[192.168.1.137] rac12cnode2[192.168.1.132] yes
rac12cnode3[192.168.1.137] rac12cnode3[192.168.1.133] yes
rac12cnode3[192.168.1.137] rac12cnode2[192.168.1.139] yes
rac12cnode3[192.168.1.137] rac12cnode2[192.168.1.136] yes
rac12cnode3[192.168.1.137] rac12cnode2[192.168.1.140] yes
rac12cnode3[192.168.1.137] rac12cnode4[192.168.1.134] yes
rac12cnode2[192.168.1.132] rac12cnode3[192.168.1.133] yes
rac12cnode2[192.168.1.132] rac12cnode2[192.168.1.139] yes
rac12cnode2[192.168.1.132] rac12cnode2[192.168.1.136] yes
rac12cnode2[192.168.1.132] rac12cnode2[192.168.1.140] yes
rac12cnode2[192.168.1.132] rac12cnode4[192.168.1.134] yes
rac12cnode3[192.168.1.133] rac12cnode2[192.168.1.139] yes
rac12cnode3[192.168.1.133] rac12cnode2[192.168.1.136] yes
rac12cnode3[192.168.1.133] rac12cnode2[192.168.1.140] yes
rac12cnode3[192.168.1.133] rac12cnode4[192.168.1.134] yes
rac12cnode2[192.168.1.139] rac12cnode2[192.168.1.136] yes
rac12cnode2[192.168.1.139] rac12cnode2[192.168.1.140] yes
rac12cnode2[192.168.1.139] rac12cnode4[192.168.1.134] yes
rac12cnode2[192.168.1.136] rac12cnode2[192.168.1.140] yes
rac12cnode2[192.168.1.136] rac12cnode4[192.168.1.134] yes
rac12cnode2[192.168.1.140] rac12cnode4[192.168.1.134] yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rac12cnode1,rac12cnode3,rac12cnode2,rac12cnode4

Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode1:192.168.1.131 rac12cnode1:192.168.1.135 passed
rac12cnode1:192.168.1.131 rac12cnode3:192.168.1.137 passed
rac12cnode1:192.168.1.131 rac12cnode2:192.168.1.132 passed
rac12cnode1:192.168.1.131 rac12cnode3:192.168.1.133 passed
rac12cnode1:192.168.1.131 rac12cnode2:192.168.1.139 passed
rac12cnode1:192.168.1.131 rac12cnode2:192.168.1.136 passed
rac12cnode1:192.168.1.131 rac12cnode2:192.168.1.140 passed
rac12cnode1:192.168.1.131 rac12cnode4:192.168.1.134 passed
Result: TCP connectivity check passed for subnet "192.168.1.0"

Check: Node connectivity using interfaces on subnet "10.11.0.0"
Check: Node connectivity of subnet "10.11.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode2[10.11.0.136] rac12cnode1[10.11.0.135] yes
rac12cnode2[10.11.0.136] rac12cnode3[10.11.0.137] yes
rac12cnode2[10.11.0.136] rac12cnode4[10.11.0.138] yes
rac12cnode1[10.11.0.135] rac12cnode3[10.11.0.137] yes
rac12cnode1[10.11.0.135] rac12cnode4[10.11.0.138] yes
rac12cnode3[10.11.0.137] rac12cnode4[10.11.0.138] yes
Result: Node connectivity passed for subnet "10.11.0.0" with node(s) rac12cnode2,rac12cnode1,rac12cnode3,rac12cnode4

Check: TCP connectivity of subnet "10.11.0.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac12cnode2:10.11.0.136 rac12cnode1:10.11.0.135 passed
rac12cnode2:10.11.0.136 rac12cnode3:10.11.0.137 passed
rac12cnode2:10.11.0.136 rac12cnode4:10.11.0.138 passed
Result: TCP connectivity check passed for subnet "10.11.0.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.1.0".
Subnet mask consistency check passed for subnet "10.10.10.0".
Subnet mask consistency check passed for subnet "10.11.0.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "10.10.10.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "10.10.10.0" for multicast communication with multicast group "224.0.0.251" passed.
Checking subnet "10.11.0.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "10.11.0.0" for multicast communication with multicast group "224.0.0.251" passed.
Check of multicast communication passed.
Checking node application existence...
Checking existence of VIP node application (required)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
rac12cnode1 yes yes passed
rac12cnode2 yes yes passed
rac12cnode3 yes yes passed
VIP node application check passed
Checking existence of NETWORK node application (required)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
rac12cnode1 yes yes passed
rac12cnode2 yes yes passed
rac12cnode3 yes yes passed
NETWORK node application check passed
Checking existence of ONS node application (optional)
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
rac12cnode1 no yes passed
rac12cnode2 no yes passed
rac12cnode3 no yes passed
ONS node application check passed

Checking to make sure user "grid" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
rac12cnode4 passed does not exist
Result: User "grid" is not part of "root" group. Check passed
Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed
Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
Node Name Status
------------------------------------ ------------------------
rac12cnode4 passed
Result: CTSS resource check passed

Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed
Check CTSS state started...
Check: CTSS state
Node Name State
------------------------------------ ------------------------
rac12cnode4 Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP

Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
rac12cnode4 yes
Result: Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
Checking whether NTP daemon or service is using UDP port 123 on all nodes
Check for NTP daemon or service using UDP port 123
Node Name Port Open?
------------------------------------ ------------------------
rac12cnode4 yes
NTP common Time Server Check started...
NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Checking on nodes "[rac12cnode4]"...
Check: Clock time offset from NTP Time Server
Time Server: .LOCL.
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
------------ ------------------------ ------------------------
rac12cnode4 0.0 passed
Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[rac12cnode4]".
Clock time offset check passed
Result: Clock synchronization check using Network Time Protocol(NTP) passed

Oracle Cluster Time Synchronization Services check passed
Post-check for node addition was successful.
[grid@rac12cnode1 ~]$

Hope It Helps

SRI

askMLabs

RAC 12c : JAN2014 PSU To 4 Node Flex Cluster

Written By askMLabs on Sunday, March 16, 2014 | 2:25 PM

RAC 12c Flex Cluster Installation

RAC : Clone Flex Cluster To Extend Cluster By Adding Hub and Lead Nodes

Written By askMLabs on Saturday, March 15, 2014 | 11:57 PM

RAC : Flex Cluster Useful Commands

Written By askMLabs on Wednesday, March 12, 2014 | 10:31 AM

RAC : Leaf Node Addition To Flex Cluster - 12101

Written By askMLabs on Tuesday, March 11, 2014 | 8:20 AM

Popular post

Labels

Blog Archive

Certifications

Total Pageviews

About askMLabs

askMLabs

RAC 12c : JAN2014 PSU To 4 Node Flex Cluster

Written By askMLabs on Sunday, March 16, 2014 | 2:25 PM

RAC 12c Flex Cluster Installation

RAC : Clone Flex Cluster To Extend Cluster By Adding Hub and Lead Nodes

Written By askMLabs on Saturday, March 15, 2014 | 11:57 PM

RAC : Flex Cluster Useful Commands

Written By askMLabs on Wednesday, March 12, 2014 | 10:31 AM

RAC : Leaf Node Addition To Flex Cluster - 12101

Written By askMLabs on Tuesday, March 11, 2014 | 8:20 AM

Popular post

Labels

Blog Archive

Subscribe To

Certifications

Total Pageviews

About askMLabs

Written By askMLabs on Sunday, March 16, 2014 | 2:25 PM

Written By askMLabs on Saturday, March 15, 2014 | 11:57 PM

Written By askMLabs on Wednesday, March 12, 2014 | 10:31 AM

Written By askMLabs on Tuesday, March 11, 2014 | 8:20 AM