Then I tried to troubleshoot the issue .....
This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.
STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.
STEP 2: Verify storage on both rac servers
Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.
STEP 3: Check high availability services and cluster ready services on both nodes with the following commands
Observation : The high availability services are up , but crs services are down.
STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.
Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.
STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.
Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can't find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.
Then i tried to see if i can see voting disks through command.
SETP 6 : Check the voting disks availability ( 11gR2 has voting disks in asm. I don't have asm instance up now. So i will get error if i try to query the voting disks on asm)
Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.
STEP 7 : Tried re-enabling the asmlibs as follows on both the nodes ...
Observation : Not able to start services even after restarting the asmlib.
STEP 8 : Tried to re-configure the asmlib as below .....
Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.
STEP 9 : Stop and Start cluster services on both the nodes.
Observation : Cluster ready services started without any issues. Now i need to verify the services status.
STEP 10 : Verify the status.
This concludes the article on troubleshooting RAC services startup issue.