Latest Post

WEBLOGIC SERVER 11G/12C ADMINISTRATION TRAINING

Written By Unknown on Wednesday, December 25, 2013 | 5:04 AM


This course trains system administrators and data center personnel on techniques for installing, configuring, deploying and securing distributed, clustered applications and systems deploy on Oracle WebLogic Server.

Current Training Schedule : 
Training Format
Online Training
Fee
USD xxx /  INR xx,xxx/-
Duration
2 weeks ( Sat & Sun ) - 4 Hr Session
Date
DD-MON-YYYY
Time
HH:MI to HH:MI
Instructor
Srikrishna Murthy Annam

Services : 
Training Format
Self Study Material with Lab practice examples
Fee
USD xxx /  INR x,xxx/-
Prepared By
Srikrishna Murthy Annam

Only Lab : 
Practice Lab
24/7 practice lab access from our data center
Fee
USD xxx /  INR x,xxx/-
Guided  By
Srikrishna Murthy Annam

Course Content : 
This course trains system administrators and data center personnel on techniques for installing, configuring, deploying and securing distributed, clustered applications and systems deploy on Oracle WebLogic Server.

This training focuses on techniques for building high availability, clustered configurations to ensure maximum application uptime. Method for hardening the implementation to provide reliable security are also emphasized.
  1. 1.Introducing Oracle Fusion Middleware Platform
    1. Describing the Oracle Fusion Middleware Environment
    2. Describing How WebLogic Server Supports Various Fusion Middleware Suites
    3. Describing How Various Fusion Middleware Suites Augment the Functions of WebLogic Server
  2. 2.Defining Java Enterprise Edition Terminology and Architecture
    1. Explaining the Motivation Behind Distributed Systems
    2. The Major Components of the Java Platform Enterprise Edition 5 (Java EE) Specification
  3. 3.Installing Oracle WebLogic Server
    1. Explaining Oracle WebLogic Server Installation Steps
    2. Installing WebLogic Server Using Both the Graphical User Interface (GUI) and Command Line
    3. Describing the Organization and Contents of the WebLogic Server Directory Structure
    4. Navigating the WebLogic Server Online and Offline Documentation
  4. 4.Configuring a Simple Domain
    1. Describing the Common Elements in a WebLogic Server Domain
    2. Describing how Domains are Used in the Enterprise
    3. Comparing Administration and Managed Servers
    4. Configuring a Domain
    5. Describing the Organization and Contents of the WLS Directory Structure
    6. Describing the use of WLST Offline to Manage Domains
    7. Creating a Simple Domain with One Managed Server
    8. Checking the Port Numbers that are Used for Components
  5. 5.Configuring a Domain Using Templates
    1. Describing the Internal Process Used to Create or Update a Domain Using a Template
    2. Describing the Process of Extending a Domain Template
    3. Explaining the Domain Templates Provided for Setting up JDeveloper, SOA, and WebCenter
  6. 6.Using Administration Console and WLST
    1. Identifying Dynamic and Nondynamic Attribute Changes in the Administration Console
    2. Customizing Monitoring Tables Within the Administration Console
    3. Using the WebLogic Scripting Tool (WLST) Command-line Utility
    4. Invoking WLST and Navigate Through the Domain Structure
    5. Describing How Change Management is Performed Internally Using Java Management Extension (JMX) and Mbeans
  7. 7.Configuring Managed Servers
    1. Configuring Managed Servers Using the Administrative Console
    2. Configuring Managed Servers Using WebLogic Scripting Tool (WLST)
    3. Starting Managed Servers
    4. Shutting Down a Server or an Entire Domain Using WLST or the Administrative Console
    5. Configuring Managed Servers on a Computer Separate from the Sdministration Server
    6. Explaining Administration and Managed Server Independence (MSI)
  8. 8.Configuring Node Managers
    1. Defining the Oracle WebLogic Server Machine
    2. Configuring a Machine and Assign Servers to it by Using the Console and WebLogic Scripting Tool (WLST)
    3. Explaining the Node Manager Architecture
    4. Describing the Organization and Contents of a Node Manager Directory Structure
    5. Configuring, Starting, and Stopping Node Managers
    6. Describing How to Start and Stop Procedures
  9. 9.Viewing and Managing Logs in Oracle WLS Environment
    1. Defining and Configuring Server and Domain Logs
    2. Viewing and Interpreting the Format of Domain and Server Log Files Using the Administration Console
    3. Configuring Server Standard Output Settings Using the Console
    4. Describing How Applications Can Integrate With WLS Logging Infrastructure (Apache commons, log4j)
    5. Accessing Online Log Message Catalogs
    6. Creating and Applying a Log Filter Using the Console
    7. Configuring Log Filter Expressions
  10. 10.Deployment Concepts
    1. Contrasting Autodeploy With Manual Deployment
    2. Configuring and Deploying Web Applications Via the Administration Console, Command Line, and WLST
    3. Configuring Deployment Descriptors
    4. Testing Deployed Applications
    5. Describing the Role of Web Servers
    6. Tracing a Typical Web Interaction Flow
    7. Contrasting Static and Dynamic Content and Deployment
    8. Front-end Deployed Applications With a Web Server
  11. 11.Deploying Java EE Applications
    1. Describing Java EE Web Applications
    2. Describing Enterprise Deployment Architectures
    3. Packaging Web Applications in Several Forms
    4. Defining Web Application Structure and Web Application Archive
    5. Explaining Why WebLogic Augments Standard Java EE Deployment Descriptors With weblogic*.xml Files
    6. Looking at Deployment Descriptors web.xml and weblogic.xml
    7. Describing URLs and Web Applications
  12. 12.Advanced Deployment
    1. Configuring an Application for Multiple Development Environments
    2. Creating a Deployment Plan
    3. Staging a Deployment Plan
    4. Using Production Redeployment
  13. 13.Understanding JDBC and Configuring Data Sources
    1. Configuring JDBC and JDBC Data Sources
    2. Configuring Data Source Scope
    3. Contrasting Two-tier and Multi-tier JDBC Architecture
    4. Configuring a Connection Pool
    5. Describing How Data Sources are Used
    6. Deploying JDBC Resources to a Target
    7. Explaining the Components of JDBC URLs
    8. Monitoring and Testing a Data Source
  14. 14.Setting Up Java Message Service (JMS) Resources
    1. Describing JMS
    2. Describing How Oracle WebLogic Server JMS is Implemented
    3. Configuring JMS Server
    4. Configuring Connection Factories
    5. Configuring Queues and Topics
    6. Configuring Persistent Messages
    7. Deploying an Application that Uses JMS
    8. Monitoring JMS Resources and Messages
  15. 15.Introduction to Clustering
    1. The Benefits of Oracle WebLogic Cluster
    2. Basic Cluster Architecture
    3. Multitier Cluster Architecture
    4. Communication Among Clustered Server Instances
    5. The Key Criteria for Selecting Suitable Cluster Architecture
  16. 16.Configuring a Cluster
    1. Preparing Your Environment for a Cluster
    2. Creating and Configuring a Cluster
    3. Adding Servers to a Cluster
    4. Starting Up and Shutting Down Clustered Servers
  17. 17.Managing Clusters
    1. Deploying Applications to a Cluster
    2. Describing the Replication of a Session State in a Cluster
    3. Configuring Replication Groups
    4. Configuring In-memory Replication
    5. Configuring Java Database Connectivity (JDBC) Replication
    6. Configuring File Replication
    7. Configuring a Multitier Cluster for Enterprise JavaBeans (EJB) Applications
  18. 18.Security Concepts and Configuration
    1. Using the WebLogic Server (WLS) Security Architecture
    2. Configuring Security Realms
    3. Configuring Users and Groups
    4. Configuring Roles
    5. Configuring Policies
    6. Configuring Protection for Web Application Resources and EJBs
  19. 19.Protecting Against Attacks
    1. Describing the Process of Configuring Secure Sockets Layer (SSL)
    2. Using the Keytool Utility to Configure Keys and Obtain Digital Certificates
    3. Configuring SSL for the WLS Server
    4. Configuring Countermeasures for Some Web-based Attacks
  20. 20.Backup and Recovery Operations
    1. Recommending a Backup and Recovery Strategy
    2. Performing a Full Offline Backup and Recovery
    3. Performing an Online and Offline Domain Backup
    4. Performing an Offline Domain Recovery
    5. Performing an Instance Home Backup and Recovery

Who Can Attend : 
  1. J2EE Developer
  2. Web Administrator
  3. Java EE Developer
  4. Sales Consultants

Course Objective :
  1. Monitor application server using GUI and command-line tools such as automation scripts
  2. Describe the architecture of WebLogic Server including domains, servers and machines
  3. Install, configure and use WebLogic Server
  4. Perform routine Oracle WebLogic Server administration functions
  5. Set up a cluster of servers and distribute applications and resources to the cluster
  6. Configure Oracle HTTP Server as the Web-tier front end for Oracle WebLogic Server instances and clusters
  7. Deploy different types of Java EE applications to Oracle WebLogic Server
  8. Deploy and manage large-scale Java EE applications to servers or clusters through the entire development and production lifecycle
  9. Configure basic resource and application security
  10. Backup and recover from various failures

Why To Choose Us:
  1. Handouts for each topics
  2. Lab facility from anywhere
  3. Guidance from expert trainers
  4. Help to setup database in local machines

Fee Details : 
  1. See Above

Data Guard Administration - Weekend Online Training

Written By Unknown on Wednesday, November 27, 2013 | 1:56 PM

Please visit our site http://askMLabs.com for more details

Direct Link For Free Demo Registration :  Free Demo

Oracle Database 11gR2 Data Guard Administration
Free Demo on 01-DEC-2013 From 10am to 11am.
Online Classes , 4 Weeks , Batch starts from 07-DEC-2013
Classes Every Week SAT and SUN From 5pm to 9pm IST
For Free Registrations Visit : http://askMLabs.com, info@askmlabs.com
24/7 Lab Access From Anywhere In The World



Data Guard Demo

RAC Administration 11gR2 - Weekend Online Training

Written By Unknown on Wednesday, November 6, 2013 | 10:15 AM

Please visit our site http://askMLabs.com for more details

Direct Link For Free Demo Registration :  Free Demo

Oracle Database 11gR2 Grid Infra with ASM and RAC Administration
Free Demo on 09-NOV-2013 From 11am to 12pm.
Online Classes , 4 Weeks , Batch starts from 16-NOV-2013
Classes Every Week SAT and SUN From 7am to 11am
For Free Registrations Visit : http://askMLabs.com, info@askmlabs.com
Lab Access From Anywhere In The World




OWEMRAC11003

Online Trainings On Oracle Database 11g

Written By Unknown on Friday, September 13, 2013 | 12:57 PM

We are starting new trainings on oracle database 11g. Please review our site http://askmlabs.com or attend our free demo to know more details.
1) Oracle Database 11g Administration ( with 12c new features )
2) Oracle Database 11gR2 Grid with ASM and RAC Administration
3) Oracle Database 11gR2 DataGuard Administration

All these trainings including lab sessions are handled by Srikrishna Murthy Annam.

askmlabs-online-trainings1

Thanks

SRI

Change Online Redolog Group Member Size

Written By Unknown on Wednesday, September 11, 2013 | 8:31 AM

In this article, i am going to show you how to change the online redolog file size. The existing configuration is having 3 redolog groups with one member in each group. Each member in group is 50MB in size. Now we are going to change the size of these online redolog group members from 50MB to 300MB.

SQL> select group#,thread#,MEMBERS,bytes/1024/1024 from v$log;
    GROUP#    THREAD#    MEMBERS BYTES/1024/1024
---------- ---------- ---------- ---------------
         1          1          1              50
         2          1          1              50
         3          1          1              50
SQL> select member from v$logfile;
MEMBER
--------------------------------------------------------------------------------
/home/apps/oem12c/oradata/oem12c/redo03.log
/home/apps/oem12c/oradata/oem12c/redo02.log
/home/apps/oem12c/oradata/oem12c/redo01.log
SQL> alter database add logfile thread 1 group 4 ('/home/apps/oem12c/oradata/oem12c/redo04.log') size 300M;
Database altered.
SQL> alter database add logfile thread 1 group 5 ('/home/apps/oem12c/oradata/oem12c/redo05.log') size 300M;
Database altered.
SQL> alter database add logfile thread 1 group 6 ('/home/apps/oem12c/oradata/oem12c/redo06.log') size 300M;
Database altered.
SQL> select group#,thread#,MEMBERS,status,bytes/1024/1024 from v$log;
    GROUP#    THREAD#    MEMBERS STATUS           BYTES/1024/1024
---------- ---------- ---------- ---------------- ---------------
         1          1          1 INACTIVE                      50
         2          1          1 CURRENT                       50
         3          1          1 INACTIVE                      50
         4          1          1 UNUSED                       300
         5          1          1 UNUSED                       300
         6          1          1 UNUSED                       300
6 rows selected.
SQL> alter database drop logfile group 1;
Database altered.
SQL> alter database drop logfile group 3;
Database altered.
SQL> alter system switch logfile;
System altered.
SQL> alter system checkpoint;
System altered.
SQL> select group#,thread#,MEMBERS,status,bytes/1024/1024 from v$log;
    GROUP#    THREAD#    MEMBERS STATUS           BYTES/1024/1024
---------- ---------- ---------- ---------------- ---------------
         2          1          1 INACTIVE                      50
         4          1          1 CURRENT                      300
         5          1          1 UNUSED                       300
         6          1          1 UNUSED                       300
SQL> alter database drop logfile group 2;
Database altered.
SQL> select group#,thread#,MEMBERS,status,bytes/1024/1024 from v$log;
    GROUP#    THREAD#    MEMBERS STATUS           BYTES/1024/1024
---------- ---------- ---------- ---------------- ---------------
         4          1          1 CURRENT                      300
         5          1          1 UNUSED                       300
         6          1          1 UNUSED                       300
SQL> 

Hope this helps
SRI

Oracle Database 12c Architecture Free Demo

Written By Unknown on Friday, September 6, 2013 | 4:27 PM

demo1

Registration :  http://askmlabs.com


Our new Website askmlabs.com

Written By Unknown on Tuesday, September 3, 2013 | 12:51 PM

We have launched a new website for oracle trainings and cloud services. http://askmlabs.com/

  • We offer corporate trainings on oracle technologies in various formats. The biggest challenge in today's IT enabled industry is the lack of good technical skills and we fill the gap by providing technical skills to resources to meet the industry standard.

  • We offer online trainings and virtual labs facility. You can attend the trainings at your convinience and practice the labs from your home. We provide virtual lab from our cloud setup.

  • We offer the ready to use virtual machines as products. You can spend your time in practicing the technology rather than setting up the environment. All these virtual machines are provided at a very low cost. Please visit our site http://askmlabs.com/ for more information or contact us at info@askmlabs.com

  • We offer PaaS and SaaS from our cloud setup. We will provide the IT infrastructure needs as per your requirements at a very low cost with high availability. Please visit our site http://askmlabs.com/ for more information or contact us at info@askmlabs.com

askMLabs


We Offer:
1) Training Solutions
2) Ready To Use Virtual Machines as Products
3) Online Virtual labs
4) PaaS and SaaS from our cloud setup
5) Knowledge base as blog
6) Forum for technical discussions
7) Login facility with your existing social account
8) Full length technical videos


Thanks

SRI

ORACLE RAC 11GR2 ADMINISTRATION TRAINING

Written By Unknown on Sunday, September 1, 2013 | 4:54 AM







This course gives you an introduction to the features and capabilities of oracle real application clusters RAC 11gR2.

Current Training Schedule : 
Training Format
Online Training
Fee
USD 300 /  INR 15,000/-
Duration
4 weeks ( Sat & Sun ) - 4 Hr Session
Date
DD-MON-YYYY
Time
HH:MI to HH:MI
Instructor
Srikrishna Murthy Annam

Services :
Training Format
Self Study Material with Lab practice examples
Fee
USD xxx /  INR x,xxx/-
Prepared By
Srikrishna Murthy Annam

Only Lab :
Practice Lab
24/7 practice lab access from our data center
Fee
USD xxx /  INR x,xxx/-
Guided  By
Srikrishna Murthy Annam

Course Content : 
This course gives you an introduction to the features and capabilities of oracle real application clusters RAC 11gR2. This course will provide you step by step guidance to create RAC environment
  1. Oracle Clusterware Installation
    1. Oracle RAC 11g Installation
    2. Cluster Setup Tasks
    3. Using Raw Partitions
    4. Cluster Configuration
  2. RAC Installation
    1. Installing Grid Infra and Automatic Storage Management
    2. Configure ASM Storage
    3. Install the Database Software
  3. Database Creation
    1. Storage Options for Database Files
    2. Create the Database
    3. Single Instance to RAC Conversion
  4. Oracle Clusterware Administration
    1. Controlling the Oracle Clusterware Stack
    2. Back Up and Recover Your Voting Disks
    3. Back Up and Recover Oracle Cluster Registry
    4. Prevent Automatic Instance Restarts
    5. Manage network settings
  5. RAC DB Administration
    1. Cluster Database Instance Administration
    2. Topology Viewer
    3. Automatic Undo Management and RAC
    4. Quiescing RAC Databases
    5. Transparent Data Encryption and Wallets in RAC
    6. ASM Instance and Crash Recovery in RAC
  6. Backup and Recover a RAC DB
    1. Protecting Against Media Failure
    2. Oracle Recovery Manager
  7. Administering ASM Instances
    1. Explain and apply ASM init.ora parameters
    2. Manage ASM Instances and associated processes
    3. Monitor ASM using v$asm dynamic performance views
  8. Administering ASM Disk groups
    1. Create and delete Automatic Storage Management (ASM) Disk groups
    2. Set the attributes of an existing ASM disk group
    3. Perform ongoing maintenance tasks on ASM disk groups
    4. Explain key performance and scalability considerations for ASM disk groups
  9. Administering ASM Files,Directories and Templates
    1. Use different client tools to access Automatic Storage Management (ASM) files
    2. Describe the format of a fully qualified ASM file name
    3. Explain how ASM files, directories, and aliases are created and managed
    4. Describe and manage disk group templates.
  10. RAC Patching
    1. Types of Patching
    2. Out-Of-Place Clusterware Upgrade
    3. Patch Properties
    4. Rolling Patches
    5. Opatch
    6. Installing Patches with Minimum Downtime
  11. Services
    1. Workload Dispatching
    2. High Availability of Services in RAC
    3. Services and Resource Manager with EM
    4. Change Service Thresholds
    5. Using Distributed Transactions with RAC
  12. High Availability of Connections
    1. Fast Application Notification
    2. Load Balancing Advisory
    3. Transparent Application Failover
  13. Troubleshooting the Oracle Clusterware components
    1. Locate Oracle Clusterware Log files
    2. Gather all log files using diagcollection.pl
    3. Enable resource debugging
    4. Enable component-level debugging
    5. Enable tracing for java-based tools
    6. Troubleshoot the Oracle Cluster Registry (OCR) file
  14. Node Addition/Removal
    1. Add and Delete Nodes and Instances
    2. Clone Oracle Clusterware
  15. Design for High Availability
    1. Maximum Availability Architecture
    2. RAC and Data Guard Topologies
    3. Extended RAC Connectivity
    4. Extended RAC disk mirroring
    5. Patch your RAC system in a rolling fashion
  16. Oracle RAC One Node
    1. Online database migration
    2. Add a RAC One Node database to an existing cluster
    3. Convert a RAC One Node Database to a RAC database
    4. Convert a single instance database to a RAC One Node database


Who Can Attend : 
  1. Data Warehouse Administrator
  2. Database Administrators
  3. Database Designers
  4. Project Manager
  5. Support Engineer
  6. System Analysts
  7. Technical Administrator
  8. Technical Consultant

Course Objective :
  1. Install, create, administer, and monitor a Real Application Clusters database
  2. Use configuration and management tools for Real Application Clusters databases
  3. Setup services for workloads management, and applications high availability
  4. Develop a backup and recovery strategy for Real Application Clusters databases
  5. Configure and monitor Oracle Clusterware resources
  6. Review high availability best practices
  7. Identify Real Application Clusters components

Why To Choose Us :
  1. Free Plug & Play virtual Machine 11g RAC
  2. Handouts for each topics
  3. Lab facility from anywhere* ( From our cloud servers )
  4. Guidance from expert trainers
  5. Help to setup database in local machines
  6. 100% Student Satisfaction
  7. Direct Interaction with Faculty at office
  8. Guidance for Interview Preparation
  9. Guidance for certification

Fee Details :
  1. Single payment 10% discount
  2. Two installments ( 50% before starting and 50% after 2 weeks )  -- 5% discount on second payment
  3. Three installments (20% before starting + 50% before 3rd week + 30% before 4th week)
  4. Refer a friend and get 20% discount on course fee
  5. Good negotiable price for group of people as a batch

How To Restore Database From Different Incarnation

Written By Unknown on Thursday, April 4, 2013 | 11:10 AM

What is incarnation?
When we open database with RESETLOGS option, the database starts a new incarnation. This new incarnation resets the log sequence number to 1, and then gives the online redo logs a new time stamp and SCN.

When the database starts a new incarnation, are we able to use the old backups taken in previous incarnation for restore and recovery purpose as the present log sequence number is reset to 1.

The present article discusses about the details of how to use the backups in old incarnation for restore and recovery purpose after the database has been opened with resetlogs for many times.

I will explain the details considering a practical approach in our training environment. First i will explain the existing environment and available backups.
Then i will take you to details of how these backups are used to restore and recover to the same server (or to different server) considering that the database has been opened many times with resetlogs option.

Environment Details :
Let us assume that we have a training environment where we have some gold backups for the database. Whenever we do any modification to the database, we will take a new goldbackup and we will keep restoring this latest new backup every week twice( Thursday night and Saturday night).

Following Details shows the different goldbackups available for the training environment.
RMAN> list backup summary;List of Backups
===============
Key     TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
327559  B  0  A SBT_TAPE    04-AUG-12       1       1       YES        NOEXPIRE
327560  B  0  A SBT_TAPE    04-AUG-12       1       1       YES        NOEXPIRE
327561  B  0  A SBT_TAPE    04-AUG-12       1       1       YES        NOEXPIRE
327562  B  0  A SBT_TAPE    04-AUG-12       1       1       YES        NOEXPIRE
327785  B  F  A SBT_TAPE    04-AUG-12       1       1       NO         NOEXPIRE
351357  B  0  A SBT_TAPE    13-OCT-12       1       1       YES        NOEXPIRE
351358  B  0  A SBT_TAPE    13-OCT-12       1       1       YES        NOEXPIRE
351359  B  0  A SBT_TAPE    13-OCT-12       1       1       YES        NOEXPIRE
351360  B  0  A SBT_TAPE    13-OCT-12       1       1       YES        NOEXPIRE
351583  B  F  A SBT_TAPE    13-OCT-12       1       1       NO         NOEXPIRE
355951  B  0  A SBT_TAPE    27-OCT-12       1       1       YES        NOEXPIRE
355952  B  0  A SBT_TAPE    27-OCT-12       1       1       YES        NOEXPIRE
355953  B  0  A SBT_TAPE    27-OCT-12       1       1       YES        NOEXPIRE
355954  B  0  A SBT_TAPE    27-OCT-12       1       1       YES        NOEXPIRE
356177  B  F  A SBT_TAPE    27-OCT-12       1       1       NO         NOEXPIRE
386046  B  0  A SBT_TAPE    26-JAN-13       1       1       YES        NOEXPIRE
386047  B  0  A SBT_TAPE    26-JAN-13       1       1       YES        NOEXPIRE
386048  B  0  A SBT_TAPE    26-JAN-13       1       1       YES        NOEXPIRE
386049  B  0  A SBT_TAPE    26-JAN-13       1       1       YES        NOEXPIRE
386275  B  F  A SBT_TAPE    26-JAN-13       1       1       NO         NOEXPIRE
387232  B  0  A SBT_TAPE    29-JAN-13       1       1       YES        NOEXPIRE
387233  B  0  A SBT_TAPE    29-JAN-13       1       1       YES        NOEXPIRE
387234  B  0  A SBT_TAPE    29-JAN-13       1       1       YES        NOEXPIRE
387235  B  0  A SBT_TAPE    29-JAN-13       1       1       YES        NOEXPIRE
387462  B  F  A SBT_TAPE    29-JAN-13       1       1       NO         NOEXPIRE
392057  B  0  A SBT_TAPE    09-FEB-13       1       1       YES        NOEXPIRE
392058  B  0  A SBT_TAPE    09-FEB-13       1       1       YES        NOEXPIRE
392059  B  0  A SBT_TAPE    09-FEB-13       1       1       YES        NOEXPIRE
392060  B  0  A SBT_TAPE    09-FEB-13       1       1       YES        NOEXPIRE
392286  B  F  A SBT_TAPE    09-FEB-13       1       1       NO         NOEXPIRE
392347  B  F  A SBT_TAPE    09-FEB-13       1       1       NO         TAG20130209T021024

RMAN>


So everytime when we restore the database from the latest gold backup, we will open the database with resetlogs option and it starts a new incarnation.
These details are shown below ..
RMAN> list incarnation of database TRNGDB;
List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
......
.......
158938  325435  TRNGDB   3679252776       ORPHAN  88080533256 29-JUL-12
158938  326880  TRNGDB   3679252776       PARENT  88080533256 02-AUG-12
158938  323440  TRNGDB   3679252776       ORPHAN  88080533256 26-JUL-12
158938  330059  TRNGDB   3679252776       ORPHAN  88080774587 09-AUG-12
158938  332058  TRNGDB   3679252776       ORPHAN  88080774587 12-AUG-12
158938  335023  TRNGDB   3679252776       ORPHAN  88080774587 23-AUG-12
158938  334177  TRNGDB   3679252776       ORPHAN  88080774587 19-AUG-12
158938  350823  TRNGDB   3679252776       PARENT  88080774587 11-OCT-12
158938  350133  TRNGDB   3679252776       ORPHAN  88080774587 07-OCT-12
158938  337328  TRNGDB   3679252776       ORPHAN  88080774587 26-AUG-12
158938  338003  TRNGDB   3679252776       ORPHAN  88080774587 30-AUG-12
158938  339685  TRNGDB   3679252776       ORPHAN  88080774587 02-SEP-12
158938  340363  TRNGDB   3679252776       ORPHAN  88080774587 06-SEP-12
158938  342063  TRNGDB   3679252776       ORPHAN  88080774587 09-SEP-12
158938  342743  TRNGDB   3679252776       ORPHAN  88080774587 13-SEP-12
158938  344429  TRNGDB   3679252776       ORPHAN  88080774587 16-SEP-12
158938  346103  TRNGDB   3679252776       ORPHAN  88080774587 23-SEP-12
158938  347243  TRNGDB   3679252776       ORPHAN  88080774587 27-SEP-12
158938  348499  TRNGDB   3679252776       ORPHAN  88080774587 30-SEP-12
158938  349183  TRNGDB   3679252776       ORPHAN  88080774587 04-OCT-12
158938  352901  TRNGDB   3679252776       ORPHAN  88081045250 18-OCT-12
158938  354443  TRNGDB   3679252776       ORPHAN  88081045250 21-OCT-12
158938  355286  TRNGDB   3679252776       PARENT  88081045250 25-OCT-12
158938  364387  TRNGDB   3679252776       ORPHAN  88081234176 22-NOV-12
158938  363526  TRNGDB   3679252776       ORPHAN  88081234176 18-NOV-12
158938  362107  TRNGDB   3679252776       ORPHAN  88081234176 15-NOV-12
158938  361247  TRNGDB   3679252776       ORPHAN  88081234176 11-NOV-12
158938  359827  TRNGDB   3679252776       ORPHAN  88081234176 08-NOV-12
158938  358965  TRNGDB   3679252776       ORPHAN  88081234176 04-NOV-12
158938  357545  TRNGDB   3679252776       ORPHAN  88081234176 01-NOV-12
158938  365807  TRNGDB   3679252776       ORPHAN  88081234176 25-NOV-12
158938  384807  TRNGDB   3679252776       PARENT  88081234176 24-JAN-13
158938  383953  TRNGDB   3679252776       ORPHAN  88081234176 20-JAN-13
158938  382547  TRNGDB   3679252776       ORPHAN  88081234176 17-JAN-13
158938  381687  TRNGDB   3679252776       ORPHAN  88081234176 13-JAN-13
158938  380267  TRNGDB   3679252776       ORPHAN  88081234176 10-JAN-13
158938  379411  TRNGDB   3679252776       ORPHAN  88081234176 06-JAN-13
158938  378007  TRNGDB   3679252776       ORPHAN  88081234176 03-JAN-13
158938  377152  TRNGDB   3679252776       ORPHAN  88081234176 30-DEC-12
158938  375747  TRNGDB   3679252776       ORPHAN  88081234176 27-DEC-12
158938  374892  TRNGDB   3679252776       ORPHAN  88081234176 23-DEC-12
158938  373487  TRNGDB   3679252776       ORPHAN  88081234176 20-DEC-12
158938  372631  TRNGDB   3679252776       ORPHAN  88081234176 16-DEC-12
158938  371227  TRNGDB   3679252776       ORPHAN  88081234176 13-DEC-12
158938  366667  TRNGDB   3679252776       ORPHAN  88081234176 29-NOV-12
158938  368086  TRNGDB   3679252776       ORPHAN  88081234176 02-DEC-12
158938  368947  TRNGDB   3679252776       ORPHAN  88081234176 06-DEC-12
158938  370366  TRNGDB   3679252776       ORPHAN  88081234176 09-DEC-12
158938  391514  TRNGDB   3679252776       PARENT  88082816786 07-FEB-13
158938  387719  TRNGDB   3679252776       ORPHAN  88082816786 01-FEB-13
158938  390618  TRNGDB   3679252776       ORPHAN  88082816786 03-FEB-13
158938  404336  TRNGDB   3679252776       ORPHAN  88082894636 21-FEB-13
158938  403425  TRNGDB   3679252776       ORPHAN  88082894636 17-FEB-13
158938  401240  TRNGDB   3679252776       ORPHAN  88082894636 14-FEB-13
158938  416836  TRNGDB   3679252776       CURRENT 88082894636 21-MAR-13
158938  412180  TRNGDB   3679252776       ORPHAN  88082894636 10-MAR-13
158938  409274  TRNGDB   3679252776       ORPHAN  88082894636 03-MAR-13
158938  415096  TRNGDB   3679252776       ORPHAN  88082894636 17-MAR-13
158938  413356  TRNGDB   3679252776       ORPHAN  88082894636 14-MAR-13
158938  406349  TRNGDB   3679252776       ORPHAN  88082894636 24-FEB-13
158938  407256  TRNGDB   3679252776       ORPHAN  88082894636 28-FEB-13
158938  410176  TRNGDB   3679252776       ORPHAN  88082894636 07-MAR-13

RMAN>


As long as i am restoring the latest goldbackups to the training environment, it uses the current incarnation and able to restore the latest backups and able to open the database with resetlogs option.

But if i want to restore any previous backups(backups taken at a different incarnation) to the training environment,i may not be able to restore the backups to the current incarnation.

Let us assume that i have my latest backups available on 09-OCT-2013 as per the above details and my current incarnation is 416836 (on 21-MAR-2013).
Now i have a requirement where i need to restore the backups available on 27-OCT-2012 to the same server. So as long as my current incarnation is 416836, i will not be able to use these backups on 27-OCT-2012 to restore the database.
So i need to first change the incarnation to be able to restore the backups on 27-OCT-2012.

For your easy understanding to identify to which incarnation to change, i have mixed backup details and incarnation details and orderded the details date wise below ....
158938  349183  TRNGDB   3679252776       ORPHAN  88080774587 04-OCT-12
13-OCT-12  <<<== Date on which we have Backups
158938  350133  TRNGDB   3679252776       ORPHAN  88080774587 07-OCT-12
158938  350823  TRNGDB   3679252776       PARENT  88080774587 11-OCT-12
158938  352901  TRNGDB   3679252776       ORPHAN  88081045250 18-OCT-12
158938  354443  TRNGDB   3679252776       ORPHAN  88081045250 21-OCT-12
158938  355286  TRNGDB   3679252776       PARENT  88081045250 25-OCT-12
27-OCT-12 <<<== Date on which we have  Backups
158938  357545  TRNGDB   3679252776       ORPHAN  88081234176 01-NOV-12
158938  358965  TRNGDB   3679252776       ORPHAN  88081234176 04-NOV-12
158938  359827  TRNGDB   3679252776       ORPHAN  88081234176 08-NOV-12
158938  361247  TRNGDB   3679252776       ORPHAN  88081234176 11-NOV-12
158938  362107  TRNGDB   3679252776       ORPHAN  88081234176 15-NOV-12
158938  363526  TRNGDB   3679252776       ORPHAN  88081234176 18-NOV-12
158938  364387  TRNGDB   3679252776       ORPHAN  88081234176 22-NOV-12
158938  365807  TRNGDB   3679252776       ORPHAN  88081234176 25-NOV-12
158938  366667  TRNGDB   3679252776       ORPHAN  88081234176 29-NOV-12

When a database goes through multiple incarnations, some backups can become orphaned. The simple way to find non-orphaned backups and orphaned backups are to form a table to list the incarnation number and orphaned and non-orphaned backups to each incarnation.
In our case, in simple words, we need to restore backups from 27-OCT-12. So identify the incarnation number for which this backup is non-orphaned.

I am considering the incarnation before and after the 27-OCT-12 to identify orphaned and non-orphaned backups. We have incarnations 357545(on 01-NOV-12)  and 355286 (25-OCT-12).
For incarnation 357545(on 01-NOV-12) : All the backups taken after the date 01-NOV-2012 are non-orphaned and all the backups before this incarnation are orphaned backups assuming that there is no other incarnation after the incarnation 357545. So in our case the backups on 27-OCT-2012 are orphaned backups if i set incarnation 357545.


For incarnation 355286(on 25-OCT-12) : All the backups taken after the date 25-OCT-2012 are non-orphaned and all the backups before this incarnation are orphaned backups assuming that there is no other incarnation after the incarnation 355286. So in our case the backups on 27-OCT-2012 are non-orphaned backups if i set incarnation 355286.

Now we have the details, which incarnation we need to set for restoring the 27-OCT-2012 backups.

Also get the controlfile information from the 27-OCT-2012 backups using
RMAN> list backup of controlfile;

NOTE : In our case the controlfile details are '4vnoosbv_1_1'

Then use the following syntax to restore and recover the database...
RMAN> set dbid=3679252776
executing command: SET DBID
database name is "TRNGDB" and DBID is 3679252776

RMAN> RESET DATABASE TO INCARNATION 355286;

database reset to incarnation 355286

RMAN>  run {
2> allocate channel t1 type 'SBT_TAPE' parms="ENV=(TDPO_OPTFILE=/tmp/askm/TRNGDB_tdpo.opt)";
3> restore controlfile from '4vnoosbv_1_1';
4> }

allocated channel: t1
channel t1: SID=396 device type=SBT_TAPE
channel t1: Data Protection for Oracle: version 5.5.2.0

Starting restore at 22-MAR-13

channel t1: restoring control file
channel t1: restore complete, elapsed time: 00:00:01
output file name=/u01/oradata/TRNGDB/control01.ctl
output file name=/u01/oradata/TRNGDB/control02.ctl
output file name=/u01/oradata/TRNGDB/control03.ctl
Finished restore at 22-MAR-13
released channel: t1

RMAN> run {
2> allocate channel t1 type 'SBT_TAPE' parms="ENV=(TDPO_OPTFILE=/tmp/askm/TRNGDB_tdpo.opt)";
3> restore controlfile from '4vnoosbv_1_1';
4> alter database mount;
5> restore database;
6> recover database noredo;
7> release channel t1;
8> }

allocated channel: t1
channel t1: SID=396 device type=SBT_TAPE
channel t1: Data Protection for Oracle: version 5.5.2.0

Starting restore at 22-MAR-13

channel t1: restoring control file
channel t1: restore complete, elapsed time: 00:00:01
output file name=/u01/oradata/TRNGDB/control01.ctl
output file name=/u01/oradata/TRNGDB/control02.ctl
output file name=/u01/oradata/TRNGDB/control03.ctl
Finished restore at 22-MAR-13

database mounted

Starting restore at 22-MAR-13

channel t1: starting datafile backup set restore
channel t1: specifying datafile(s) to restore from backup set
channel t1: restoring datafile 00003 to /u01/oradata/TRNGDB/sysaux01.dbf
channel t1: restoring datafile 00004 to /u01/oradata/TRNGDB/psdefault.dbf
channel t1: restoring datafile 00009 to /u01/oradata/TRNGDB/aalarge.dbf
channel t1: restoring datafile 00011 to /u01/oradata/TRNGDB/adapp.dbf
channel t1: restoring datafile 00012 to /u01/oradata/TRNGDB/adapp001002.dbf
channel t1: restoring datafile 00014 to /u01/oradata/TRNGDB/avapp.dbf
channel t1: restoring datafile 00015 to /u01/oradata/TRNGDB/avapp001002.dbf
channel t1: restoring datafile 00017 to /u01/oradata/TRNGDB/avapp001004.dbf
channel t1: restoring datafile 00022 to /u01/oradata/TRNGDB/ccapp.dbf
channel t1: restoring datafile 00030 to /u01/oradata/TRNGDB/cularg3.dbf
channel t1: restoring datafile 00031 to /u01/oradata/TRNGDB/diapp.dbf
channel t1: restoring datafile 00036 to /u01/oradata/TRNGDB/faapp001002.dbf
channel t1: restoring datafile 00038 to /u01/oradata/TRNGDB/fgapp.dbf
channel t1: restoring datafile 00041 to /u01/oradata/TRNGDB/giapp.dbf
channel t1: restoring datafile 00043 to /u01/oradata/TRNGDB/glapp.dbf
channel t1: restoring datafile 00047 to /u01/oradata/TRNGDB/hpapp001002.dbf
channel t1: restoring datafile 00054 to /u01/oradata/TRNGDB/hrapp4001002.dbf
channel t1: restoring datafile 00063 to /u01/oradata/TRNGDB/hrwork001002.dbf
channel t1: restoring datafile 00065 to /u01/oradata/TRNGDB/htapp001002.dbf
channel t1: restoring datafile 00067 to /u01/oradata/TRNGDB/inlarge.dbf
channel t1: restoring datafile 00068 to /u01/oradata/TRNGDB/paapp.dbf
channel t1: restoring datafile 00069 to /u01/oradata/TRNGDB/paapp001001.dbf
channel t1: restoring datafile 00070 to /u01/oradata/TRNGDB/palarge.dbf
channel t1: restoring datafile 00077 to /u01/oradata/TRNGDB/piwork001002.dbf
channel t1: restoring datafile 00081 to /u01/oradata/TRNGDB/psindx001002.dbf
channel t1: restoring datafile 00085 to /u01/oradata/TRNGDB/psindx001006.dbf
channel t1: restoring datafile 00086 to /u01/oradata/TRNGDB/psindx001007.dbf
channel t1: restoring datafile 00089 to /u01/oradata/TRNGDB/psindx001010.dbf
channel t1: restoring datafile 00098 to /u01/oradata/TRNGDB/ptcmstar.dbf
channel t1: restoring datafile 00100 to /u01/oradata/TRNGDB/ptprc.dbf
channel t1: restoring datafile 00107 to /u01/oradata/TRNGDB/pttlrg001002.dbf
channel t1: restoring datafile 00115 to /u01/oradata/TRNGDB/pywork.dbf
channel t1: restoring datafile 00121 to /u01/oradata/TRNGDB/saapp001005.dbf
channel t1: restoring datafile 00125 to /u01/oradata/TRNGDB/salarge.dbf
channel t1: restoring datafile 00136 to /u01/oradata/TRNGDB/tlapp001002.dbf
channel t1: restoring datafile 00140 to /u01/oradata/TRNGDB/tlwork001001.dbf
channel t1: restoring datafile 00142 to /u01/oradata/TRNGDB/waapp.dbf
channel t1: restoring datafile 00147 to /u01/oradata/TRNGDB/rbks001002.dbf
channel t1: restoring datafile 00149 to /u01/oradata/TRNGDB/epapp001001.dbf
channel t1: restoring datafile 00161 to /u01/oradata/TRNGDB/eoeiapp001001.dbf
channel t1: restoring datafile 00162 to /u01/oradata/TRNGDB/eodsapp001001.dbf
channel t1: restoring datafile 00166 to /u01/oradata/TRNGDB/hrslarge001001.dbf
channel t1: restoring datafile 00170 to /u01/oradata/TRNGDB/eopplrg001001.dbf
channel t1: restoring datafile 00174 to /u01/oradata/TRNGDB/eocuapp001001.dbf
channel t1: restoring datafile 00178 to /u01/oradata/TRNGDB/eoeilrg001001.dbf
channel t1: restoring datafile 00182 to /u01/oradata/TRNGDB/eoiuwrk001001.dbf
channel t1: restoring datafile 00186 to /u01/oradata/TRNGDB/erwork001001.dbf
channel t1: restoring datafile 00190 to /u01/oradata/TRNGDB/hrlarg1001001.dbf
channel t1: restoring datafile 00192 to /u01/oradata/TRNGDB/psimgr001001.dbf
channel t1: restoring datafile 00196 to /u01/oradata/TRNGDB/pswork001001.dbf
channel t1: restoring datafile 00200 to /u01/oradata/TRNGDB/hrapp001004.dbf
channel t1: restoring datafile 00203 to /u01/oradata/TRNGDB/clnd001001.dbf
channel t1: restoring datafile 00205 to /u01/oradata/TRNGDB/clnd001002
channel t1: reading from backup piece backup_TRNGDB_3679252776_set155_piece1_copy1_20121027_4rnoorig_1_1
channel t1: piece handle=backup_TRNGDB_3679252776_set155_piece1_copy1_20121027_4rnoorig_1_1 tag=NOEXPIRE

..
....
RMAN> alter database open resetlogs;

database opened
new incarnation of database registered in recovery catalog
starting full resync of recovery catalog
full resync complete

RMAN> exit


So we are able to successfully restore the backups from a different incarnation. The same procedure can be applied if you plan to restore the database to a different server as long as the tsm configuration is properly done.

Hope It Helps

SRI

Troubleshooting RAC Services Startup

Written By Unknown on Wednesday, April 3, 2013 | 3:03 PM

Recently we had a DR(Disaster Recovery) testing going on and we had RAC replica at the DR site for DR testing. The RAC servers are copied to DR setup, bit by bit. Then tried to start the RAC services at DR site and could not start them due to some issues.

Then I tried to troubleshoot the issue .....

This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.

STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.

STEP 2: Verify storage on both rac servers
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su -
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh        8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:1:1 sdk        8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:1:0 sdj        8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg        8:96  [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su -
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk        8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:0:1 sdh        8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:0:0 sdg        8:96  [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj        8:144 [active][ready]
[root@askmdbrac02 ~]#

Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.

STEP 3: Check high availability services and cluster ready services on both nodes with the following commands
[grid@askmdbrac01 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$

Observation : The high availability services are up , but crs services are down.

STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.

Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.

STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource 'ora.diskmon' has failed on server 'askmdbrac01'.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
.....
.....
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'askmdbrac01'.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource 'ora.cssd'; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource 'ora.diskmon' has failed on server 'askmdbrac01'.

Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can't find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.

Then i tried to see if i can see voting disks through command.

SETP 6 :  Check the voting disks availability ( 11gR2 has voting disks in asm. I don't have asm instance up now. So i will get error if i try to query the voting disks on asm)
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$

Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.

STEP 7 :  Tried re-enabling the asmlibs as follows on both the nodes ...
[root@askmdbrac01 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#  /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Not able to start services even after restarting the asmlib.

STEP 8 : Tried to re-configure the asmlib as below .....
[root@askmdbrac01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#

[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#


Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.

STEP 9 : Stop and Start cluster services on both the nodes.
[root@askmdbrac01 ~]#  /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'askmdbrac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.racdb.racdb_service.svc' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmdbrac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'askmdbrac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.racdb.racdb_service.svc' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.askmdbrac01.vip' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'askmdbrac01'
CRS-2677: Stop of 'ora.askmdbrac01.vip' on 'askmdbrac01' succeeded
CRS-2672: Attempting to start 'ora.askmdbrac01.vip' on 'askmdbrac02'
CRS-2676: Start of 'ora.scan1.vip' on 'askmdbrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02'
CRS-2677: Stop of 'ora.racdb.db' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.PRDSHR_DATA.dg' on 'askmdbrac01'
CRS-2676: Start of 'ora.askmdbrac01.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.PRDSHR_DATA.dg' on 'askmdbrac01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac01'
CRS-2677: Stop of 'ora.asm' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'askmdbrac01'
CRS-2677: Stop of 'ora.ons' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'askmdbrac01'
CRS-2677: Stop of 'ora.net1.network' on 'askmdbrac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'askmdbrac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.crf' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac01'
CRS-2677: Stop of 'ora.mdnsd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.crf' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'askmdbrac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'askmdbrac01'
CRS-2677: Stop of 'ora.cssd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.diskmon' on 'askmdbrac01'
CRS-2677: Stop of 'ora.gipcd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'askmdbrac01'
CRS-2677: Stop of 'ora.gpnpd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'askmdbrac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'askmdbrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'askmdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.askmdbrac02.vip' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.racdb.racdb_service.svc' on 'askmdbrac02'
CRS-2677: Stop of 'ora.askmdbrac02.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.racdb.racdb_service.svc' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.racdb.db' on 'askmdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.askmdbrac01.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.scan1.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.askmdbrac01.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.scan2.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.racdb.db' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.PRDSHR_DATA.dg' on 'askmdbrac02'
CRS-2677: Stop of 'ora.FRA.dg' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.PRDSHR_DATA.dg' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac02'
CRS-2677: Stop of 'ora.asm' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'askmdbrac02'
CRS-2677: Stop of 'ora.ons' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'askmdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'askmdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'askmdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.evmd' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.cssd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.crf' on 'askmdbrac02'
CRS-2677: Stop of 'ora.crf' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.diskmon' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'askmdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'askmdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#

Observation : Cluster ready services started without any issues. Now i need to verify the services status.

STEP 10 :  Verify the status.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora....up.type ONLINE    ONLINE    askmdbrac01
ora....ER.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....N1.lsnr ora....er.type ONLINE    OFFLINE
ora....N2.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....N3.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....DATA.dg ora....up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora....SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora....01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora....l01.gsd application    OFFLINE   OFFLINE
ora....l01.ons application    ONLINE    ONLINE    askmdbrac01
ora....l01.vip ora....t1.type ONLINE    ONLINE    askmdbrac01
ora....SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora....02.lsnr application    ONLINE    OFFLINE
ora....l02.gsd application    OFFLINE   OFFLINE
ora....l02.ons application    ONLINE    OFFLINE
ora....l02.vip ora....t1.type ONLINE    OFFLINE
ora....ry.acfs ora....fs.type ONLINE    OFFLINE
ora.scan1.vip  ora....ip.type ONLINE    OFFLINE
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora....se.type ONLINE    ONLINE    askmdbrac01
ora....ice.svc ora....ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac01 ~]# sudo su - grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
-----------                                        ------     -----
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
-----------                                        ------     -----
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora....up.type ONLINE    ONLINE    askmdbrac01
ora....ER.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....N1.lsnr ora....er.type ONLINE    ONLINE    askmdbrac02
ora....N2.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....N3.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....DATA.dg ora....up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora....SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora....01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora....l01.gsd application    OFFLINE   OFFLINE
ora....l01.ons application    ONLINE    ONLINE    askmdbrac01
ora....l01.vip ora....t1.type ONLINE    ONLINE    askmdbrac01
ora....SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora....02.lsnr application    ONLINE    ONLINE    askmdbrac02
ora....l02.gsd application    OFFLINE   OFFLINE
ora....l02.ons application    ONLINE    ONLINE    askmdbrac02
ora....l02.vip ora....t1.type ONLINE    ONLINE    askmdbrac02
ora....ry.acfs ora....fs.type ONLINE    OFFLINE
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    askmdbrac02
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora....se.type ONLINE    ONLINE    askmdbrac01
ora....ice.svc ora....ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac02 ~]# sudo su - grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource                                        Target     State
-----------                                        ------     -----
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$

This concludes the article on troubleshooting RAC services startup issue.

Thanks
SRI
 
Support :
Copyright © 2013. askMLabs - All Rights Reserved
Proudly powered by Blogger