askMLabs

This article explains the OEM grid control 11g ( 11.1.0.1.0 ) installation on Linux 5.4 (32-bit) OS with 11gR2 database.

Software Requirements:

SUN JDK 1.6_18 ( for 32/64 bit Linux OS) : Download
WebLogic Server 10.3.2(32bit) : Download
Oracle Database 11gR2 : Download
OEM 11g Grid Control( 11.1.0.1.0 ) : Download

Grid Control 11g uses WebLogic Server instead of OC4J which was traditional with Grid Control versions 10.1 and 10.2. Grid Control 11g will deploy from an existing Weblogic Server during the Grid Control Installation. Weblogic server installation requires a JDK 1.6_18 as a pre requisite. A database for repository use must exist at the time of Grid Control 11.1.0.1.0 installation. The option to "Install Grid Control with a new database" is no longer offered. So we need to have  the database created before starting the grid control 11g installation. Please review the Oracle Enterprise Manager Grid Control Certification Checker @ 412431.1 for the database version.

So the sequence of steps to install 11g grid control including all the external components are ..

1.Create Linux Machine:

If you are planning to install OEM grid control in a vmware linux machine, Please follow the article VMware Server - Linux installation  and complete the linux machine creation. You can omit this point if you are directly provided with linux machine with OS linux 5.4(32-bit).

2.Prepare the linux machine for OEM grid control 11g installation:

Prepare the linux machine ( Linux 5.4 ) for the OEM grid control 11g installation. Please use the article Prepare Linux virtual machine for 11gR2 installation on Linux 5.x. Though this is to prepare the linux machine for 11gR2 database, it hold good for OEM 11g installation also. You can also refer the installation document here.

Check the following RPMs and make sure all the RPMs are installed.

[sourcecode language="bash"]
[oracle@oem11gmach01 ~]$ rpm -qa | egrep -i "binutils|compat-libstdc|elfutils-libelf|glibc|glibc-common|ksh|libaio|libgcc|libstdc|make"
libgcc-4.1.2-48.el5
automake16-1.6.3-8.el5.1
elfutils-libelf-devel-static-0.137-3.el5
compat-libgcc-296-2.96-138
libstdc++-4.1.2-48.el5
imake-1.0.2-3
compat-libstdc++-33-3.2.3-61
glibc-headers-2.5-49
automake14-1.4p6-13.el5.1
ksh-20100202-1.el5
glibc-common-2.5-49
glibc-2.5-49
elfutils-libelf-0.137-3.el5
binutils-2.17.50.0.6-14.el5
compat-libstdc++-296-2.96-138
glibc-devel-2.5-49
automake-1.9.6-2.3.el5
automake17-1.7.9-7.el5.2
make-3.81-3.el5
libaio-0.3.106-5
automake15-1.5-16.el5.2
elfutils-libelf-devel-0.137-3.el5
libstdc++-devel-4.1.2-48.el5
MAKEDEV-3.23-1.2
[oracle@oem11gmach01 ~]$

[oracle@oem11gmach01 ~]$ rpm -qa | egrep -i "elfutils-libelf-devel|kernel-headers|glibc-headers|glibc-devel|libstdc++-devel|libgomp|gcc-|gcc-c++|libaio-devel|sysstat|unixODBC"
kernel-headers-2.6.18-194.el5
libgcc-4.1.2-48.el5
gcc-gfortran-4.1.2-48.el5
elfutils-libelf-devel-static-0.137-3.el5
compat-libgcc-296-2.96-138
glibc-headers-2.5-49
libgomp-4.4.0-6.el5
glibc-devel-2.5-49
gcc-4.1.2-48.el5
gcc-c++-4.1.2-48.el5
elfutils-libelf-devel-0.137-3.el5
[oracle@oem11gmach01 ~]$ rpm -qa | grep libstdc++-devel
libstdc++-devel-4.1.2-48.el5
[oracle@oem11gmach01 ~]$ rpm -qa | grep libaio-devel
[oracle@oem11gmach01 ~]$

[oracle@oem11gmach01 ~]$ rpm -qa | egrep -i "rng-utils|make|binutils|gcc|libaio|glibc-common|compat-libstdc++|libstdc++|setarch|compat-db"
libgcc-4.1.2-48.el5
automake16-1.6.3-8.el5.1
gcc-gfortran-4.1.2-48.el5
compat-libgcc-296-2.96-138
libstdc++-4.1.2-48.el5
imake-1.0.2-3
compat-libstdc++-33-3.2.3-61
automake14-1.4p6-13.el5.1
rng-utils-2.0-1.14.1.fc6
glibc-common-2.5-49
binutils-2.17.50.0.6-14.el5
compat-libstdc++-296-2.96-138
automake-1.9.6-2.3.el5
automake17-1.7.9-7.el5.2
make-3.81-3.el5
setarch-2.0-1.1
libaio-0.3.106-5
automake15-1.5-16.el5.2
gcc-4.1.2-48.el5
gcc-c++-4.1.2-48.el5
libstdc++-devel-4.1.2-48.el5
MAKEDEV-3.23-1.2
[oracle@oem11gmach01 ~]$ rpm -qa | grep -i compat-db
[oracle@oem11gmach01 ~]$

[root@oem11gmach01 sf_Shared_folder]# rpm -ivh libaio-devel-0.3.106-3.2.i386.rpm
warning: libaio-devel-0.3.106-3.2.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing...                ########################################### [100%]
1:libaio-devel           ########################################### [100%]
[root@oem11gmach01 sf_Shared_folder]# rpm -qa | grep -i libaio-devel
libaio-devel-0.3.106-3.2
[root@oem11gmach01 sf_Shared_folder]# rpm -ivh compat-db-4.2.52-5.1.i386.rpm
warning: compat-db-4.2.52-5.1.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing...                ########################################### [100%]
1:compat-db              ########################################### [100%]
[root@oem11gmach01 sf_Shared_folder]# rpm -qa | grep -i compat-db
compat-db-4.2.52-5.1
[root@oem11gmach01 sf_Shared_folder]# rpm -ivh sysstat-7.0.0-3.el5.i386.rpm
warning: sysstat-7.0.0-3.el5.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing...                ########################################### [100%]
1:sysstat                ########################################### [100%]
[root@oem11gmach01 sf_Shared_folder]# rpm -ivh unixODBC-2.2.11-7.1.i386.rpm unixODBC-devel-2.2.11-7.1.i386.rpm
warning: unixODBC-2.2.11-7.1.i386.rpm: Header V3 DSA signature: NOKEY, key ID 37017186
Preparing...                ########################################### [100%]
1:unixODBC               ########################################### [ 50%]
2:unixODBC-devel         ########################################### [100%]
[root@oem11gmach01 sf_Shared_folder]#
[/sourcecode]

Create Directory Structure:

Create following directories and make sure you give read / write permission to oracle user.


[sourcecode language="bash"]
/u01/app/OEM11g/Middleware
/u01/app/OEM11g/Middleware/oms11g
/u01/app/OEM11g/Middleware/agent11g
/u01/app/OEM11g/Middleware/Oracle_WT
/u01/app/OEM11g/Middleware/oracle_common
/u01/app/OEM11g/gc_inst
/u01/app/oracle/oradata/OEMDB
[/sourcecode]

3.Install JDK 1.6_18:

Installing JDK depends on the OS bit version you are using. If you are using 32-bit version OS, you will find weblogic server bundled with JDK. But if you are using 64-bit OS,you must install an appropriate jdk to support the WebLogic Server and Grid Control 11.1.0.1.0 installations on 64bit platforms, since the required jdk is not bundles with the installer as it is with the 32bit platforms.
In this demo, i am using a 32-bit OS and i can skip this part of installing JDK. For those we need to install JDK 1.6_18 for 64-bit OS, Please follow the detailed procedure explained with screen shots here. For more details, Please follow the MOS ID : 1063587.1.

NOTE: jdk 1.6.0_18 is recommended due to potential problems with newer versions.

jdk 1.6.0 updates 19 and higher do not allow subsequent patching with required WLS patch WDJ7, so these versions are not effective and must be avoided. They may be used when a cacerts file that allows Smartupdate can be appropriately substituted for the native one.

[sourcecode language="bash"]
[root@oem11gmach01 Grid Control 11g]# mkdir -p /u01/app/OEM11g/Middleware
[root@oem11gmach01 Grid Control 11g]# chown -R oracle:oinstall /u01/app/OEM11g/Middleware/
[root@oem11gmach01 Grid Control 11g]# chmod -R 755 /u01/app/OEM11g/Middleware/
[root@oem11gmach01 Grid Control 11g]# cd /u01/app/OEM11g/Middleware/
[root@oem11gmach01 Middleware]# mkdir jdk
[root@oem11gmach01 Middleware]# chown -R oracle:oinstall jdk
[root@oem11gmach01 Middleware]# chmod 755 jdk
[root@oem11gmach01 Middleware]# cd -
/media/sf_Shared_folder/Grid Control 11g
[root@oem11gmach01 Grid Control 11g]# cp jdk-6u18-linux-i586.bin /u01/app/OEM11g/Middleware/jdk/
[root@oem11gmach01 Grid Control 11g]# chown oracle:oinstall /u01/app/OEM11g/Middleware/jdk/jdk-6u18-linux-i586.bin
[root@oem11gmach01 Grid Control 11g]# chmod 755 /u01/app/OEM11g/Middleware/jdk/jdk-6u18-linux-i586.bin
[root@oem11gmach01 Grid Control 11g]# su - oracle
/u01/app/oracle/product/11.2.0/db_1
OEMDB
[oracle@oem11gmach01 ~]$ cd /u01/app/OEM11g/Middleware/jdk/
[oracle@oem11gmach01 jdk]$

[oracle@oem11gmach01 bin]$ ./jdk-6u18-linux-i586.bin

[oracle@oem11gmach01 jdk]$ pwd
/u01/app/OEM11g/Middleware/jdk
[oracle@oem11gmach01 jdk]$ ls
jdk1.6.0_18  jdk-6u18-linux-i586.bin
[oracle@oem11gmach01 jdk]$ cd jdk1.6.0_18/
[oracle@oem11gmach01 jdk1.6.0_18]$ ls
bin        db    include  lib      man          README_ja.html     register.html     register_zh_CN.html  src.zip
COPYRIGHT  demo  jre      LICENSE  README.html  README_zh_CN.html  register_ja.html  sample               THIRDPARTYLICENSEREADME.txt
[oracle@oem11gmach01 jdk1.6.0_18]$ cd bin
[oracle@oem11gmach01 bin]$ ./java -version
java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Client VM (build 16.0-b13, mixed mode, sharing)
[oracle@oem11gmach01 bin]$
[oracle@oem11gmach01 ~]$ java -fullversion
java full version "1.6.0_18-b07"
[oracle@oem11gmach01 ~]$
[/sourcecode]

4.Installing Weblogic Server 10.3.2:

The only version of the WebLogic Server that can be used to deploy Grid Control is 10.3.2.The WebLogic Server associated with Oracle products such as SOA Suite or Identity Management is not appropriate for use in deploying Grid Control.

[youtube http://www.youtube.com/watch?v=B2EJfoC0aS8&hl=en&fs=1]

After completing the installation, we need to apply the recommended patches to WLS for grid control installation. Either use SmartUpdate tool or use offline method to install patch. Download the WLS patch WDJ7 manually and place it in the directory WLS_ORACLE_HOME/utils/bsu/cache_dir.

Offline Method:

[sourcecode language="bash"]

Apply the patch by running the following command:-
./bsu.sh -prod_dir=WLS_ORACLE_HOME/wlserver_10.3 -patchlist=patch number -verbose -install
For example:
./bsu.sh -prod_dir=/scratch/gc/Oracle/Middleware/wlserver_10.3 -patchlist=WDJ7 -verbose -install

You can view the report of applied patched by running the following command:-
./bsu.sh -report -patch_id_mask=patch number
For example: ./bsu.sh -report -patch_id_mask=WDJ7

[/sourcecode]

Using SmartUpdateTool(please watch the demo)

Watch this demo which shows how to apply the recommended patches to weblogic server.

[youtube http://www.youtube.com/watch?v=ejnKK6BpzNs&hl=en&fs=1]

A step-by-step description with graphical representation of the installer progress is available here.

5.Install Database 11.2.0.1:

The option to "Install Grid Control with a new database" is no longer offered. So we need to have  the database created before starting the grid control 11g installation.

Please follow the article "Installing 11gR2 Single Instance Database With Non-ASM Storage" for installling 11g database software and creating a database for OEM grid control repository.

[youtube http://www.youtube.com/watch?v=ePitGvp_zDs&hl=en&fs=1]

This procedure with step-by-step screen shot graphics can be found here.  ( This guide installs 11.1.0.7 Database, but we are installing 11gR2 in our demo ).

6.Prepare the database for OEM 11g Installation.

[sourcecode language="sql"]
SQL> select active_state from v$instance;

ACTIVE_ST
---------
NORMAL

SQL> SELECT COUNT(*) FROM ALL_USERS WHERE USERNAME='SYSMAN';

COUNT(*)
----------
1

SQL> SELECT COUNT(*) FROM ALL_USERS WHERE USERNAME IN ('SYSMAN','SYSMAN_MDS');

COUNT(*)
----------
1

SQL> !

=>Deconfigure DB console :

[oracle@oem11gmach01 database]$ $ORACLE_HOME/bin/emca -deconfig dbcontrol db -repos drop -SYS_PWD welcome1 -SYSMAN_PWD welcome1

STARTED EMCA at Apr 27, 2011 10:38:10 PM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: OEMDB
Listener port number: 1521

Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 27, 2011 10:38:48 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/OEMDB/emca_2011_04_27_22_38_10.log.
Apr 27, 2011 10:38:49 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Apr 27, 2011 10:39:23 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
Apr 27, 2011 10:43:19 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 27, 2011 10:43:34 PM
[oracle@oem11gmach01 database]$ exit
exit

SQL> SELECT COUNT(*) FROM ALL_USERS WHERE USERNAME='SYSMAN';

COUNT(*)
----------
0

SQL> SELECT COUNT(*) FROM ALL_USERS WHERE USERNAME IN ('SYSMAN','SYSMAN_MDS');

COUNT(*)
----------
0

SQL> select value from v$option where parameter='Fine-grained access control';

VALUE
----------------------------------------------------------------
TRUE

SQL>
SQL> select sum(decode(autoextensible,'YES',200*1024*1024+1,bytes)) total from dba_data_files f, dba_tablespaces s
2  where s.contents='UNDO' and s.tablespace_name=f.tablespace_name;

TOTAL
----------
209715201

SQL> select count(*) from dba_temp_files where tablespace_name='TEMP' and AUTOEXTENSIBLE <> 'YES';

COUNT(*)
----------
0

SQL> select count(*) from dba_data_files where tablespace_name='UNDOTBS' and AUTOEXTENSIBLE <> 'YES';

COUNT(*)
----------
0

SQL> select log_mode from v$database;

LOG_MODE
------------
NOARCHIVELOG

SQL>
[/sourcecode]

Optional : Verify the init.ora parameters as specified in the installation document

7.Install OEM grid control 11.1.0.1.0:

There are three ways to install Enterprise Manager.
1) Installing Enterprise Manager System with graphical mode including all configuration tools
2) Installing Enterprise Manager Software Only in Graphical Mode and Configuring Later
3) Installing Enterprise Manager Software Only in Silent Mode and Configuring Later

In the present demo i am choosing option 2 to install Enterprise Manager.

Go to the directory where the software is unzipped and start the installation with the following command ...

[sourcecode language="bash"]
<Software_Location>/runInstaller -noconfig
[/sourcecode]

After successful installation of the OEM grid control 11g software, install the configuration tools using

[sourcecode language="bash"]
$<OMS_ORACLE_HOME>/perl/bin/perl <ORACLE_HOME>/sysman/install/ConfigureGC.pl
Ex : /u01/app/OEM11g/Middleware/oms11g/perl/bin/perl /u01/app/OEM11g/Middleware/oms11g/sysman/install/ConfigureGC.pl
[/sourcecode]

Watch the video to install OEM grid control 11g software.

[youtube http://www.youtube.com/watch?v=VmhzWjgsprg&hl=en&fs=1]

For detailed explanation, verify MOS ID 1064495.1 and for step by step screen shot explanation of the installation, refer the the document here.

8.Status Check:

[sourcecode language="bash" highlight="015,023,054,063,091,101,102"]
[oracle@oem11gmach01 ~]$ echo $ORACLE_HOME
/u01/app/oracle/product/11.2.0/db_1
[oracle@oem11gmach01 ~]$ echo $ORACLE_SID
OEMDB
[oracle@oem11gmach01 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.1.0 Production on Sat Apr 30 02:27:52 2011

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select instance_name,status from v$instance;

INSTANCE_NAME    STATUS
---------------- ------------
OEMDB            OPEN

SQL>


[oracle@oem11gmach01 ~]$ lsnrctl status listener

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 30-APR-2011 02:28:14

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
STATUS of the LISTENER
------------------------
Alias                     listener
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                29-APR-2011 23:55:06
Uptime                    0 days 2 hr. 33 min. 9 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/oem11gmach01/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oem11gmach01.localdomain)(PORT=1521)))
Services Summary...
Service "OEMDB.localdomain" has 1 instance(s).
Instance "OEMDB", status READY, has 1 handler(s) for this service...
Service "OEMDBXDB.localdomain" has 1 instance(s).
Instance "OEMDB", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@oem11gmach01 ~]$



[oracle@oem11gmach01 bin]$ pwd
/u01/app/OEM11g/Middleware/oms11g/bin
[oracle@oem11gmach01 bin]$ ./emctl status oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
[oracle@oem11gmach01 bin]$



[oracle@oem11gmach01 bin]$ pwd
/u01/app/OEM11g/Middleware/agent11g/bin
[oracle@oem11gmach01 bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /u01/app/OEM11g/Middleware/agent11g
Agent binaries    : /u01/app/OEM11g/Middleware/agent11g
Agent Process ID  : 10432
Parent Process ID : 10400
Agent URL         : https://oem11gmach01.localdomain:3872/emd/main/
Repository URL    : https://oem11gmach01.localdomain:4900/em/upload
Started at        : 2011-04-30 02:04:10
Started by user   : oracle
Last Reload       : 2011-04-30 02:05:44
Last successful upload                       : 2011-04-30 02:25:26
Total Megabytes of XML files uploaded so far :    21.91
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    25.10%
Last successful heartbeat to OMS             : 2011-04-30 02:24:52
---------------------------------------------------------------
Agent is Running and Ready
[oracle@oem11gmach01 bin]$



[oracle@oem11gmach01 bin]$ pwd
/u01/app/OEM11g/gc_inst/WebTierIH1/bin
[oracle@oem11gmach01 bin]$ ./opmnctl status

Processes in Instance: instance1
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------
ohs1                             | OHS                |    7870 | Alive

[oracle@oem11gmach01 bin]$


Access the URL:

https://oem11gmach01.localdomain:4900/em/
[/sourcecode]

9.Issues and Resolutions

During Enterprise Manager installation, the configuration tool "Repository Configuration Assistant" failed.

Reviewed the following logfiles:
$<OMS_ORACLE_HOME>/cfgtoollogs/cfgfw/emmdscreate_<timestamp>.log
$<OMS_ORACLE_HOME>/sysman/log/schemamanager/m_<timestamp>/m_<timestamp>.<ACTION>/

Following information found in various logfiles under the directories mentioned above:

[sourcecode language="bash"]
[Initializing the EM RCU Runner, Failed Parameter Validation]
action logger path calculated finally: /u01/app/OEM11g/Middleware/oms11g/sysman/log/schemamanager/m_042911_1243_AM/m_042911_1243_AM.TRANSX/

==================================================

2011-04-29 00:56:36,991 [vd92mfm9eocy] INFO  emschema.vd92mfm9eocy - not found repos connection
2011-04-29 00:56:36,992 [vd92mfm9eocy] INFO  emschema.vd92mfm9eocy - error: not found repos connection for transx repos user exists: false mgmt_messages exists: false; found connection for repos user: false
2011-04-29 00:56:36,992 [vd92mfm9eocy] ERROR emschema.vd92mfm9eocy - Failed Parameter Validation
2011-04-29 00:56:36,992 [vd92mfm9eocy] ERROR emschema.vd92mfm9eocy - NULL Exception was thrown!
2011-04-29 00:56:36,992 [vd92mfm9eocy] ERROR emschema.vd92mfm9eocy - Failed Parameter Validation
"/u01/app/OEM11g/Middleware/oms11g/sysman/log/schemamanager/m_042911_1243_AM/m_042911_1243_AM.TRANSX/emschema.log.TRANSX" 407L, 44064C     407,1         Bot

========================

INFO: oracle.sysman.top.oms:[Initializing the EM RCU Runner, Failed Parameter Validation]
INFO: oracle.sysman.top.oms: action logger path calculated finally: /u01/app/OEM11g/Middleware/oms11g/sysman/log/schemamanager/m_042911_0115_AM/m_042911_0115_AM.TRANSX/
INFO: oracle.sysman.top.oms:The transx command execution is completed with the status 0
INFO: oracle.sysman.top.oms:process_id:
INFO: oracle.sysman.top.oms:The Process ID: [] does not exist.  Either it has been completed, or this is not the executing JVM
INFO: oracle.sysman.top.oms:The plug-in Repository Configuration has failed its perform method

====================

2011-04-29 01:29:44,272 [xkdp7xnv3zx2] INFO  emschema.xkdp7xnv3zx2 - not found repos connection
2011-04-29 01:29:44,272 [xkdp7xnv3zx2] INFO  emschema.xkdp7xnv3zx2 - error: not found repos connection for transx repos user exists: false mgmt_messages exists: false; found connection for repos user: false
2011-04-29 01:29:44,273 [xkdp7xnv3zx2] ERROR emschema.xkdp7xnv3zx2 - Failed Parameter Validation
2011-04-29 01:29:44,273 [xkdp7xnv3zx2] ERROR emschema.xkdp7xnv3zx2 - NULL Exception was thrown!
2011-04-29 01:29:44,273 [xkdp7xnv3zx2] ERROR emschema.xkdp7xnv3zx2 - Failed Parameter Validation

===========================

INFO: oracle.sysman.top.oms:The command run for transx is /u01/app/OEM11g/Middleware/oms11g/sysman/admin/emdrep/bin/RepManager -connect (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem11gmach01.localdomain)(PORT=1521)))(CONNECT_DATA=(SID=OEMDB))) -action transx  -repos_user SYSMAN -dbUser SYS -mwHome /u01/app/OEM11g/Middleware -mwOraHome /u01/app/OEM11g/Middleware/oms11g
INFO: oracle.sysman.top.oms:processing arguments
INFO: oracle.sysman.top.oms:compiling arguments for validation
INFO: oracle.sysman.top.oms:stty: standard input: Invalid argument
INFO: oracle.sysman.top.oms:stty: standard input: Invalid argument
INFO: oracle.sysman.top.oms: Enter SYSMAN repository password :
INFO: oracle.sysman.top.oms:stty: standard input: Invalid argument
INFO: oracle.sysman.top.oms:stty: standard input: Invalid argument
INFO: oracle.sysman.top.oms:Verify :
INFO: oracle.sysman.top.oms:Enter password for: SYSMAN
INFO: oracle.sysman.top.oms:^H ^Hdone checking required user credentials...
INFO: oracle.sysman.top.oms:done creating arguments...
[/sourcecode]

Solution:

[sourcecode language="bash"]
Clean up the Management Repository by running the following command
/u01/app/OEM11g/Middleware/oms11g/sysman/admin/emdrep/bin/RepManager oem11gmach01.localdomain 1521 OEMDB -action dropall -dbUser SYS -dbPassword xxxxxxx -dbRole sysdba -mwHome /u01/app/OEM11g/Middleware -mwOraHome /u01/app/OEM11g/Middleware/oms11g -oracleHome /u01/app/OEM11g/Middleware/oms11g  -continueOnError

Then re-run the configuration tool script.
/u01/app/OEM11g/Middleware/oms11g/perl/bin/perl /u01/app/OEM11g/Middleware/oms11g/sysman/install/ConfigureGC.pl
[/sourcecode]

If there are any failed configuration tools, please refer the document here.

Reference MOS IDs :  1067438.1 , 1106105.1 , 1063587.1 , 1063112.1 , 1064441.1 , 1064495.1 , Install Guide

Thanks
SRI

This article describes enabling and disabling the SCAN listener tracing. Before capturing the scan listener tracing,please make sure your scan listener is configured as per the following Oracle Support IDs.
1070607.1
975457.1

Some where you should find some clue for your SCAN connectivity issues. If you are still unable to find the exact problem with the scan connectivity, you need to enable the tracing and analyze the trace files or upload the trace files to oracle support.

To enable level 16 tracing for SCAN listeners, add the following parameters to listener.ora and  try to reproduce the connectivity issue.

TRACE_LEVEL_<listener_name> = 16
TRACE_DIRECTORY_<listener_name> =<dir location>
TRACE_TIMESTAMP_<listener_name> = TRUE
DIAG_ADR_ENABLED_<listenername>=off

Then try to reload all the listeners after these parameters are in place using lsnrctl reload <listener_name>.

Reproduce the issue and verify that the trace and log files are created in the directory mentioned in the parameter TRACE_DIRECTORY_<listener_name>.

To disable scan listener tracing:

$lsnrctl
LSNRCTL>set current_listener <listener_name>
LSNRCTL>set trc_level OFF
LSNRCTL>save_config

In the above tracing we disabled diagnostic repository with the parameter DIAG_ADR_ENABLED_<listenername> and collected trace files. Now if you want to re-enable DIAG_ADR, just comment out DIAG_ADR_ENABLED_<listenername> in listener.ora and reload all the listeners.

NOTE: ADR is the new concept introduced from 11g and it is a file system repository to store all diagnostic data. Please review the article  Automatic Diagnostic Repository  to know more on ADR concepts.

Session Log to show enabling SCAN Tracing:

[grid@b600e6tr01 admin]$ cp listener.ora listener.ora_bak
[grid@b600e6tr01 admin]$ vi listener.ora   ==> Here add trace parameter
[grid@b600e6tr01 admin]$ clear
[grid@b600e6tr01 admin]$ ps -ef | grep -i tns
grid      5943  4623  0 07:12 pts/1    00:00:00 grep -i tns
grid     18139     1  0 May03 ?        00:01:01 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid     18142     1  0 May03 ?        00:01:14 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
grid     18164     1  0 May03 ?        00:01:18 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
[grid@b600e6tr01 admin]$ lsnrctl reload listener
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:12:38
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
The command completed successfully
[grid@b600e6tr01 admin]$ lsnrctl reload LISTENER_SCAN2
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:12:55
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
The command completed successfully
[grid@b600e6tr01 admin]$ lsnrctl reload LISTENER_SCAN3
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:13:02
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
The command completed successfully
[grid@b600e6tr01 admin]$
[grid@b600e6tr02 admin]$ cp listener.ora listener.ora_bak
[grid@b600e6tr02 admin]$ vi listener.ora  ==> Here add trace parameters
[grid@b600e6tr02 ~]$ ps -ef | grep -i tns
grid      1042     1  0 May03 ?        00:01:20 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid      1171     1  0 May03 ?        00:01:04 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      8231  7935  0 07:06 pts/1    00:00:00 grep -i tns
[grid@b600e6tr02 ~]
[grid@b600e6tr02 admin]$ lsnrctl reload LISTENER
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:13:26
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
The command completed successfully
[grid@b600e6tr02 admin]$ lsnrctl reload LISTENER_SCAN1
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:13:35
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
The command completed successfully
[grid@b600e6tr02 admin]$

Session Log to show disabling SCAN Tracing:

[grid@b600e6tr01 ~]$ ps -ef | grep -i tns
grid      5309  4623  0 07:06 pts/1    00:00:00 grep -i tns
grid     18139     1  0 May03 ?        00:01:01 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid     18142     1  0 May03 ?        00:01:14 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
grid     18164     1  0 May03 ?        00:01:18 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
[grid@b600e6tr01 ~]$ lsnrctl
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:07:02
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Welcome to LSNRCTL, type "help" for information.
LSNRCTL> set current_listener LISTENER
Current Listener is LISTENER
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
LISTENER parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
Saved LISTENER configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL> set current_listener LISTENER_SCAN2
Current Listener is LISTENER_SCAN2
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
LISTENER_SCAN2 parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
Saved LISTENER_SCAN2 configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL> set current_listener LISTENER_SCAN3
Current Listener is LISTENER_SCAN3
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
LISTENER_SCAN3 parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
Saved LISTENER_SCAN3 configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL>
[grid@b600e6tr02 ~]$ ps -ef | grep -i tns
grid      1042     1  0 May03 ?        00:01:20 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid      1171     1  0 May03 ?        00:01:04 /u01/home/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      8231  7935  0 07:06 pts/1    00:00:00 grep -i tns
[grid@b600e6tr02 ~]$ lsnrctl
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 04-MAY-2011 07:10:37
Copyright (c) 1991, 2009, Oracle.  All rights reserved.
Welcome to LSNRCTL, type "help" for information.
LSNRCTL> set current_listener LISTENER
Current Listener is LISTENER
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
LISTENER parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
Saved LISTENER configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL> set current_listener LISTENER_SCAN1
Current Listener is LISTENER_SCAN1
LSNRCTL> set trc_level OFF
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
LISTENER_SCAN1 parameter "trc_level" set to off
The command completed successfully
LSNRCTL> save_config
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
Saved LISTENER_SCAN1 configuration parameters.
Listener Parameter File   /u01/home/11.2.0/grid/network/admin/listener.ora
Old Parameter File   /u01/home/11.2.0/grid/network/admin/listener.bak
The command completed successfully
LSNRCTL>

Hope it helps
SRI

In the present demo, i am going to show you how to create and delete a domain in weblogic server.

Assume that you have installed weblogic as specified in the article <Place Holer>. The next step is to create a domain. Domain is the basic administration unit for Oracle Weblogic Server.Everything that we do in weblogic, everything that gets configured such as administration server,the managed servers,connection pools and data sources,cluster machines,applications that get deployed, all of the security settings; all of that information is contained within this administrative unit that we call it domain.

Domain always includes one weblogic server instance,which is configured as being the administration server. So, there is a one-to-one correspondence between domain and an administration server.An administration server only serves one domain, and a domain always has exactly one administration server in it. Then you can optionally have additional managed servers,so you can have zero or more managed servers within your domain.

The central point of control for a domain is the administration server. That, usually what happens when we have a domain is we start the administration server first,the administration server will go out and read its configuration information from a folder in its domain structure called the config folder. So the administration server goes out and  reads the config.xml file,which essentially tells it how to build all of the Mbeans.

The following demo shows you how to create a domain in weblogic server 10.3.4 and delete it.

Please watch this demo and give me your comments.



Thanks

SRI

This article is continuation to the previous article Applying PSU Patch 11.2.0.1.2 To A Two Node RAC

In this article i want to show you the session output for PSU patch application and the session output for  roll backing a PSU patch.

This is the best example to show the hot patching in RAC 11gR2. The patch is applied in a rolling fashion on each node of the cluster. The clusterware automatically relocates the services to other node while patching is running on some node. The end user does not feel any service interruption.

PSU Patch On GRID_HOME(Node1 and Node2)

On b600e6tr01:

[root@b600e6tr01 ~]# cd /u01/home/oracle/product/11.2.0/db_1/patches/psupatch
[root@b600e6tr01 psupatch]# opatch auto . -oh /u01/home/11.2.0/grid
Executing /usr/bin/perl /u01/home/11.2.0/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/home/11.2.0/grid -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-19_11-28-59.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/11.2.0/grid/ocm.rsp
Unable to determine if /u01/home/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u01/home/11.2.0/grid
patch ././9655006  apply successful for home  /u01/home/11.2.0/grid
patch ././9654983  apply successful for home  /u01/home/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@b600e6tr01 psupatch]#

On b600e6tr02:

[root@b600e6tr02 psupatch]# opatch auto . -oh /u01/home/11.2.0/grid
Executing /usr/bin/perl /u01/home/11.2.0/grid/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/home/11.2.0/grid -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-19_12-26-36.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/11.2.0/grid/ocm.rsp
Unable to determine if /u01/home/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u01/home/11.2.0/grid
patch ././9655006  apply successful for home  /u01/home/11.2.0/grid
patch ././9654983  apply successful for home  /u01/home/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@b600e6tr02 psupatch]#

PSU Patch On RDBMS_HOME(Node1 and Node2)
On b600e6tr01 :

[root@b600e6tr01 psupatch]# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto . -oh  /u01/home/oracle/product/11.2.0/db_1
Executing /usr/bin/perl /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/home/oracle/product/11.2.0/db_1 -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-19_13-52-07.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/oracle/product/11.2.0/db_1/ocm.rsp
Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch ././9655006/custom/server/9655006  apply successful for home  /u01/home/oracle/product/11.2.0/db_1
patch ././9654983  apply successful for home  /u01/home/oracle/product/11.2.0/db_1
[root@b600e6tr01 psupatch]#

On b600e6tr02:

[root@b600e6tr02 psupatch]# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto . -oh  /u01/home/oracle/product/11.2.0/db_1
Executing /usr/bin/perl /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/home/oracle/product/11.2.0/db_1 -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-19_13-28-55.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/oracle/product/11.2.0/db_1/ocm.rsp
Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch ././9655006/custom/server/9655006  apply successful for home  /u01/home/oracle/product/11.2.0/db_1
patch ././9654983  apply successful for home  /u01/home/oracle/product/11.2.0/db_1
[root@b600e6tr02 psupatch]#

RollBack PSU Patch On GRID_HOME(Node1 and Node2)
On b600e6tr01:

[oracle@b600e6tr01 ~]$ sudo su -
[root@b600e6tr01 ~]# /u01/home/11.2.0/grid/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -rollback -oh /u01/home/11.2.0/grid
Executing /usr/bin/perl /u01/home/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /u01/home/oracle/product/11.2.0/db_1/patches -patchn psupatch -rollback -oh /u01/home/11.2.0/grid -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2011-05-04_07-27-40.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/11.2.0/grid/ocm.rsp
Unable to determine if /u01/home/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u01/home/11.2.0/grid
patch 9655006  rollback successful for home /u01/home/11.2.0/grid
patch 9654983  rollback successful for home /u01/home/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@b600e6tr01 ~]#

On b600e6tr02:

[oracle@b600e6tr02 ~]$ sudo su -
[root@b600e6tr02 ~]# /u01/home/11.2.0/grid/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -rollback -oh /u01/home/11.2.0/grid
Executing /usr/bin/perl /u01/home/11.2.0/grid/OPatch/crs/patch112.pl -patchdir /u01/home/oracle/product/11.2.0/db_1/patches -patchn psupatch -rollback -oh /u01/home/11.2.0/grid -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/11.2.0/grid/OPatch/crs/../../cfgtoollogs/opatchauto2011-05-04_07-41-37.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/11.2.0/grid/ocm.rsp
Unable to determine if /u01/home/11.2.0/grid is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
Successfully unlock /u01/home/11.2.0/grid
patch 9655006  rollback successful for home /u01/home/11.2.0/grid
patch 9654983  rollback successful for home /u01/home/11.2.0/grid
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-4123: Oracle High Availability Services has been started.
[root@b600e6tr02 ~]#

RollBack PSU Patch ON RDBMS_HOME(Node1 and Node2)
On b600e6tr01 :

[root@b600e6tr01 ~]# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -rollback  -oh  /u01/home/oracle/product/11.2.0/db_1
Executing /usr/bin/perl /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir /u01/home/oracle/product/11.2.0/db_1/patches -patchn psupatch -rollback -oh /u01/home/oracle/product/11.2.0/db_1 -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-05-04_07-54-30.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/oracle/product/11.2.0/db_1/ocm.rsp
Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch 9655006  rollback successful for home /u01/home/oracle/product/11.2.0/db_1
patch 9654983  rollback successful for home /u01/home/oracle/product/11.2.0/db_1
[root@b600e6tr01 ~]#

On b600e6tr02:

[root@b600e6tr02 ~]#
[root@b600e6tr02 ~]# /u01/home/oracle/product/11.2.0/db_1/OPatch/opatch auto /u01/home/oracle/product/11.2.0/db_1/patches/psupatch -rollback  -oh  /u01/home/oracle/product/11.2.0/db_1
Executing /usr/bin/perl /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/patch112.pl -patchdir /u01/home/oracle/product/11.2.0/db_1/patches -patchn psupatch -rollback -oh /u01/home/oracle/product/11.2.0/db_1 -paramfile /u01/home/11.2.0/grid/crs/install/crsconfig_params
opatch auto log file location is /u01/home/oracle/product/11.2.0/db_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-05-04_08-01-19.log
Detected Oracle Clusterware install
Using configuration parameter file: /u01/home/11.2.0/grid/crs/install/crsconfig_params
OPatch  is bundled with OCM, Enter the absolute OCM response file path:
/u01/home/oracle/product/11.2.0/db_1/ocm.rsp
Unable to determine if /u01/home/oracle/product/11.2.0/db_1 is shared oracle home
Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no):yes
patch 9655006  rollback successful for home /u01/home/oracle/product/11.2.0/db_1
patch 9654983  rollback successful for home /u01/home/oracle/product/11.2.0/db_1
[root@b600e6tr02 ~]#

Hope it helps

SRI

This is an informative article to troubleshoot an issue in 11gR2 RAC.
In this article, i want to present one of the issue we identified and solved in 11gR2 RAC. This issue was caused due to one DBA accidentally running 755 permission on all the files in GRID_HOME/bin directory. We observed that the database running on this node is down and it is not communicating with ASM instance running on the same node after running the permissions command.

The error messages when trying to start the database are :

[oracle@b600e6tr01 oracle]$ srvctl start database -d racdb
PRCR-1079 : Failed to start resource ora.racdb.db
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
ORA-01078: failure in processing system parameters
CRS-2674: Start of 'ora.racdb.db' on 'b600e6tr01' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
CRS-2528: Unable to place an instance of 'ora.racdb.db' as all possible servers are occupied by the resource already
[oracle@b600e6tr01 oracle]$

Alert log file showed the following content :

ORA-00210: cannot open the specified control file
ORA-00202: control file: '+FRA/racdb/controlfile/current.256.715627363'
ORA-17503: ksfdopn:2 Failed to open file +FRA/racdb/controlfile/current.256.715627363
ORA-15001: diskgroup "FRA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 32768
Additional information: 10
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+RACDB_DATA/racdb/controlfile/current.256.715627363'
ORA-17503: ksfdopn:2 Failed to open file +RACDB_DATA/racdb/controlfile/current.256.715627363
ORA-15001: diskgroup "RACDB_DATA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 32768
Additional information: 10
Starting background process ASMB

We started comparing the permissions on node 1 with permission on node 2 for all the files in GRID_HOME/bin directory. Then identified that some files in GRID_HOME/bin are missing suid and sgid permission on the executable. The executable to our interest here is "oracle". This executable is missing the suid and sgid permission.

[grid@b600e6tr01 bin]$ ls -lrt oracle
-rwxr-x--x 1 grid dba 184431149 Apr 19 11:54 oracle*
[grid@b600e6tr01 bin]$
[grid@b600e6tr02 bin]$ ls -lrt oracle
-rwsr-s--x 1 grid dba 184431149 Apr 19 12:36 oracle*
[grid@b600e6tr02 bin]$
[grid@b600e6tr01 bin]$ chmod ug+s oracle
[oracle@b600e6tr01 bin]$ srvctl start instance -d racdb -i racdb1
[oracle@b600e6tr01 bin]$ srvctl status database -d racdb
Instance racdb1 is running on node b600e6tr01
Instance racdb2 is running on node b600e6tr02
[oracle@b600e6tr01 bin]$

That solves the issue ...... :-)

Hope it helps
SRI

NOTE : This article is aimed at showing the issues and resolution while applying the PSU patch 11.2.0.1.2 (9655006) to a specific environment and not generalized. Please refer the readme of the PSU patch for detailed procedure to apply PSU patch.

Video Demos are also uploaded to show you the errors and successfull installation of the PSU to 2 node RAC.

1. Record Pre Patch Information.
2. OPatch Utility Information.
3. OCM Configuration.
4. Validation of Oracle Inventory.
5. One-off Patch Conflict Detection and Resolution.
6. Download and Unzip the PSU Patch 9655006
7. Patching GI Home
8. Patching RAC Database Homes
9. Loading Modified SQL Files into the Database.
10. Patch Successful Verification Steps.
11. Issues & Resolutions.

1. Record pre patch information : 

$GRID_ORACLE_HOME/OPatch/opatch lsinventory$GRID_ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'GI PSU'

$ORACLE_HOME/OPatch/opatch lsinventory$ORACLE_HOME/OPatch/opatch lsinventory -bugs_fixed | grep -i 'DATABASE PSU'

SQL> select comp_name,version,status from dba_registry;

2. OPatch Utility Information.

$ORACLE_HOME/OPatch/opatch version -h /u01/home/oracle/product/11.2.0/db_1
$GRID_ORACLE_HOME/OPatch/opatch version -h /u01/home/11.2.0/grid

3. OCM Configuration.Create ocm response file using the following command and provide appropriate values for the prompts.

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp

Verify the created file using,

$GRID_ORACLE_HOME/OPatch/ocm/bin/emocmrsp –verbose ocm.rsp

NOTE: The Opatch utility will prompt for your OCM (Oracle Configuration Manager) response file when it is run. Without which we cant proceed further.

4. Validation of Oracle Inventory.

$GRID_ORACLE_HOME/OPatch/opatch lsinventory -detail -oh /u01/home/11.2.0/grid
$ORACLE_HOME/OPatch/opatch lsinventory -detail –oh /u01/home/oracle/product/11.2.0/db_1

5. One-off Patch Conflict Detection and Resolution.NA

In the present article i am going to describe how to refresh a database. Assume that you have some consistant backups available on TAPE for the database to be refreshed.

Just follow the following sequence of steps to refresh a database ....

$sqlplus '/as sysdba'
SQL> select dbid,created,controlfile_created,last_open_incarnation# from v$database;
-- Note down the dbid and incarnation
SQL> archive log list
-- Note down sequence numbers
-- Now create one user and input some data to user. This is to verify  that after refresh we should not get this data.
SQL> create user askm identified by askm;
SQL> grant connect,resource to askm;
SQL> conn askm/askm
SQL> create table test_table as select * from sys.dba_objects;
-- Verify the tape configuration
$ tdpoconf SHOWENVironment
$ tdpoconf SHOWENVironment -tdpo_optfile=tdpo.opt
-- Connect to RMAN and verify the available backups
$ rman target / catalog rman/rman@rmancat
RMAN> list backup summary;
-- Exit from RMAN session and shutdown the database
SQL> shut immediate

RMAN Refresh Script:

$ rman target / catalog rman/rman@rmancat
RMAN> startup nomount;
RMAN> run {
allocate channel t1 type 'SBT_TAPE' parms="ENV=(TDPO_OPTFILE=tdpo.opt)";
restore controlfile;
alter database mount;
restore database;
recover database noredo;
release channel t1;
}
alter database open resetlogs;

Verification Steps :

$sqlplus '/as sysdba'
SQL> select dbid,created,controlfile_created,last_open_incarnation# from v$database;
-- compare incarnation with earlier value
SQL> archive log list
-- compare sequence numbers,it now starts from 0.
SQL> select username ,created from dba_users where username='ASKM';
-- Should return no rows.

Follow the video demo showing how to restore a database from backups available on tape. This is a private video so, send me a personal request if you need access to this video.



Hope it helps
SRI

In this article i am gooing to describe the issues we faced during 10.2.0.1 oem grid controll on a RHEL 5.4.

Issue 1:

When i start installing the oem grid control after completing all the pre requesites, we got the following error. The cursor hangs there instead of populating OUI window.

When i observe the error message i thought it may the error with LD_LIBRARY_PATH and tried all the possible values for this variable , but no luck. Then after googling i came to know that there is one rpm missing in the OS RHEL 5.4.  Then i installed "libXp-1.0.0-8.1.el5.i386.rpm" and tried. It works .....

RPMs Locations

Issue 2:

error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory

Solution: Install libaio and libaio-devel packages. If packages already installed and error still occurs try execute "ldconfig" as root.

Issue 3:

Issue 4:

Issue 5:

Check complete. The overall result of this check is: Failed <<<<
Solution: Install missing package or set check system parameters (See reason of failure).

Hope it helps

SRI