facebook twitter youtube facebook facebook facebook

E-Mail : info@askmlabs.com

Phone : +1.215.353.8306

Latest Post

Exalogic Apps-to-Disk Monitoring Setup Using OEM12c

Written By askMLabs on Sunday, December 25, 2016 | 10:35 PM

Exalogic machine is one of the Engineered system which provides excellent consolidation platform for hosting wide range of application workloads. OEM 12c provides the capability of monitoring the exalogic machine as a system there by enabling us to monitor the applications , storage, softwares and different hardware components of it.

In this article, We will see the detailed steps to setup monitoring for Exalogic machine in Oracle Enterprise Manager 12c. Usually there are two types of exalogic implementations, Virtual and Physical. The present procedure to discover exalogic machine is valid for the virtual exalogic configuration with Exalogic Elastic Cloud Software (EECS) version 2.0.6.0.0 or later and Oracle Enterprise Manager 12.1.0.3 or later.

  1. Verify the correct software versions and Plug-ins required
  2. Install OEM agent onto OVMM and EMOC Server.
  3. Install the required plug-ins 
  4. Configuring agents installed step 2 for exalogic discovery.
  5. Discovering Oracle VM Manager
  6. Discovering Oracle ZFS Storage Appliance
  7. Discovering Exalogic Virtual Machine
  8. Any Optional Configurations.

1. Verify the correct software versions and Plug-ins required The Exalogic version can be found using the EMOC console. EMOC is a default setup as part of Exalogic implementation. The default URL for EMOC is  https://<emoc server>:9443/emoc/ .  The following screenshot will help you to identify the exalogic version. 



Following list of plug-ins are required to be installed on OMS and agents installed on OVMM and EMOC servers. If you have both OVMM and EMOC installed on a single server, you need to install plug-ins on only one agent.
  • Oracle Virtualization Plug-in
  • Oracle ZFS Storage Appliance Plug-in
  • Fusion Middleware Plug-in
Detailed screenshots are provided in step 3 on how to install these plug-ins.


2. Install OEM agent onto OVMM and EMOC Server(s).
In any exalogic implementation, the OVMM and EMOC can be either installed in a single vServer or it can be installed in two different vServers. In the present setup, we assume that both OVMM and EMOC are installed on a single vServer. Installing agent on a vServer is a direct process. You can use any of the agent installation methods to install agent on this vServer. If you need more help on installing agent, please refer my other blog posts.



3. Install the required plug-ins In step-1 we listed the plug-ins required for the discovery of the each component in exalogic. Now we need to install plug-ins on OMS and agent installed in step-2 above. Please make sure you have updated the latest plug-ins before deployment. Following screenshots will help you to understand plug-in updates and deploying these plug-in onto OMS and agent.


Deploying Virtualization Plug-in : Here I am deploying the plug-in onto agent. This plug-in is already installed on OMS.








Deploying the ZFS Plug-in : Here you can see the plug-in is updated to the latest version and then deployed to the OMS and agent.












4. Configuring agents installed in step-2 for exalogic discovery
We need to configure the agent installed in step-2 for the the exalogic discovery. This configuration includes importing certificates. 
  • Need to import the Oracle Enterprise Manager Ops Center Certificate to the agent keystore
Login to EMOC vServer and execute the following command as root user to export the certificate.

$JAVA_HOME/jre/bin/keytool -export -alias cacao_agent -file oc.crt -keystore truststore -storepass trustpass

[root@elxxxx-vm ~]# JAVA_HOME=/u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.4.0/jdk
[root@elxxxx-vm ~]# export PATH=$JAVA_HOME/bin:$PATH
[root@elxxxx-vm ~]# which java
/u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.4.0/jdk/bin/java
[root@elxxxx-vm ~]# cd /etc/opt/sun/cacao2/instances/oem-ec/security/jsse
[root@elxxxx-vm jsse]# ls -lrt
total 16
-rw-r--r-- 1 root root 2064 Jun 23  2015 keystore
-rw-r--r-- 1 root root 1014 Jun 23  2015 agent.cert
-rw-r--r-- 1 root root 1658 Jun 23  2015 truststore
-rw-r--r-- 1 root root 1658 Jun 23  2015 truststore_gf
[root@elxxxx-vm jsse]# $JAVA_HOME/jre/bin/keytool -list -keystore /u01/app/oracle/product/12.1.0/agent12c/agent_inst/sysman/config/montrust/AgentTrust.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 9 entries
..
...

[root@elxxxx-vm jsse]# pwd
/etc/opt/sun/cacao2/instances/oem-ec/security/jsse
[root@elxxxx-vm jsse]# $JAVA_HOME/jre/bin/keytool -export -alias cacao_agent -file oc.crt -keystore truststore -storepass trustpass
Certificate stored in file <oc.crt>
[root@elxxxx-vm jsse]# ls -lrt
total 20
-rw-r--r-- 1 root root 2064 Jun 23  2015 keystore
-rw-r--r-- 1 root root 1014 Jun 23  2015 agent.cert
-rw-r--r-- 1 root root 1658 Jun 23  2015 truststore
-rw-r--r-- 1 root root 1658 Jun 23  2015 truststore_gf
-rw-r--r-- 1 root root  706 Feb 18 10:51 oc.crt

Import the certificate using the following command as root user ...

$JAVA_HOME/jre/bin/keytool -import -keystore /u01/app/oracle/product/12.1.0/agent12c/agent_inst/sysman/config/montrust/AgentTrust.jks -alias wlscertgencab -file /etc/opt/sun/cacao2/instances/oem-ec/security/jsse/oc.crt

[root@elxxxx-vm jsse]# $JAVA_HOME/jre/bin/keytool -import -keystore /u01/app/oracle/product/12.1.0/agent12c/agent_inst/sysman/config/montrust/AgentTrust.jks -alias wlscertgencab -file /etc/opt/sun/cacao2/instances/oem-ec/security/jsse/oc.crt
Enter keystore password:
Owner: CN=elxxxx-vm_oem-ec_agent
Issuer: CN=elxxxx-vm_oem-ec_agent
Serial number: 11cb1895
Valid from: Wed Dec 31 19:00:00 EST 1969 until: Sat Mar 10 10:47:14 EST 2035
Certificate fingerprints:
         MD5:  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
         SHA1: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
         Signature algorithm name: MD5withRSA
         Version: 3
Trust this certificate? [no]:  yes
Certificate was added to keystore
[root@elxxxx-vm jsse]# $JAVA_HOME/jre/bin/keytool -list -keystore /u01/app/oracle/product/12.1.0/agent12c/agent_inst/sysman/config/montrust/AgentTrust.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 10 entries
...
...
...
Restart the agent before the next certificate export and import.

  • Need to import the Oracle VM Manager certificate into the agent
As the root user on the Oracle VM Manager host, export the certificate from Oracle VM Manager:
/u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.4.0/jdk/bin/keytool -keystore ./ovmmCoreTcps.ks -exportcert -alias ovmm -file ovmm.cr

[root@elxxxx-vm jsse]# cd /u01/app/oracle/ovm-manager-3/
[root@elxxxx-vm ovm-manager-3]# ls -lrt
total 32
drwxr-xr-x 5 oracle dba  4096 Feb 17  2014 ovm_shell
drwxr-xr-x 3 oracle dba  4096 Jun 22  2015 machine1
-rw-r----- 1 oracle dba    92 Jun 22  2015 ovm_console_configuration.properties
drwxr-xr-x 2 root   root 4096 Jun 22  2015 bin
-rw-r--r-- 1 root   root 1369 Jun 22  2015 ovmmCoreTcps.ks
drwxr-xr-x 7 oracle dba  4096 Jun 23  2015 ovm_upgrade
drwxr-xr-x 4 oracle dba  4096 Jun 23  2015 weblogic
drwxr-xr-x 9 oracle dba  4096 Jun 24  2015 ovm_cli
[root@elxxxx-vm ovm-manager-3]# ls -lrt /u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.3.0/jdk/bin/keytool
[root@elxxxx-vm ovm-manager-3]# ls -lrt /u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.4.0/jdk/bin/keytool
-rwxr-xr-x 1 oracle dba 51979 Mar 12  2013 /u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.4.0/jdk/bin/keytool
[root@elxxxx-vm ovm-manager-3]# /u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.4.0/jdk/bin/keytool -keystore ./ovmmCoreTcps.ks -exportcert -alias ovmm -file ovmm.cr
Enter keystore password:
*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************
Certificate stored in file <ovmm.cr>
[root@elxxxx-vm ovm-manager-3]# ls -lrt
total 36
drwxr-xr-x 5 oracle dba  4096 Feb 17  2014 ovm_shell
drwxr-xr-x 3 oracle dba  4096 Jun 22  2015 machine1
-rw-r----- 1 oracle dba    92 Jun 22  2015 ovm_console_configuration.properties
drwxr-xr-x 2 root   root 4096 Jun 22  2015 bin
-rw-r--r-- 1 root   root 1369 Jun 22  2015 ovmmCoreTcps.ks
drwxr-xr-x 7 oracle dba  4096 Jun 23  2015 ovm_upgrade
drwxr-xr-x 4 oracle dba  4096 Jun 23  2015 weblogic
drwxr-xr-x 9 oracle dba  4096 Jun 24  2015 ovm_cli
-rw-r--r-- 1 root   root  601 Feb 18 10:55 ovmm.cr
[root@elxxxx-vm ovm-manager-3]# echo $JAVA_HOME
/u01/app/oracle/product/12.1.0/agent12c/core/12.1.0.4.0/jdk
[root@elxxxx-vm ovm-manager-3]# $JAVA_HOME/jre/bin/keytool -list -keystore /u01/app/oracle/ovm-manager-3/ovmmCoreTcps.ks
Enter keystore password:
*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
ovmm, Jun 22, 2015, PrivateKeyEntry,
Certificate fingerprint (MD5): XXXXXXXXXXXXXXXXXXXXXXX
[root@elxxxx-vm ovm-manager-3]#

As the EM Agent install user on the Enterprise Manager Agent host monitoring the Oracle VM Manager, to import the certificate into the Agent's truststore, the EMCTL secure command needs to be executed from the host on which the Agent is located:

$AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc /u01/app/oracle/ovm-manager-3/ovmm.cr -alias ovmm

[oracle@elxxxx-vm ~]$ $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks -trust_certs_loc /u01/app/oracle/ovm-manager-3/ovmm.cr -alias ovmm
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Password:
Message   :   Certificate was added to keystore
ExitStatus: SUCCESS
[oracle@elxxxx-vm ~]$
[oracle@elxxxx-vm ~]$ $JAVA_HOME/jre/bin/keytool -list -keystore /u01/app/oracle/product/12.1.0/agent12c/agent_inst/sysman/config/montrust/AgentTrust.jks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 11 entries
...
..
Now restart the agent after successful import of the certificates.



5. Discovering the Oracle VM Manager
Before proceeding with the discovery, make sure you have deployed the Oracle Virtualization Plug-in as described in step-2.
Following screenshots will help you to discover the Oracle VM Manager.







Once the VM Manager is discovered , very important step is to configure it for read-only. Otherwise Oracle considers this as unsupported configuration.
Perform the following steps to configure Oracle VM Manager discovered in OEM as read-only.


Login to the OVMM vServer as root user to get the following details ...

[root@elxxxx-vm ovm-manager-3]# cat /var/exalogic/info/em-context.info
ExalogicID=Oracle Exalogic X2-2 AK00000000
[root@elxxxx-vm ovm-manager-3]# cd /u01/app/oracle/ovm-manager-3/ovm_shell
[root@elxxxx-vm ovm_shell]# /opt/sun/cacao2/bin/cacaoadm get-param jmxmp-connector-port -i oem-ec
jmxmp-connector-port=11172
[root@elxxxx-vm ovm_shell]#


Login to the OVMM vServer as oracle user and perform the following steps .....

[oracle@elxxxx-vm ~]$ $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.
[oracle@elxxxx-vm ~]$ $AGENT_HOME/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
Starting agent ......... started.
[oracle@elxxxx-vm ~]$ cd /u01/app/oracle/ovm-manager-3/ovm_shell
[oracle@elxxxx-vm ovm_shell]$ sh ovm_shell.sh  --url=tcp://localhost:54321 --username=admin --password=**********
OVM Shell: 3.2.8.733 Interactive Mode
>>> from com.oracle.ovm.mgr.api import *
>>> from com.oracle.ovm.mgr.api.event import *
>>> from com.oracle.ovm.mgr.api.virtual import *
>>> from com.oracle.ovm.mgr.api.physical import *
>>> from com.oracle.ovm.mgr.api.physical.network import *
>>> from com.oracle.ovm.mgr.api.physical.storage import *
---
>>> ovm = OvmClient.getOvmManager ()
>>> f = ovm.getFoundryContext ()
>>> j = ovm.createJob ( 'Setting AK000000000' );
>>> j.begin ();
>>> f.setAsset ( "EXALOGIC_ID", "AK000000000");
'AK000000000'
>>> j.commit ();
>>>
[oracle@elxxxx-vm ovm_shell]$

Press CTRL-D to complete the sequence.


Validate the VM Manager is configured with read-only.


6. Discovering Oracle ZFS Storage Appliance
Before proceeding with the discovery, make sure you have deployed the Oracle ZFS storage Appliance Plug-in as described in step-2.
One important configuration step, we need to do before the discovery of ZFS appliance is that ,we need to configure the ZFS Storage Appliance for OEM monitoring.

Please follow the screenshot to complete the configuration steps. Access the ZFS storage appliance using "https://<IP of  ZFS Appliance>:215/"









Once the configuration is complete, The ZFS Storage Appliance is now ready for discovery.

Follow the screenshots to complete the discovery of ZFS Storage Appliance.









Repeat the same step-6 for all the ZFS appliances in the exalogic machine. For a complete single rack exalogic machine, you will have 2 ZFS storage appliances.

7. Discovering Exalogic Virtual Machine
The final step is to discover the Exalogic machine. Follow the following screenshots to complete the discovery of the exalogic machine.








8. Optional Configurations 
The following configurations are optional and it is not part of the actual discovery process of Exalogic machine.
  • You need to deploy the necessary plug-ins required for the monitoring capabilities of the softwares that you will be installing on vServers like Weblogic Servers, Webcenter , OTDs , Peoplesoft etc ...
  • You can install "Oracle Engineered System Healthchecks" plug-in and configure the targets to perform the exalogic health checks.

Please leave your comments and suggestions if this article helps you.

Thanks
SRI

Migrating Level-4 OEM to New Datacenter With Zero Downtime

Written By askMLabs on Thursday, December 15, 2016 | 10:01 PM

I am working on a project to migrate a level-4 OEM(Oracle Enterprise Manager) running on Oracle Database Appliances(ODA) from one Datacenter to another Datacenter. I am also planning to write a white paper on this and will  submit to Oracle.

I proposed the following action plan to migrate these ODAs to new datacenter with zero downtime. 

I am going to explain the outlined procedure to perform this activity in the present article, but will also write some more articles on the same topic with detailed step by step procedure.

In order to understand this article, Please note the following keywords.
  • OEM - Oracle Enterprise Manager
  • ODA - Oracle Database Appliance
  • OMR - Oracle Management Repository ( Oracle Database )
  • OMS - Oracle Management Service
  • OMA - Oracle Management Agent
  • PDC - Primary Datacenter
  • SDC - Second Datacenter or Standby Datacenter
  • TDC - Third Datacenter
  • DNS - Domain Name Service

Present Setup : 
  • Two primary ODAs with 12.1.0.4 OEM 12c installed - At PDC
  • Two secondary ODAs with 12.1.0.4 OEM 12c (Standby with ASM replication from primary for OMS binaries) - At SDC
  • Database and Grid are on 11.2.0.3
  • Oracle Database is two instance RAC both on primary and secondary
  • ODAs are with old control version (oakcli)
Targetted Setup : 
  • Two primary ODAs with 12.1.0.4 OEM 12c - At PDC 
  • Two secondary ODAs with 12.1.0.4 OEM 12c (Standby with ASM replication from primary for OMS binaries) - At TDC
  • Database with 12.1.0.2 and 11.2.0.3
  • Grid with 12.1.0.2 ( without flex ) 
  • Oracle Database is two instance RAC both on primary and secondary
  • ODAs with the latest control version ( oakcli )

Slide-1

  • Primary ODAs at PDC are hosting OEM12c with 12.1.0.4 version. The database is RAC and it has two OMS services  running.
  • Secondary ODAs at SDC are standby for Primary and it hosts the same versions as primary.
  • The primary to secondary OMS replication is setup using ASM file system. So it is ASM replication from primary to secondary. ( Note that it is not standby weblogic setup ) . Please read my other article on how to convert Standby weblogic Server setup to ASM replicaiton.

Slide-2

  • The secondary datacenter SDC will be replaced with a third datacenter TDC. So we got new ODAs and they are stacked and installed at new datacenter. These ODAs are called toda01 and toda02. The next step is to create a new standby database at TDC with primary database at PDC.
  • Grid software will be 12.1.0.2.
  • Database software will be 11.2.0.3 and 12.1.0.2 ( Need to install both the binaries, so that the latest version can be used for database upgrade)
Slide-3
  • In this step, we are going to create a new standby database TOEMDB for the primary database POEMDB. At this point of time, we have  two standby databases up and running. We need to make sure that the archive logs are shipped to both the standby locations. Once the second standby database is created, we need to prepare the toda's for the OMS creation.
Slide-4
  • We need to prepare the ODAs at TDC for ASM replication. The procedure and detailed steps will be posted in a different article. 
  • Disable the ASM replication from PDC to SDC.
  • Enable the ASM replication from PDC to TDC.
  • Note that, at this point, we still have two standby databases, but only one ASM replication from PDC to TDC. The DNS is still pointing to SDC.
Slide-5
  • Now the standby setup for database at SDC can be disabled. Need to remove all the configurations for SDC standby database at primary database running at PDC.
  • Now primary database is at PDC and One Standby Database is at TDC. Primary OMS is at PDC and Standby OMS is at TDC.
  • DNS ( loadbalancer ) updates need to be done to reflect the new ODAs at TDC. All the references for ODAs at SDC should be removed from DNS and loadbalancer.
Slide-6
  • We can now switch off or decommission the ODAs at SDC as per your enterprise standards.
  • At this point, we have PDC as primary OEM and TDC with the standby OEM setup.
  • Database at PDC is with 11.2.0.3 and at TDC with 11.2.0.3. Note that we still have binaries 12.1.0.2 created for future database upgrade.
  • Grid at PDC is 11.2.0.3 and Grid at TDC is 12.1.0.2.
  • ODA control software oakcli is with old version at PDC and at latest version at TDC.

Slide-7
  • The OEM setup looks as shown above after completely replacing the SDC datacenter ODAs with TDC ODAs.

Now we are going to repeat the same steps for replacing the ODAs at PDC. For this replacement, we are using the same datacenter PDC. So in the same datacenter PDC, we will replace the old ODAs with the new ODAs. For this to perform ,we need to switch the roles of the present OEM.

Slide-8
  • Rack and stack the ODAs at FDC.
  • Database with version 11.2.0.3 and 12.1.0.2
  • Grid wit h12.1.0.2 version.
Slide-9
  • Swith the roles of the OEMs at PDC and TDC.
  • Once the OEM at PDC is standby, we can repeat the same steps from 1-7.
Slide-10
Slide-11

Slide-12

Slide-13


Slide-14
  • When the ODAs at PDC are replaced completely with new ODAs, we can switch back the roles. So that, ODAs at PDC will run the primary OEM and PDAs at TDC will run the standby OEM.



The Complete procedure is shown in the following flow diagram for easy reference to understand what we are doing at each stage.




Hope this information helps you to understand the migration of  Level-4 OEM with zero downtime.

Please comment or reply to me if you have any questions or doubts.

Thanks
SRI
 
Support :
Copyright © 2013. askMLabs - All Rights Reserved
Proudly powered by Blogger