Thursday, March 29, 2012

Oracle Enterprise Manager 12c Cloud Control: Plugin Deployment

Oracle Enterprise Manager 12c Cloud Control incorporates the use of plugins to extend its functionality for management of the complete cloud stack. Plugins are provided for databases, middleware, virtualization among others.

For a few weeks I had been struggling to deploy Oracle Enterprise Manager 12c agents on IBM AIX 6.1 targets. This became a very lengthy cycle of patching, searching and tweeting in an attempt to get things going. I finally had the agent deployed on the hosts but no targets were being discovered. After coming across MOS Note [ID 1388143.1] I discovered that I needed to update and deploy the database plugin in order to discover the database targets.

To deploy plugins go to Setup->Extensibility->Self Update

From this screen you can see a list of plugins and their status. If there is an updated plugin available it will have a non-zero revision number in the Revision column and the status will be available. The Oracle Database plugin update (20111221) above was downloaded by selecting download from the Actions drop-down menu. Under the Past Activities section you can see the different activities that were performed and the status. After the plugin has been downloaded it first needs to be deployed on the Management Server and then the Management Agents.
Click the Plug-in Icon to go to the plugin deployment screen.


In the Plug-ins screen you can see a list of plugins with the version available and downloaded. You may have to click on the icon beside the Databases folder to expand all database plugins. Clicking on the Oracle Database plugin will show the details in the bottom section of the screen. This displays the plugin versions downloaded, plugin ID, Vendor, version deployed on Management Server and Agents. Here you can see that three versions of the oracle.sysman.db plugin have been downloaded. Only the base version 12.1.0.1.0 has been deployed on Management Server and Agents. To deploy the latest downloaded version click on the Deploy On menu item then select Management Server.


I received the above error when I attempted to deploy the plugin. What? But the plugin was  already downloaded. At this point I was stuck again! After several futile attempts and twitter rants I created a SR for the issue. Surprisingly, Oracle Support was very responsive and proceeded with a series of calls and web conferences to try and help me resolve the issue. However, after many attempts and hours of phone support they were unable to find the cause of the error. We tried deleting and downloading the plugins again both in online and offline mode. Offline mode requires the use of emcli. At this time the recommendation was to apply Bundle Patch 1 for Enterprise Manager 12c or do a complete reinstall. Since this was a test system I was fairly comfortable with either approach. However, I decided to try to use emcli command to deploy the plugin to see if it made any difference.
The emcli command is installed by default in the OMS_HOME/bin directory. You first need to login using
emcli login -username=SYSMAN

After logging I issued the commands below:

-bash-3.2$ ./emcli deploy_plugin_on_server -plugin=oracle.sysman.db:12.1.0.1.0
Enter repository DB sys password:

Performing pre-requisites check... This will take a while.
Pre-requisites check succeeded
Deployment of plug-in on the management servers is in progress
Use "emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db" to track plug-in deployment status.

 To check the status of the deployment I issued the following command:

-bash-3.2$ ./emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db
Plugin Deployment/Undeployment Status

Destination          : OMS - lnxvmoidt02.smrcy.com:4890_Management_Service
Plugin Name          : Oracle Database
Version:             : 12.1.0.1.0
ID:                  : oracle.sysman.db
Content              : Plugin
Action               : Deployment
Status               : Deploying
Steps Info:
---------------------------------------- ------------------------- ------------------------- ----------
Step                                     Start Time                End Time                  Status
---------------------------------------- ------------------------- ------------------------- ----------
Start deployment                         3/27/12 11:06:56 AM CDT   3/27/12 11:06:56 AM CDT   Success

Initialize                               3/27/12 11:07:14 AM CDT   N/A                       Running

---------------------------------------- ------------------------- ------------------------- ----------

Here I could see some definite progress. Things looking good so far. Issue the same command after a few minutes to check the status.

-bash-3.2$ ./emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db
Plugin Deployment/Undeployment Status

Destination          : OMS - lnxvmoidt02.smrcy.com:4890_Management_Service
Plugin Name          : Oracle Database
Version:             : 12.1.0.1.0
ID:                  : oracle.sysman.db
Content              : Plugin
Action               : Deployment
Status               : Success
Steps Info:
---------------------------------------- ------------------------- ------------------------- ----------
Step                                     Start Time                End Time                  Status
---------------------------------------- ------------------------- ------------------------- ----------
Start deployment                         3/27/12 11:06:56 AM CDT   3/27/12 11:06:56 AM CDT   Success

Initialize                               3/27/12 11:07:14 AM CDT   3/27/12 11:07:22 AM CDT   Success

Validate plug-in home                    3/27/12 11:07:24 AM CDT   3/27/12 11:07:24 AM CDT   Success

Check mandatory patches                  3/27/12 11:07:24 AM CDT   3/27/12 11:07:25 AM CDT   Success

Update inventory                         3/27/12 11:07:25 AM CDT   3/27/12 11:07:26 AM CDT   Success

---------------------------------------- ------------------------- ------------------------- ----------


Success! Now that's what I'm talking about. How do I check if the plugin is actually deployed on the server. Oh, there's a command for that too.

-bash-3.2$ ./emcli list_plugins_on_server
OMS name is EMGC_OMS1
Plug-in Name                                                                                        Version [revision]

Oracle MOS (My Oracle Support)                                                                      12.1.0.1.0
Oracle Fusion Middleware                                                                            12.1.0.1.0
Oracle Database                                                                                     12.1.0.1.0[20111221]
Oracle Exadata                                                                                      12.1.0.1.0


Here you can see the highlighted plugin with the latest update. You can also check from the Enterprise Manager console Setup-> Extensibility-> Self-Update


Here you can see the status of the updated plugin is Applied. Go to the Plug-ins section to verify the updated plugin has been deployed to the Management server.


The next step is to deploy the plugin to the targets. In my environment, I wanted to deploy plugin on to AIX targets consisting of Oracle 11gR2 Real Application Clusters databases.

Using emcli again to deploy the plugins on the management agents.

-bash-3.2$ ./emcli deploy_plugin_on_agent -plugin=oracle.sysman.db:12.1.0.1.0 -agent=oradba10t:3872
Agent side plug-in deployment is in progress
Use "emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db" to track plug-in deployment status.
-bash-3.2$ ./emcli deploy_plugin_on_agent -plugin=oracle.sysman.db:12.1.0.1.0 -agent=oradba11t:3872
Agent side plug-in deployment is in progress
Use "emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db" to track plug-in deployment status.

Note that both targets could have been deployed at the same time by separating them with a semicolon ";".
Check on the deployment status
-bash-3.2$ ./emcli get_plugin_deployment_status -plugin_id=oracle.sysman.db
Plugin Deployment/Undeployment Status

Destination          : AGENT - oradba11t:3872
Plugin Name          : Oracle Database
Version:             : 12.1.0.1.0
ID:                  : oracle.sysman.db
Content              : Plugin
Action               : Deployment
Status               : Success
Steps Info:
---------------------------------------- ------------------------- ------------------------- ----------
Step                                     Start Time                End Time                  Status
---------------------------------------- ------------------------- ------------------------- ----------
Start deployment                         3/27/12 11:18:41 AM CDT   3/27/12 11:18:41 AM CDT   Success

Initialize                               3/27/12 11:21:20 AM CDT   3/27/12 11:21:20 AM CDT   Success

Copy archives                            3/27/12 11:21:20 AM CDT   3/27/12 11:21:20 AM CDT   Success

Configure plug-in on management agent    3/27/12 11:21:20 AM CDT   3/27/12 11:23:56 AM CDT   Success

Update inventory                         3/27/12 11:21:20 AM CDT   3/27/12 11:23:56 AM CDT   Success

---------------------------------------- ------------------------- ------------------------- ----------

Destination          : AGENT - oradba11t:3872
Plugin Name          : Oracle Database
Version:             : 12.1.0.1.0
ID:                  : oracle.sysman.db
Content              : Discovery
Action               : Deployment
Status               : Success
Steps Info:
---------------------------------------- ------------------------- ------------------------- ----------
Step                                     Start Time                End Time                  Status
---------------------------------------- ------------------------- ------------------------- ----------
Start deployment                         3/27/12 11:18:41 AM CDT   3/27/12 11:18:41 AM CDT   Success

Initialize                               3/27/12 11:18:53 AM CDT   3/27/12 11:18:53 AM CDT   Success

Copy archives                            3/27/12 11:18:53 AM CDT   3/27/12 11:18:53 AM CDT   Success

Configure plug-in on management agent    3/27/12 11:18:53 AM CDT   3/27/12 11:21:20 AM CDT   Success

Update inventory                         3/27/12 11:18:53 AM CDT   3/27/12 11:21:20 AM CDT   Success

---------------------------------------- ------------------------- ------------------------- ----------



You can also verify the updated plugin has been deployed in Enterprise Manager console. 



Click the Recent Deployments tab in the details section to see a list of deployments to both the management server and agent.


To view a list of all plugins deployed on agents using emcli execute the following: 
(Output truncated for brevity)
-bash-3.2$ ./emcli list_plugins_on_agent
The Agent URL is https://oradba11t:3872/emd/main/ -
Plug-in Name                                                                                        Version

Oracle Database                                                                                     12.1.0.1.0[20111221]
Oracle Home                                                                                         12.1.0.1.0
Oracle Home                                                                                         12.1.0.1.0
Oracle Database                                                                                     12.1.0.1.0
Oracle Home                                                                                         12.1.0.1.0
Oracle Fusion Middleware                                                                            12.1.0.1.0
Oracle Beacon                                                                                       12.1.0.1.0
Management Services and Repository                                                                  12.1.0.1.0
Oracle Home                                                                                         12.1.0.1.0
Oracle Database                                                                                     12.1.0.1.0
Oracle Home                                                                                         12.1.0.1.0
Oracle Fusion Middleware                                                                            12.1.0.1.0
Oracle Database                                                                                     12.1.0.1.0
Oracle Fusion Middleware                                                                            12.1.0.1.0
Oracle Home                                                                                         12.1.0.1.0
Oracle Database                                                                                     12.1.0.1.0
Oracle Home                                                                                         12.1.0.1.0
Oracle Database                                                                                     12.1.0.1.0[20111221]
Oracle Home                                                                                         12.1.0.1.0
Oracle Database                                                                                     12.1.0.1.0[20111221]

After the updated plugins were deployed to the Management Server and Agents I was able to successfully discover all targets on AIX. This leads me to my next post on managing RAC 11gR2 with Enterprise Manager 12c. Stay tuned :) 



Wednesday, March 28, 2012

Managing Oracle 11g RAC and Grid Infrastructure with Oracle Enterprise Manager 12c Cloud Control

This is a place holder for a blog post on using Enterprise Manager 12c Cloud Control for managing Oracle 11gR2 RAC and Grid Infrastructure. Hopefully, I'll get some time to actually do a write-up. For now, enjoy the screenshots.


Friday, March 16, 2012

Enterprise Manger 12c Cloud Control Mobile for iPhone

Oracle Enterprise Manager 12c Cloud Control now has a mobile app that enables you to view and manage incidents and problems generated from targets. No you can't manage or monitor your targets (yet). But you can see errors related to your targets - hosts, databases, clusters, middleware etc. which should make proactive maintenance easier when you're away from your desktop.

Please Note: It's highly recommended that your EM infrastructure be in a secured network behind a firewall. I accessed my EM console via a VPN connection.

Here are some screenshots from the new mobile iPhone app for Oracle Enterprise 12c Cloud Control.
After you do the initial setup, which I admit is a bit wonky, you can then select your EM site. You will see the login screen like the one below.



View all open problems.

Viewing a specific problem.




Listing fo all open incidents


All unacknowledged incidents

Manage problems by updating status, owner, priority, escalation status and add comments.


View incident status

I think this is a good start, but I hope more functionality is added over time to make this an even more useful add-on to an excellent product.

Thursday, March 8, 2012

Grid Infrastructure 11.2.0.3 for AIX 64 includes Cluster Health Monitor

Cluster Health Monitor or CHM (formerly IPD/OS) is a utility used to gather OS metrics such as physical memory, swap, CPU usage etc. It's useful for troubleshooting node reboots or evictions, instance evictions as well as poor performance. CHM gathers data in real time with little resource overhead and stores the metrics which can then be retrieved for root-cause analysis. This is very important as other utilities often cannot provide such detailed metrics if there's a resource issue on a node.
CHM was bundled with  Grid Infrastructure for Linux and Solaris (x86-64) platforms in 11.2.0.2. However, the feature has now been made available in Grid Infrastructure 11.2.0.3.0 for AIX. After upgrading from 11.2.0.2 to 11.2.0.3 a new resource ora.crf can be seen in the CRS stack.

grid@oradba10t[+ASM1]-/home/grid >crsctl stat res -t -init

--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE oradba10t Started
ora.cluster_interconnect.haip
1 ONLINE ONLINE oradba10t
ora.crf
1 ONLINE ONLINE oradba10t
ora.crsd
1 ONLINE ONLINE oradba10t
ora.cssd
1 ONLINE ONLINE oradba10t
ora.cssdmonitor
1 ONLINE ONLINE oradba10t
ora.ctssd
1 ONLINE ONLINE oradba10t OBSERVER
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE oradba10t
ora.evmd
1 ONLINE ONLINE oradba10t
ora.gipcd
1 ONLINE ONLINE oradba10t
ora.gpnpd
1 ONLINE ONLINE oradba10t
ora.mdnsd
1 ONLINE ONLINE oradba10t


There are actually two new processes used for gathering CHM data - oleggerd and sysmond. You can see them highlighted in red below.


grid@oradba10t[+ASM1]-/home/grid >ps -ef|grep d.bin|grep -v grep
root 6881454 1 0 10:05:09 - 2:37 /u01/11.2.0.3/grid/bin/orarootagent.bin
root 7274618 1 0 10:05:13 - 14:34 /u01/11.2.0.3/grid/bin/osysmond.bin
grid 7471320 33882340 0 10:06:44 - 0:00 /u01/11.2.0.3/grid/bin/evmlogger.bin -o /u01/11.2.0.3/grid/evm/log/evmlogger.info -l /u01/11.2.0.3/grid/evm/log/evmlogger.log
oracle 8781958 1 0 10:07:18 - 2:33 /u01/11.2.0.3/grid/bin/oraagent.bin
root 9109720 1 0 10:04:37 - 0:00 /bin/sh /u01/11.2.0.3/grid/bin/ocssd
grid 13041882 1 0 10:03:52 - 2:32 /u01/11.2.0.3/grid/bin/oraagent.bin
root 13697042 1 2 10:02:47 - 7:17 /u01/11.2.0.3/grid/bin/ohasd.bin reboot
grid 15138940 1 0 10:07:10 - 0:00 /u01/11.2.0.3/grid/bin/tnslsnr LISTENER -inherit
root 16515292 1 0 10:06:32 - 9:22 /u01/11.2.0.3/grid/bin/crsd.bin reboot
root 19202134 1 0 10:04:14 - 0:14 /u01/11.2.0.3/grid/bin/cssdagent
grid 21102764 1 1 10:04:04 - 6:57 /u01/11.2.0.3/grid/bin/gipcd.bin
root 21233726 1 0 10:05:10 - 7:48 /u01/11.2.0.3/grid/bin/octssd.bin
root 22544568 1 0 10:04:04 - 0:14 /u01/11.2.0.3/grid/bin/cssdmonitor
grid 24182816 1 0 10:24:55 - 0:00 /u01/11.2.0.3/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
root 24838342 1 1 10:06:48 - 9:49 /u01/11.2.0.3/grid/bin/orarootagent.bin
grid 25624680 1 0 10:04:00 - 0:00 /u01/11.2.0.3/grid/bin/mdnsd.bin
root 26280110 1 0 10:06:17 - 1:35 /u01/11.2.0.3/grid/bin/ologgerd -M -d /u01/11.2.0.3/grid/crf/db/oradba10t
grid 29687946 1 0 10:04:01 - 0:10 /u01/11.2.0.3/grid/bin/gpnpd.bin
grid 30605512 1 0 10:24:54 - 0:04 /u01/11.2.0.3/grid/bin/scriptagent.bin
grid 31457480 1 0 10:24:55 - 0:00 /u01/11.2.0.3/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
grid 32178268 1 0 10:06:48 - 1:25 /u01/11.2.0.3/grid/bin/oraagent.bin
grid 32309280 9109720 2 10:04:37 - 9:10 /u01/11.2.0.3/grid/bin/ocssd.bin
grid 33882340 1 0 10:06:38 - 8:03 /u01/11.2.0.3/grid/bin/evmd.bin
grid 34144466 1 0 10:34:16 - 0:00 /u01/11.2.0.3/grid/bin/tnslsnr LISTENER_SCAN1 -inheri

There is one master logger deamon with replicas running on the other nodes. You can see from the other node that there is another ologgerd process.

root  9437368        1   0 10:30:47      -  0:18 /u01/11.2.0.3/grid/bin/ologgerd -m oradba10t -r -d /u01/11.2.0.3/grid/crf/db/oradba11t

Note that the process on the first node is running with  a -M option (master) while on the second node it's running with a -r option (replica).

The oleggerd process saves it's data to a Berkley DB. The replica processes sends its data to the master process. Sysmond gathers the metrics on all local nodes and sends it to the master ologgerd.

CHM reporting can be done in either interactive or command line mode using the oclumon command.
To see the list of available options type the oclumon command passing the -h parameter:


grid@oradba10t[+ASM1]-/home/grid >oclumon help
For help from command line : oclumon -h
For help in interactive mode : -h
Currently supported verbs are :
showobjects, dumpnodeview, manage, version, debug, quit, exit, and help

The size of the repository depends on the number of days of metrics that is retain. The maximum retention period is 3 days and the default size is 1GB. To get the current size of the repository in seconds run the following command:
grid@oradba10t[+ASM1]-/home/grid >oclumon manage -get repsize
CHM Repository Size = 61646
Done
The repository can be resized to retain 24 hours worth of data using the following command:


grid@oradba10t[+ASM1]-/home/grid >oclumon manage -repos resize 86400
oradba10t --> retention check successful
oradba11t --> retention check successful
New retention is 86400 and will use 1504880640 bytes of disk space
CRS-9115-Cluster Health Monitor repository size change completed on all nodes.
Done


There are two commands used to collect CHM data - diagcollection.pl and oclumon.
For example:

diagcollection.pl --collect --crshome /u01/11.2.0.3/grid --chmos --incidenttime 03/08/201215:00:00 --incidentduration 00:15
The above command collects data form the specified date/time for a 15 minute duration.
Example output:


[root@oradba10t chm]$ s --incidenttime 03/08/201215:00:00 --incidentduration 00:15 <
Production Copyright 2004, 2010, Oracle. All rights reserved
Cluster Ready Services (CRS) diagnostic collection tool
Collecting Cluster Health Monitor (OS) data
Collecting OS logs
[root@oradba10t chm]$ ls -ltr
total 2208
-rw-r--r-- 1 root system 887008 Mar 8 15:26 chmosData_oradba10t_20120308_1526.tar.gz
-rw-r--r-- 1 root system 231485 Mar 8 15:26 osData_oradba10t_20120308_1526.tar.gz


oclumon can be used to dump data in the repository

grid@oradba10t[+ASM1]-/u02/chm >oclumon dumpnodeview -allnodes -v -last "00:00:10" > chmos_data.out

Here's the complete output.

This is just an introduction to what is available with CHM.