Tuesday, July 30, 2013

Oracle Enterprise Manager Cloud Control 12c Shared Agent Install

This post is mostly a reminder to myself on how to install an EM12c shared agent. I found the documentation somewhat lacking, so I hope this will help someone else attempting to perform a shared agent installation.
With a shared agent installation, a master agent is installed on a shared filesystem such as NFS and a shared agent is "installed" on remote hosts. The remote hosts should have read-only access to the share. The documentation lists the following pre-requisites for the mount options on the remote host.
mount -t nfs -o ro :
For example, run the following command:
mount -t nfs -o ro abc.oracle.com:/scratch/masterAgent /scratch/sharedAgent
Since I use a NAS for serving NFS shares in my environment I replaced the master_agent_host_name with the name of the filer and the exported volume.

The shared agent will have its agent instance located on a local file system which contains configuration and host-specific information.
When deploying an agent on NFS, the default location of the central inventory is based on the inventory pointer location (/etc/oraInst.loc on some Unix based OSes). Normally, this would be a similar shared filesystem. If you plan on making the agent a master agent, then the inventory needs to be moved to a local filesystem as per the installation requirements. It may also be of benefit to create a separate inventory for the EM12c agent on local filesystems regardless of the type of installation. This would make it easier to modify in case of any installation issues without affecting other products.
Agent installation can be done using either the GUI or in silent mode. Here I will demonstrate using silent mode agent installations for master and shared agents.

Install Master Agent using agentDeploy.sh

1. Use EMCLI to login and perform a sync operation
[oracle@bumblebee ~]$ emcli login -username=sysman
Enter password :
Login successful
[oracle@bumblebee ~]$ emcli sync
Synchronized successfully
[oracle@bumblebee ~]$ emcli get_supported_platforms
Getting list of platforms ...
Check the logs at /opt/oracle/.emcli/agent.log
About to access self-update code path to retrieve the platforms list..
Getting Platforms list ...
Version =
Platform = Linux x86-64
Version =
Platform = IBM AIX on POWER Systems (64-bit)
Platforms list displayed successfully
2. Get the image for the target platform from the OMS. Target host is AIX in this case. Output has been truncated.
$ emcli get_agentimage -destination=/u02/orastage -platform="IBM AIX on POWER Systems (64-bit)" -version=
Platform:IBM AIX on POWER Systems (64-bit)
=== Partition Detail ===
Space free : 6 GB
Space required : 1 GB
Check the logs at /opt/oracle/.emcli/get_agentimage_2013-07-25_17-25-06-PM.log
Setting property ORACLE_HOME to:/u01/app/oracle/Middleware/oms
calling pulloneoffs with arguments:/u01/app/oracle/Middleware/oms/orasource/Linux/EM/swlib/8/CF6F07082B141DC6E043AD54020A8F7312.
Check this logs for more information: /u01/app/oracle/Middleware/oms/sysman/prov/agentpush/logs
cat /opt/oracle/.emcli/get_agentimage_2013-07-25_17-25-06-PM.log
Platform:IBM AIX on POWER Systems (64-bit)
Getting Agent Image ...
Getting Platforms list ...

Agent Image Name:
Final zip location given by the getFullAgentImage api is:/u02/orastage/
Check this logs for more information: /u01/app/oracle/Middleware/oms/sysman/prov/agentpush/logs
Agent image name:
Plugin image name:
Executing command: /u02/orastage/zip /u02/orastage/ /u02/orastage/ -j
adding: (stored 0%)
Command:/u02/orastage/zip /u02/orastage/ /u02/orastage/ -j completed with status=0
3. Verify that the agent has been extracted to the OMS filesystem
$ ls -ltr /u02/orastage/
-rw-r----- 1 oracle dba 340630674 Jul 25 17:25 /u02/orastage/
4. Copy the agent image to the target host
$ scp /u02/orastage/ lab10d:/u02/orastage
The authenticity of host 'lab10d (' can't be established.
RSA key fingerprint is 46:4d:62:00:1c:43:d8:2c:ce:71:df:f1:1e:f2:df:70.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'lab10d,' (RSA) to the list of known hosts.
oracle@lab10d's password: 100% 325MB 32.5MB/s 00:10
5. Extract the agent binaries
$ unzip
inflating: unzip
inflating: agentDeploy.sh
inflating: agentimage.properties
inflating: agent.rsp
extracting: agentcoreimage.zip

6. Create a response file for the agent (agent.rsp) by supplying values for the below parameters.
NOTE: The agent instance should be place on a local filesystem.
7. Execute the agentDeploy.sh script and pass the INVENTORY_LOCATION parameter to point to a local filesystem
./agentDeploy.sh AGENT_BASE_DIR=/u02/oracle/agent12cR2 RESPONSE_FILE=/u02/orastage/agent.rsp INVENTORY_LOCATION=/home/oracle/oraInventory
-e Validating the OMS_HOST & EM_UPLOAD_PORT
Executing command : /u02/oracle/agent12cR2/core/ -classpath /u02/oracle/agent12cR2/core/ oracle.sysman.agent.installer.AgentInstaller /u02/oracle/agent12cR2/core/ /u02/orastage /u02/oracle/agent12cR2 -prereq
Validating oms host & port with url: http://bumblebee.example.com:4899/empbs/genwallet
Validating oms host & port with url: https://bumblebee.example.com:4899/empbs/genwallet
Return status:3
Unzipping the agentcoreimage.zip to /u02/oracle/agent12cR2 ....
Executing command : /u02/orastage/unzip -o /u02/orastage/ -d /u02/oracle/agent12cR2
Executing command : /u02/oracle/agent12cR2/core/ -d64 -classpath /u02/oracle/agent12cR2/core/ oracle.sysman.agent.installer.AgentInstaller /u02/oracle/agent12cR2/core/ /u02/orastage /u02/oracle/agent12cR2 /home/oracle/agent12. AGENT_BASE_DIR=/u02/oracle/agent12cR2 AGENT_BASE_DIR=/u02/oracle/agent12cR2 RESPONSE_FILE=/u02/orastage/agent.rsp INVENTORY_LOCATION=/home/oracle/oraInventory
Executing agent install prereqs...
Executing command: /u02/oracle/agent12cR2/core/ -ignoreSysPrereqs -prereqchecker -silent -ignoreSysPrereqs -waitForCompletion -prereqlogloc /u02/oracle/agent12cR2/core/ -entryPoint oracle.sysman.top.agent_Complete -detailedExitCodes PREREQ_CONFIG_LOCATION=/u02/oracle/agent12cR2/core/ -J-DORACLE_HOSTNAME=lab10d.example.com INVENTORY_LOCATION=/home/oracle/oraInventory -J-DAGENT_PORT=3872 -J-DAGENT_BASE_DIR=/u02/oracle/agent12cR2
Prereq Logs Location:/u02/oracle/agent12cR2/core/
Agent install prereqs completed successfully
Cloning the agent home...
Executing command: /u02/oracle/agent12cR2/core/ -ignoreSysPrereqs -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/u02/oracle/agent12cR2/core/ -responseFile /u02/orastage/agent.rsp AGENT_BASE_DIR=/u02/oracle/agent12cR2 AGENT_BASE_DIR=/u02/oracle/agent12cR2 RESPONSE_FILE=/u02/orastage/agent.rsp INVENTORY_LOCATION=/home/oracle/oraInventory -noconfig ORACLE_HOME_NAME=agent12c1 INVENTORY_LOCATION=/home/oracle/oraInventory -force b_noUpgrade=true
Clone Action Logs Location:/home/oracle/oraInventory/logs/cloneActions.log
Cloning of agent home completed successfully
Attaching sbin home...
Executing command: /u02/oracle/agent12cR2/core/ -ignoreSysPrereqs -attachHome -waitForCompletion -nowait ORACLE_HOME=/u02/oracle/agent12cR2/sbin ORACLE_HOME_NAME=sbin12c1 INVENTORY_LOCATION=/home/oracle/oraInventory -force
Attach Home Logs Location:/u02/oracle/agent12cR2/core/
Attach home for sbin home completed successfully.
Updating home dependencies...
Executing command: /u02/oracle/agent12cR2/core/ -ignoreSysPrereqs -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/u02/oracle/agent12cR2/sbin:/u02/oracle/agent12cR2/core/,} -invPtrLoc /u02/oracle/agent12cR2/core/ -force
Update Home Dependencies Location:/u02/oracle/agent12cR2/core/
Update home dependency completed successfully.
Performing the agent configuration...
Executing command: /u02/oracle/agent12cR2/core/ ORACLE_HOME=/u02/oracle/agent12cR2/core/ RESPONSE_FILE=/u02/oracle/agent12cR2/core/ ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true
Configuration Log Location:/u02/oracle/agent12cR2/core/
Agent Configuration completed successfully
The following configuration scripts need to be executed as the "root" user.
#Root script to run
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts
Agent Deployment Successful.
Agent deployment log location:
Agent deployment completed successfully.
8. Execute root.sh script

Perform Shared Agent Installation on remote server using AgentNFS.pl

9. Create agent response file
cat /home/oracle/AgentNFS.rsp
10. Execute the AgentNFS.pl script from the target host and pass the location of the inventory by specifying the INVENTORY_LOCATION parameter
/u02/oracle/agent12cR2/core/ /u02/oracle/agent12cR2/core/ -responseFile /home/oracle/AgentNFS.rsp INVENTORY_LOCATION=/home/oracle/oraInventory
Checking whether Agent Instance Home already exists
Checking whether Agent Instance Home directory has write permission
Creating the Agent Instance Home directory : /home/oracle/agent12.
mkdir /home/oracle/agent12.
mkdir /home/oracle/agent12.
mkdir /home/oracle/agent12.
Attaching the composite home to local inventory
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 8192 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-07-26_12-58-26AM. Please wait ...The inventory pointer is located at /u02/oracle/agent12cR2/core/
The inventory is located at /home/oracle/oraInventory
'AttachHome' was successful.
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 8192 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-07-26_12-59-06AM. Please wait ...The inventory pointer is located at /u02/oracle/agent12cR2/core/
The inventory is located at /home/oracle/oraInventory
'AttachHome' was successful.
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 8192 MB Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-07-26_12-59-42AM. Please wait ...The inventory pointer is located at /u02/oracle/agent12cR2/core/
The inventory is located at /home/oracle/oraInventory
'UpdateHomeDeps' was successful.
Configuring the Agent Instance Home
Copying install related scripts
Generating root.sh
Completed the Shared Agent Installation
NOTE: Post install steps to follow :
Please execute /home/oracle/agent12. script as root
11. Execute the root.sh script
$ /home/oracle/agent12.
/etc exist
12. Verify that the agents are up and communicating with the OMS
$AGENT_INSTANCE/bin/emctl status agent
That's all there is to it. You can add any number of shared agents to the master agent as long as they are on the same operating system.

Thursday, July 25, 2013

Installing Oracle EM 12c Agents with EMCLI

Oracle Enterprise Manager Cloud Control 12c introduced several ways to deploy agents to hosts. The Oracle Management Agents are responsible for gathering metrics from targets such as database instances, listeners and the host themselves and sending them to the Oracle Management repository and Server for further processing. It also interacts with targets by executing commands sent via the OMS Console and the EM Command Line Interface (EMCLI).

EM12c release 1 introduced four ways of deploying Management Agents
  1. Add Target Host Wizard - Agents are mass deployed using the Management Console.
  2. AgentPull - Silent install script to automatically download and deploy agents on remote targets using a response file.
  3. agentDeploy - Silent install script to deploy agents using a response file after manually downloading a compressed package of the agent using EMCLI and transferring it to remote targets.
  4. RPM - Use a .rpm package to deploy to targets that support it (Linux x86-64). 
Starting with EM12c release 3 ( agents can now be mass deployed using EMCLI. This enables scripting to ensure rapid and consistent deployment of Management Agents to remote hosts. The third release of EM12c also provides enhanced scripting capability via a Jython interpreter. This is beyond the scope of this article but I hope to discuss this in another post.

In this post I will demonstrate how to deploy Management Agents to a single Linux x86-64 host using EMCLI.
Before installing the agent it should meet all prerequisites as specified in the Oracle Enterprise Manager Cloud Control 12c Installation guide.
To deploy agents we use the EMCLI command with the submit_add_host verb.
emcli submit_add_host
You can find all the options for the command by passing the help verb. 
Note:The output below only shows the mandatory options.
$ emcli help submit_add_host
  emcli submit_add_host
        -host_names="List of host names."
        -platform="Platform id"
        -installation_base_directory="Installation base directory."
        -credential_name="Credential Name"
The syntax requires the host names which is a host or list of hosts to deploy the agents to, platform is the platform id of the host (more about this below), installation base directory is the root path where the agent binaries will be install, credential name is the named credential of the user who will own the binaries on the remote hosts.
To determine the platform id of the target host use emcli with the list_add_host_platforms verb.
emcli list_add_host_platforms -all
Platform ID  Platform Name
226          Linux x86-64
23           Oracle Solaris on SPARC (64-bit) [Agent Software Unavailable]
46           Linux x86 [Agent Software Unavailable]
59           HP-UX PA-RISC (64-bit) [Agent Software Unavailable]
197          HP-UX Itanium [Agent Software Unavailable]
211          IBM S/390 Based Linux (31-bit) [Agent Software Unavailable]
212          IBM AIX on POWER Systems (64-bit) [Agent Software Unavailable]
227          IBM: Linux on POWER Systems [Agent Software Unavailable]
233          Microsoft Windows x64 (64-bit) [Agent Software Unavailable]
267          Oracle Solaris on x86-64 (64-bit) [Agent Software Unavailable]
912          Microsoft Windows (32-bit) [Agent Software Unavailable]
209          IBM: Linux on System z [Agent Software Unavailable]
The output lists the platform id and name of the platforms and indicates whether or not the agent software is available on the OMS for deployment. If the agent software is unavailable you first need to download it using Self-Update in online or offline mode.

Now that we know the platform_id and it's assumed that the host names and base directory is also known, we can deploy the software. It may be a good time to look at all the available options to see if you need to further customize your deployment including changing the agent port, properties or specify whether the deployment is for a fresh install, clone or shared install.
I like to also check which named credentials are available. You can do so using the list_named_credentials verb.
$ emcli list_named_credentials
Credential Name Credential Owner Authenticating Target Type Cred Type Name Target Name Target Username
NC_EM_SFM_ADMIN SYSMAN host HostCreds em_smf_admin
NC_ORACLE SYSMAN host HostCreds oracle
ORACLE_HOST_USER SYSMAN host HostCreds oem5.homenet oracle
With all the mandatory information available the agent can now be deployed.
$ emcli submit_add_host -host_names=db12c01.homenet -platform=226 -installation_base_dir=/u02/oracle/agent12c -credential_name=ORACLE_HOST_USER -session_name="DB12c Add Host"
An Add Host session with the name "DB12c Add Host" has been submitted.
You can track the progress of this session using the command "emcli get_add_host_status -session_name=DB12c Add Host"
The example above will submit a job to deploy an agent to a 64-bit Linux host (platform id 226) to a host name db12c01.homenet. The named credential for the host is ORACLE_HOST_USER. A name for the session was also provided to make it easily identified. Once the job has been submitted you can track its status.
$ emcli get_add_host_status -session_name="DB12c Add Host"
OverAll Status : Initialization in Progress
Host             Platform Name  Initialization  Remote Prerequisite  Agent Deployment  Error
db12c01.homenet  Linux x86-64   In Progress     Not Run              Not Run
All the normal stages of agent deployment will be updated. You can issue the command after a couple minutes to track its status. There's also a -wait_for_completion option that will run the Add Host operation synchronously. However I haven't tested this yet.
$ emcli get_add_host_status -session_name="DB12c Add Host"
OverAll Status : Agent Deployment in Progress
Host             Platform Name  Initialization  Remote Prerequisite  Agent Deployment  Error
db12c01.homenet  Linux x86-64   Succeeded       Succeeded            In Progress

$ emcli get_add_host_status -session_name="DB12c Add Host"
OverAll Status : Agent Deployment Succeeded
Host             Platform Name  Initialization  Remote Prerequisite  Agent Deployment  Error
db12c01.homenet  Linux x86-64   Succeeded       Succeeded            Succeeded
After installation has completed successfully run the $AGENT_HOME/root.sh script if the  -postinstallation_script_run_as_root option was not specified.

In summary we have seen how simple and quick we can deploy agents using using the new EMCLI verb. This should take away some of the drudgery from having to click through multiple screens and lay the framework for creating an automated and repeatable method for managing and monitoring hosts and their dependencies using Oracle Enterprise Manager Cloud Control 12c.

Thursday, July 11, 2013

Oracle Enterprise Manager Cloud Control 12c Release 3 Upgrade

The third release of Oracle Enterprise Manager Cloud Control is now available. It includes numerous enhancements such as updated user interface, improved EMCLI with scripting options, Administration Group enhancements, OMS Disaster improvements among others. For the complete list see the New Features page.
For users still running on pre-12c or, it is a highly recommended upgrade. If you are running EM you may see some benefits of upgrading but I personally wouldn't rush off unless there were some compelling features/fixes that would be of benefit. That said, if you do decide on going full speed with the new release, here is a step-by-step document on how I upgraded a POC environment with a single OMS and repository on the same server from with PSU3 to
The official Oracle documentation lists the detailed steps for the upgrade here.

The pre-requisite steps needed for the upgrade are listed below:

0. Allocate additional space (14GB on Linux x86-64) for new OMS home (out-of-place upgrade)
1. Copy EMKey to repository
2. Unset http_proxy environment variable
3. Check repository database pre-requisites. Run emprereqkit.
4. Stop existing OMS
5. Backup repository, OMS, inventory and software library.

Note: The upgrade is an out-of-place upgrade for the OMS even in a One-System Upgrade. A new OMS home will be created while the schema objects will be upgraded in the repository database.

Download the installation media from OTN or Oracle e-delivery (Software Delivery Cloud) and extract and stage it in a location accessible to the server.
Execute runInstaller from the installation directory. You will be prompted for the usual email address if you need Oracle to provide you with information about updates. Also if you have a valid CSI number you can opt to provide it. Click next when done.

The next screen will execute pre-requisite checks. If any should fail then fix it before continuing.

In this case I chose the second option to upgrade using One-System Upgrade. Select the radio button next to the home to be upgraded. Click next.

Since this is an out-of-place upgrade you need to enter the path with sufficient space where the new OMS home will be installed.

Provide the password for the SYS and SYSMAN user used for the existing management repository database. Also check the box to confirm that you have backed up your management repository. For a production environment I would do another backup to be on the safe side
 Repository pre-req checks may fail if not already addressed. You can opt to fix the problem yourself by clicking No or allow OUI to fix them by clicking Yes.

You may see warnings related to database parameters and configuration settings which aren't required for the installation but necessary for optimal operation of OEM. Click OK to fix the issues manually and re-execute the checks or click Cancel to continue.

All existing plug-ins will be upgraded automatically. Click next to continue.

If you have downloaded additional plug-ins you can check the boxes next to the corresponding plug-in. Click next to continue.
Provide the name of the Admin Server, port, weblogic username and password. Also enter the path for the new OEM instance.
Note that place the instance outside of the Middleware home by default.

Review the summary of the upgrade and click Install to begin the upgrade.

The upgrade begins and lists the steps and overall progress.
All the steps should complete successfully for the upgrade to be successful. If any steps failed click on the "View Log" link next to the step to see the details on the failure and fix it if possible.

Run the allroot.sh script as the root user to complete the upgrade.

Review the summary of the installation and make a note of any changes.

Check the status of the OMS using emctl status oms -details.
Once the OMS has been upgraded you also need to upgrade the central agent installed on the OMS server and other agents in the environment. This will be covered in a separate post.

The environment used for this test consisted of a single virtual server for the OMS and repository with 8GB physical memory and 2 cores. There were 80 targets and repository was 8GB. Total time for the upgrade was 2 hrs.