Collaborate 13 - Denver

The 20th anniversary of the Collaborate IOUG Forum will be held this year in Denver Colorado from April 7 to 11. This will be my second year attending and presenting at Collaborate. I am really looking forward to networking with peers and friends - especially those in my social media community who I interact with on a daily basis.
This year I will be particularly busy with presenting two sessions and doing hosting duties for a couple of RAC SIG sessions. You can catch me at one of the sessions below.

Date	Start Time	End Time	Session ID	Session Details	Product Line	Track
4/8/13	3:45 PM	4:45 PM	539	RAC Experts Panel	Oracle Database, Real Application Clusters	Database
4/9/13	10:45 AM	11:45 AM	154	Oracle Database Appliance Customer Panel	Oracle Database, Real Application Clusters	Engineered Systems
4/9/13	4:30 PM	5:30 PM	326	Configuring Oracle Enterprise Manager Cloud Control 12c for High Availability	Oracle Enterprise Manager	Cloud Computing,Manageability
4/10/13	3:00 PM	4:00 PM	864	Upgrading from Oracle Enterprise Manager 10g to Cloud Control 12c with Zero Downtime	Oracle Enterprise Manager	Cloud Computing,Manageability

Getting NFS to play nice with ASM

Recently I was running some tests in my lab environment consisting of two Oracle 11.2.0.3 databases on separate RHEL 6.2 VMs. The databases were configured in a Data Guard configuration and used ASM as storage. However, I ran into an issue where my standby database was becoming out of sync with the primary database because the standby was out of disk space. At the time I couldn't request additional storage for my standby (that's what happens when you work at 1:00 am while everyone works normal hours) and I needed to complete my testing. The standby server had some NFS mounted shares that had plenty of space however. This is where social interaction is of benefit because I can recall having several discussions with fellow colleagues on twitter about choosing to use NFS as ASM devices and under what conditions one would choose such a route. Yury also wrote up a blog post about it.

This seemed like a good opportunity to test it out. Only this time I will be mixing ASM block devices on a cooked filesystem along with ASM raw devices (these were actually created on top of a NFS datastore but presented to the guests as LUNs).

Please note this is for educational and testing purposes only. I wouldn't recommend doing this in your production environment. Use at your own risk.

Mount the NFS volume with the correct mount options

[root@el6 ~]#  mount -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600 192.168.1.101:/export/oracle/dnfs /oradata

Check the existing ASM devices

[grid@el6 ~]$ asmcmd lsdsk -G FLASH
Path
/dev/oracleasm/disks/ASMDISK2

Create a new ASM device on the NFS mounted file system

[grid@el6 ~]$ dd if=/dev/zero of=/oradata/nfs_asm-disk1 bs=1K count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 547.629 s, 1.9 MB/s

Check the value of the ASM_DISKSTRING initialization parameter. This tells ASM where to look for ASM devices.

[grid@el6 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Thu Jan 17 02:37:46 2013
Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Automatic Storage Management option
SQL> show parameter asm_diskstring
NAME     TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring     string
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*','/oradata/nfs_asm-disk1';
System altered.
SQL> alter diskgroup flash add disk '/oradata/nfs_asm-disk1';
Diskgroup altered.

List the ASM devices in the diskgroup. Here you'll see both the LUN and new device on the NFS volume in the same diskgroup.

[grid@el6 ~]$ asmcmd lsdsk -G FLASH
Path
/dev/oracleasm/disks/ASMDISK2
/oradata/nfs_asm-disk1

Note: If you see errors similar to the following in your alert log, you may need to change the ownership and permissions on your NFS mounted ASM devices so that they are owned by the "ASM" user.

SUCCESS: disk FLASH_0001 (1.3915921253) added to diskgroup FLASH

Thu Jan 17 02:47:45 2013

Direct NFS: write FAILED 13

Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:

ORA-17500: ODM err:KGNFS WRITE FAIL:Permission denied

WARNING: Write Failed. group:1 disk:1 AU:4 offset:114688 size:16384

Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:

ORA-15080: synchronous I/O operation to a disk failed

WARNING: failed to write mirror side 1 of virtual extent 1 logical extent 0 of file 256 in group 1 on disk 1 allocation unit 4 

Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:

ORA-00206: error in writing (block 15, # blocks 1) of control file

ORA-00202: control file: '+FLASH/orcl1/controlfile/current.256.768325217'

ORA-15081: failed to submit an I/O operation to a disk

ORA-15081: failed to submit an I/O operation to a disk

minact-scn: useg scan erroring out with error e:221

Now your ASM will be playing nicely together with both NFS and LUNs.

Oracle Enterprise Manager 12.1.0.2.0 PSU 1

Thanks to @BrianPardy for tweeting the availability of the first patch set update for Oracle Enterprise Manager 12.1.0.2.0. Brian also blogged about it.
The PSU Readme lists quite a number of bugs fixed in the PSU. Surprisingly the application of the patch is quite easy.

I applied it to my 12.1.0.2.0 lab environment without any issues. Just to test for real application on test and production servers.

• Download the patch 14840279  from My Oracle Support to your OMS server.
• Extract the zip file
• Make sure opatch is in your PATH.

[oracle@oem3 14840279]$ which opatch
/usr/bin/which: no opatch in (/u01/app/oracle/Middleware/oms/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/oracle/bin)
[oracle@oem3 14840279]$ export PATH=$OMS_HOME/OPatch:$PATH
[oracle@oem3 14840279]$ which opatch
/u01/app/oracle/Middleware/oms/OPatch/opatch 

• Set your ORACLE_HOME environment variable to your OMS Home.

[oracle@oem3 14840279]$ echo $ORACLE_HOME
[oracle@oem3 14840279]$ export ORACLE_HOME=$OMS_HOME

• Use opatch lsinventory to verify the status of the inventory and version of opatch. Opatch should be the latest available version. In this case it was 11.1.0.9.8 which was the version bundled with 12.1.0.2.0 so no update was necessary.

[oracle@oem3 14840279]$ opatch lsinventory
Oracle Interim Patch Installer version 11.1.0.9.8
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/Middleware/oms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/Middleware/oms/oraInst.loc
OPatch version    : 11.1.0.9.8
OUI version       : 11.1.0.9.0
Log file location : /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/opatch2013-01-16_23-49-17PM_1.log

OPatch detects the Middleware Home as "/u01/app/oracle/Middleware"
Lsinventory Output file location : /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/lsinv/lsinventory2013-01-16_23-49-17PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
EM Platform (OMS)                                                    12.1.0.2.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :
Patch  13983293     : applied on Sat Sep 15 18:55:57 CDT 2012
Unique Patch ID:  14779750
   Created on 25 Apr 2012, 02:18:06 hrs PST8PDT
   Bugs fixed:
     13587457, 13425845, 11822929


--------------------------------------------------------------------------------
OPatch succeeded.
[oracle@oem3 14840279]$ cd ..

• Stop the oms

• Run prerequisite checks to check for any conflicts with one-off patches. If there are one-offs they may have to be rolled back prior to applying the PSU. The Readme contains instructions on what action to take in this case.

[oracle@oem3 ~]$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./14840279
Oracle Interim Patch Installer version 11.1.0.9.8
Copyright (c) 2012, Oracle Corporation.  All rights reserved.
PREREQ session
Oracle Home       : /u01/app/oracle/Middleware/oms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/Middleware/oms/oraInst.loc
OPatch version    : 11.1.0.9.8
OUI version       : 11.1.0.9.0
Log file location : /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/opatch2013-01-16_23-53-49PM_1.log

OPatch detects the Middleware Home as "/u01/app/oracle/Middleware"
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

• Apply the patch

[oracle@oem3 14840279]$ opatch apply
Oracle Interim Patch Installer version 11.1.0.9.8
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/Middleware/oms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/Middleware/oms/oraInst.loc
OPatch version    : 11.1.0.9.8
OUI version       : 11.1.0.9.0
Log file location : /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/14840279_Jan_16_2013_23_54_24/apply2013-01-16_23-54-24PM_1.log

OPatch detects the Middleware Home as "/u01/app/oracle/Middleware"
Applying interim patch '14840279' to OH '/u01/app/oracle/Middleware/oms'
Verifying environment and performing prerequisite checks...
All checks passed.
Backing up files...
Patching component oracle.sysman.oms.core, 12.1.0.2.0...
Verifying the update...
Patch 14840279 successfully applied
Log file location: /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/14840279_Jan_16_2013_23_54_24/apply2013-01-16_23-54-24PM_1.log
OPatch succeeded.

• Make the post_deploy.sh script executable.

[oracle@oem3 14840279]$ chmod +x post_deploy.sh 

• Execute the post deploy script (repository database and listener should be running)

[oracle@oem3 14840279]$ ./post_deploy.sh
Post deploy OMS automation script [Version: 11.1.0.9.9, Build date: November 12, 2012]
_______________________________________________________________________________________
Please enter Patch Location: /home/oracle/14840279
Please enter SYSMAN password:

The script will run the following command(s):
/u01/app/oracle/Middleware/oms/bin/emctl applypatch repos -patchHome /home/oracle/14840279
/u01/app/oracle/Middleware/oms/bin/emctl register oms metadata -service namedsql -file /u01/app/oracle/Middleware/oms/sysman/metadata/namedsql/loader_report_sql.xml -core
/u01/app/oracle/Middleware/oms/bin/emctl register oms metadata  -service report -file /u01/app/oracle/Middleware/oms/sysman/metadata/reports/Loader_Statistic.xml -core
/u01/app/oracle/Middleware/oms/bin/emctl register oms metadata -service swlib -file /u01/app/oracle/Middleware/oms/sysman/metadata/swlib/multioms -core
/u01/app/oracle/Middleware/oms/bin/emctl register oms metadata -service swlib -file /u01/app/oracle/Middleware/oms/sysman/metadata/swlib/patch -core

Please monitor log file "/home/oracle/14840279/log/2013-01-16-23-58-44.log"...

Script completed successfully.

• Start the OMS 

[oracle@oem3 14840279]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

 That's it. Hopefully this solves a lot of issues that users have been complaining about.

Oracle Enterprise Manager Cloud Control 12.1.0.2 - Database Template

The latest release of Oracle Enterprise Manager Cloud Control 12c (12.1.0.2) is accompanied by a Database template pre-configured for the repository. I was quite pleased to see this addition, since installing Enterprise Manager required some upfront footwork in configuring the database to be used as the repository. Personally, I was a bit annoyed every time the installer complained about parameters that it needed changed on the repository database during installation  (though some we not mandatory during installation).

As a pre-requisite to using the template you should have a software-only installation of a Oracle 11.2.0.3 RDBMS software.

The installation instructions warn that the template is suitable for a Simple Install. However you may also use it as a basis of an Advanced Installation. You can read the rest of the instructions here.

Note: The instructions in the documentation has a screenshot of a Oracle Database 12c DBCA instead of the 11.2.0.3 DBCA. I suspect this is a documentation bug.

To get started you will need to download the new Database Template. This is shown in the red box below.

You can either extract the software on your host or a location where the installer can be accessed by the host and copy them to the ORACLE_HOME/assistants/dbca/templates directory of the database home

or unzip them directly to the location. I chose to do the former.

[oracle@oem3 orastage]$ cd EM12cR2/

[oracle@oem3 EM12cR2]$ ls

DB11.2.0.3_EM_seed_database_linux64      em12cr2_linux64_disk1.zip  em12cr2_linux64_disk3.zip  jdk       oms      Release_Notes.pdf  runInstaller  wls

DB11.2.0.3_EM_seed_database_linux64.zip  em12cr2_linux64_disk2.zip  install                    libskgxn  plugins  response           stage         WT.zip

[oracle@oem3 EM12cR2]$ cd DB11.2.0.3_EM_seed_database_linux64

[oracle@oem3 DB11.2.0.3_EM_seed_database_linux64]$ ls

DB11.2.0.3_EM_seed_database.ctl  DB11.2.0.3_EM_seed_database.dbc  DB11.2.0.3_EM_seed_database.dfb  shpool.sql

[oracle@oem3 DB11.2.0.3_EM_seed_database_linux64]$ cp * /u01/app/oracle/product/11.2.0/dbhome_1/assistants/dbca/templates/

Proceed to run DBCA in order to create the repository. Here I will only show the screens specific to the template.

Choose Create Database then select the option with the new DB11.2.0.3 seed database

 Enter your database name and SID

I noticed that the "Configure Enterprise Manager" checkbox was still checked. I don't think this should be selected since we will be using it for a new repository.

 Uncheck the box and continue to the next screen.

The template will also execute a script after the database has been created. This script sets the shared pool size to 300M.

Memory parameters are conservative for a simple installation. AMM is enabled, SGA set to 768M and PGA 256M. (Server was configured with 4GB)

The processes parameter has been set to 300 as is recommended by the installer in 12.1.0.1

At this point it's business as usual and you can proceed to click finish to review the summary for the database.
There you have it - a pre-configured database ready for your new EM installation without any annoying  dialog boxes... or is it?

I actually kicked off the EM 12.1.0.2 installer and received the following warning message.

It indicated that it wasn't able to complete some of the pre-requisites checks for the repository database and suggested that it be done manually via the EMPrereqkit. The logs didn't offer any help in identifying the error so I clicked OK as this is just a sandbox. For production or test environments I recommend to re-run the EMPrereqkit manually and try to identify any checks that fail.

In summary, this is a very welcomed addition to the Enterprise Manager 12c. It simplifies and speeds up the process of deploying the software.

Now to take EM12c R2 for a test drive...

Learning Oracle VM Server with Oracle VM VirtualBox

In theme with my Collaborate 12 presentation - Learning Oracle with Oracle VM VirtualBox, I've decided to put a quick post together on using VirtualBox to learn Oracle VM Server 3.1. A VM within a VM you say? Reminds me of Inception. But yes it works!

I've been wanting to learn/play with Oracle VM Server for some time now. However, due to the bare metal requirements of the hypervisor I've been unable/unwilling to procure a dedicated machine for such a hobby. Previous attempts at installing OVM Server on VirtualBox also failed and posts by other users on several forums have also confirmed similar problems.
I must say big thank to my friend @Yuri for providing me access to his OVM Server all the way in Australia to help expose me to the technology. However, not having physical access limits the learning experience. 

Wim Coekaerts tweeted about Oracle VM Manager and Server 3.1 was now available as VirtualBox templates (appliances). Now this kind of stuff makes me feel like a kid in a candy store. I quickly downloaded the templates and began importing them into Virtualbox (as soon as I completed working on a prod issue).

There's a Quick Start Guide for the templates and I recommend that you read this first. The OVM Manager templates also come with tutorials on OVM Server to get you familar with the concepts quickly.

I will be using OVM Server to test building RAC and Enterprise Manager 12c environments from templates. 

I love Virtualbox!

My Collaborate 12 Presentations

IOUG Collaborate 12 has come and gone and here I am two months letter without an update. As the saying goes -  better late than never.
This was my first time at Collaborate and presenting to a user group. It was a very personal and professionally gratifying experience. My quick tip session (30 minutes) about RAC/Clusterware Diagnostic Tools went very well for a rookie. I received some great feedback. However, the session proved to be too short to cover the scope of the content and I'm working on preparing an extended version for anyone interested. My second presentation on Learning Oracle with Oracle VM VirtualBox was not as I expected. Maybe it was because of the massive room that I presented in - less intimate and very intimidating. Nevertheless, it was an OK presentation and I had a couple of very engaged attendees who had many questions at the end of the session. Questions are good - in my opinion.
In addition to the my sessions, I found some time to attend others. A few of the ones that stood out to me were Yury Velikanov presentation on "10 Problems with your RMAN Backup Script" - which was an exciting and well attended session on problems and suggestions for improving backup recoverability when using RMAN. There was also "How Oracle Locking Works" by the venerable Arup Nanda - always informative, clear and concise. Riyaj Shamsudeen also presented on "SCAN, VIP, HAIP and other RAC Acronyms" were he went into the internals of SCAN and RAC networking in general.The RAC SIG Expert Panel and Birds of a Feather (I am RAC SIG board member) was also excellent. The Expert Panel had such esteemed panelists as Graham Wood from Oracle's Real World Performance Team, Christo Kutrovsky from Pythian, Markus Michalewicz from Oracle RAC and Arup Nanda "Long Time DBA". Yury has provided some links to their presentations here.

However, the most satisfying part of the entire conference was networking and meeting with many professionals from all over the world. I met some smart Oracle professionals from different companies (Pythian, Walgreens, Facebook , Amazon etc). The folks at Pythian were awesome as usual - I think they had at least six presenters including Alex Gorbachev who did a High Availability Deep Dive session. I personally received lots of support and encouragement from @yvelikanov and @fuadar among others in my social network.

I wouldn't exactly describe myself as a party animal but I have to give kudos to the IOUG staff for putting on a great 80's party to top things off. What happens in Vegas.... :-)
Overall it was a very good conference and I highly recommend it to any Oracle technologist or professional.

Here are my presentations.

Learning Oracle with Oracle VM VirtualBox Whitepaper

View more documents from Leighton Nelson

Learning Oracle with Oracle VM VirtualBox

View more PowerPoint from Leighton Nelson

Looking at RAC, GI/Clusterware Diagnostic Tools

View more PowerPoint from Leighton Nelson

Collaborate 12 Sessions

I've been lucky enough to have been accepted as a presenter at IOUG Collaborate 12 in Las Vegas, NV to be held April 22-26. I will be presenting the following sessions:

High Availability Boot Camp: Looking at RAC, Clusterware/Grid Infrastructure Diagnostic Tools
Learning Oracle with Oracle VM VirtualBox

As RAC SIG US Events Chair I will also be organizing the RAC SIG Experts Panel and Birds of a Feather. 

I'm excited and anxious for what will be my first public presentations. Hope the attendees will enjoy it.

See you in Las Vegas!