Wednesday, January 23, 2013

Getting NFS to play nice with ASM

Recently I was running some tests in my lab environment consisting of two Oracle 11.2.0.3 databases on separate RHEL 6.2 VMs. The databases were configured in a Data Guard configuration and used ASM as storage. However, I ran into an issue where my standby database was becoming out of sync with the primary database because the standby was out of disk space. At the time I couldn't request additional storage for my standby (that's what happens when you work at 1:00 am while everyone works normal hours) and I needed to complete my testing. The standby server had some NFS mounted shares that had plenty of space however. This is where social interaction is of benefit because I can recall having several discussions with fellow colleagues on twitter about choosing to use NFS as ASM devices and under what conditions one would choose such a route. Yury also wrote up a blog post about it.

This seemed like a good opportunity to test it out. Only this time I will be mixing ASM block devices on a cooked filesystem along with ASM raw devices (these were actually created on top of a NFS datastore but presented to the guests as LUNs).

Please note this is for educational and testing purposes only. I wouldn't recommend doing this in your production environment. Use at your own risk.

Mount the NFS volume with the correct mount options
[root@el6 ~]#  mount -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600 192.168.1.101:/export/oracle/dnfs /oradata

Check the existing ASM devices
[grid@el6 ~]$ asmcmd lsdsk -G FLASH
Path
/dev/oracleasm/disks/ASMDISK2

Create a new ASM device on the NFS mounted file system
[grid@el6 ~]$ dd if=/dev/zero of=/oradata/nfs_asm-disk1 bs=1K count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 547.629 s, 1.9 MB/s

Check the value of the ASM_DISKSTRING initialization parameter. This tells ASM where to look for ASM devices.
[grid@el6 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Thu Jan 17 02:37:46 2013
Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Automatic Storage Management option
SQL> show parameter asm_diskstring
NAME     TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring     string
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*','/oradata/nfs_asm-disk1';
System altered.
SQL> alter diskgroup flash add disk '/oradata/nfs_asm-disk1';
Diskgroup altered.

List the ASM devices in the diskgroup. Here you'll see both the LUN and new device on the NFS volume in the same diskgroup.
[grid@el6 ~]$ asmcmd lsdsk -G FLASH
Path
/dev/oracleasm/disks/ASMDISK2
/oradata/nfs_asm-disk1

Note: If you see errors similar to the following in your alert log, you may need to change the ownership and permissions on your NFS mounted ASM devices so that they are owned by the "ASM" user.

SUCCESS: disk FLASH_0001 (1.3915921253) added to diskgroup FLASH
Thu Jan 17 02:47:45 2013
Direct NFS: write FAILED 13
Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:
ORA-17500: ODM err:KGNFS WRITE FAIL:Permission denied
WARNING: Write Failed. group:1 disk:1 AU:4 offset:114688 size:16384
Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 1 logical extent 0 of file 256 in group 1 on disk 1 allocation unit 4 
Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:
ORA-00206: error in writing (block 15, # blocks 1) of control file
ORA-00202: control file: '+FLASH/orcl1/controlfile/current.256.768325217'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
minact-scn: useg scan erroring out with error e:221

Now your ASM will be playing nicely together with both NFS and LUNs.



Thursday, January 17, 2013

Oracle Enterprise Manager 12.1.0.2.0 PSU 1

Thanks to @BrianPardy for tweeting the availability of the first patch set update for Oracle Enterprise Manager 12.1.0.2.0. Brian also blogged about it.
The PSU Readme lists quite a number of bugs fixed in the PSU. Surprisingly the application of the patch is quite easy.

I applied it to my 12.1.0.2.0 lab environment without any issues. Just to test for real application on test and production servers.

  • Download the patch 14840279  from My Oracle Support to your OMS server.
  • Extract the zip file
  • Make sure opatch is in your PATH.


[oracle@oem3 14840279]$ which opatch
/usr/bin/which: no opatch in (/u01/app/oracle/Middleware/oms/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/oracle/bin)
[oracle@oem3 14840279]$ export PATH=$OMS_HOME/OPatch:$PATH
[oracle@oem3 14840279]$ which opatch
/u01/app/oracle/Middleware/oms/OPatch/opatch 
  • Set your ORACLE_HOME environment variable to your OMS Home.
[oracle@oem3 14840279]$ echo $ORACLE_HOME
[oracle@oem3 14840279]$ export ORACLE_HOME=$OMS_HOME
  • Use opatch lsinventory to verify the status of the inventory and version of opatch. Opatch should be the latest available version. In this case it was 11.1.0.9.8 which was the version bundled with 12.1.0.2.0 so no update was necessary.
[oracle@oem3 14840279]$ opatch lsinventory
Oracle Interim Patch Installer version 11.1.0.9.8
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/Middleware/oms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/Middleware/oms/oraInst.loc
OPatch version    : 11.1.0.9.8
OUI version       : 11.1.0.9.0
Log file location : /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/opatch2013-01-16_23-49-17PM_1.log

OPatch detects the Middleware Home as "/u01/app/oracle/Middleware"
Lsinventory Output file location : /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/lsinv/lsinventory2013-01-16_23-49-17PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
EM Platform (OMS)                                                    12.1.0.2.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :
Patch  13983293     : applied on Sat Sep 15 18:55:57 CDT 2012
Unique Patch ID:  14779750
   Created on 25 Apr 2012, 02:18:06 hrs PST8PDT
   Bugs fixed:
     13587457, 13425845, 11822929


--------------------------------------------------------------------------------
OPatch succeeded.
[oracle@oem3 14840279]$ cd ..

  • Stop the oms

  • Run prerequisite checks to check for any conflicts with one-off patches. If there are one-offs they may have to be rolled back prior to applying the PSU. The Readme contains instructions on what action to take in this case.
[oracle@oem3 ~]$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./14840279
Oracle Interim Patch Installer version 11.1.0.9.8
Copyright (c) 2012, Oracle Corporation.  All rights reserved.
PREREQ session
Oracle Home       : /u01/app/oracle/Middleware/oms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/Middleware/oms/oraInst.loc
OPatch version    : 11.1.0.9.8
OUI version       : 11.1.0.9.0
Log file location : /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/opatch2013-01-16_23-53-49PM_1.log

OPatch detects the Middleware Home as "/u01/app/oracle/Middleware"
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
  • Apply the patch
[oracle@oem3 14840279]$ opatch apply
Oracle Interim Patch Installer version 11.1.0.9.8
Copyright (c) 2012, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/Middleware/oms
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/Middleware/oms/oraInst.loc
OPatch version    : 11.1.0.9.8
OUI version       : 11.1.0.9.0
Log file location : /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/14840279_Jan_16_2013_23_54_24/apply2013-01-16_23-54-24PM_1.log

OPatch detects the Middleware Home as "/u01/app/oracle/Middleware"
Applying interim patch '14840279' to OH '/u01/app/oracle/Middleware/oms'
Verifying environment and performing prerequisite checks...
All checks passed.
Backing up files...
Patching component oracle.sysman.oms.core, 12.1.0.2.0...
Verifying the update...
Patch 14840279 successfully applied
Log file location: /u01/app/oracle/Middleware/oms/cfgtoollogs/opatch/14840279_Jan_16_2013_23_54_24/apply2013-01-16_23-54-24PM_1.log
OPatch succeeded.
  • Make the post_deploy.sh script executable.
[oracle@oem3 14840279]$ chmod +x post_deploy.sh 
  • Execute the post deploy script (repository database and listener should be running)
[oracle@oem3 14840279]$ ./post_deploy.sh
Post deploy OMS automation script [Version: 11.1.0.9.9, Build date: November 12, 2012]
_______________________________________________________________________________________
Please enter Patch Location: /home/oracle/14840279
Please enter SYSMAN password:

The script will run the following command(s):
/u01/app/oracle/Middleware/oms/bin/emctl applypatch repos -patchHome /home/oracle/14840279
/u01/app/oracle/Middleware/oms/bin/emctl register oms metadata -service namedsql -file /u01/app/oracle/Middleware/oms/sysman/metadata/namedsql/loader_report_sql.xml -core
/u01/app/oracle/Middleware/oms/bin/emctl register oms metadata  -service report -file /u01/app/oracle/Middleware/oms/sysman/metadata/reports/Loader_Statistic.xml -core
/u01/app/oracle/Middleware/oms/bin/emctl register oms metadata -service swlib -file /u01/app/oracle/Middleware/oms/sysman/metadata/swlib/multioms -core
/u01/app/oracle/Middleware/oms/bin/emctl register oms metadata -service swlib -file /u01/app/oracle/Middleware/oms/sysman/metadata/swlib/patch -core

Please monitor log file "/home/oracle/14840279/log/2013-01-16-23-58-44.log"...

Script completed successfully.
  • Start the OMS 
[oracle@oem3 14840279]$ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up

 That's it. Hopefully this solves a lot of issues that users have been complaining about.