Getting NFS to play nice with ASM

Recently I was running some tests in my lab environment consisting of two Oracle databases on separate RHEL 6.2 VMs. The databases were configured in a Data Guard configuration and used ASM as storage. However, I ran into an issue where my standby database was becoming out of sync with the primary database because the standby was out of disk space. At the time I couldn't request additional storage for my standby (that's what happens when you work at 1:00 am while everyone works normal hours) and I needed to complete my testing. The standby server had some NFS mounted shares that had plenty of space however. This is where social interaction is of benefit because I can recall having several discussions with fellow colleagues on twitter about choosing to use NFS as ASM devices and under what conditions one would choose such a route. Yury also wrote up a blog post about it.

This seemed like a good opportunity to test it out. Only this time I will be mixing ASM block devices on a cooked filesystem along with ASM raw devices (these were actually created on top of a NFS datastore but presented to the guests as LUNs).

Please note this is for educational and testing purposes only. I wouldn't recommend doing this in your production environment. Use at your own risk.

Mount the NFS volume with the correct mount options
[root@el6 ~]#  mount -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600 /oradata

Check the existing ASM devices
[grid@el6 ~]$ asmcmd lsdsk -G FLASH

Create a new ASM device on the NFS mounted file system
[grid@el6 ~]$ dd if=/dev/zero of=/oradata/nfs_asm-disk1 bs=1K count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 547.629 s, 1.9 MB/s

Check the value of the ASM_DISKSTRING initialization parameter. This tells ASM where to look for ASM devices.
[grid@el6 ~]$ sqlplus / as sysasm
SQL*Plus: Release Production on Thu Jan 17 02:37:46 2013
Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release - 64bit Production
With the Automatic Storage Management option
SQL> show parameter asm_diskstring
------------------------------------ ----------- ------------------------------
asm_diskstring     string
SQL> alter system set asm_diskstring='/dev/oracleasm/disks/*','/oradata/nfs_asm-disk1';
System altered.
SQL> alter diskgroup flash add disk '/oradata/nfs_asm-disk1';
Diskgroup altered.

List the ASM devices in the diskgroup. Here you'll see both the LUN and new device on the NFS volume in the same diskgroup.
[grid@el6 ~]$ asmcmd lsdsk -G FLASH

Note: If you see errors similar to the following in your alert log, you may need to change the ownership and permissions on your NFS mounted ASM devices so that they are owned by the "ASM" user.

SUCCESS: disk FLASH_0001 (1.3915921253) added to diskgroup FLASH
Thu Jan 17 02:47:45 2013
Direct NFS: write FAILED 13
Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:
ORA-17500: ODM err:KGNFS WRITE FAIL:Permission denied
WARNING: Write Failed. group:1 disk:1 AU:4 offset:114688 size:16384
Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 1 logical extent 0 of file 256 in group 1 on disk 1 allocation unit 4 
Errors in file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_mmon_4037.trc:
ORA-00206: error in writing (block 15, # blocks 1) of control file
ORA-00202: control file: '+FLASH/orcl1/controlfile/current.256.768325217'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
minact-scn: useg scan erroring out with error e:221

Now your ASM will be playing nicely together with both NFS and LUNs.


Popular posts from this blog

Viewing ASM trace files and alert logs in Unix/Linux

ORA-00020: maximum number of processes (%s) exceeded

Troubleshooting RAC Public Network Failure