Friday, November 22, 2013

Oracle Database 12c Direct NFS

Oracle direct NFS or dNFS introduced in Oracle 11gR2 allows the NFS operations to be moved to the database layer instead of operating system kernel. After installing Oracle RAC 12.1.0.1 it appears the direct NFS is enabled by default.
You can check the alert log to verify that the direct NFS ODM library has been loaded. Messages similar to the following will be displayed.

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0

Direct NFS may use several configuration files to determine the mount points on which it is used.
$ORACLE_HOME/dbs/oranfstab
/etc/oranfstab
/etc/mtab

If oranfstab is not configured then it will check the currently mounted filesystems from /etc/mtab and if the mount options are correct it will use dNFS for database I/O.
For example:
cat /etc/mtab | grep oraclelinuxtest
filer1:/vol/volnds001_oraclelinuxtest_db/nds001 /u03/oradata nfs rw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0,addr=10.10.88.19 0 0
filer1:/vol/volnds001_oraclelinuxtest_cf/nds001 /u03/racctl nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,actimeo=0,addr=10.10.88.19 0 0
You can verify from the v$dnfs views whether or not Oracle is actively using direct NFS.
SQL> select * from v$dnfs_servers;
no rows selected
The query above checks if Oracle is actively using dNFS filers defined in either /etc/oranfstab, $ORACLE_HOME/oranfstab or /etc/mtab. This view is populated the first time that Oracle accesses the filers. Based on our output either Oracle hasn't performed any IO using dNFS.

Looking at the alert log we can see that direct NFS is defined on Filer1. There are other seemingly benign messages to check oradism. After all there are no warning or error messages.

tail alert_racdb1.log
Fri Sep 06 15:01:01 2013
Direct NFS: please check that oradism is setuid
Fri Sep 06 15:48:12 2013
Direct NFS: please check that oradism is setuid
Fri Sep 06 16:51:50 2013
Direct NFS: channel id [0] path [Filer1] to filer [Filer1] via local [] is UP
Fri Sep 06 16:51:50 2013
Direct NFS: attempting to mount /vol/volnds001_oraclelinuxtest_db/nds001 on filer Filer1 defined in mtab
Fri Sep 06 16:51:50 2013
Direct NFS: please check that oradism is setuid

The following MOS note describes this issue:
Database Startup Failed with "Direct NFS: please check that oradism is setuid" (Doc ID 1430654.1)

Check permissions on oradism. It should be owned by root.

[root@lnxvmoraract02 ~]# cd /u01/app/oracle/product/12.1.0/dbhome_1/bin
[root@lnxvmoraract02 bin]# ls -l oradism
-rwxr-x--- 1 oracle oinstall 105070 Sep  4 16:27 oradism

The permissions for oradism is incorrect and setuid is not set.
Change the permissions (as root) so that it's owned by root:root and change permissions to enable the suid.
[root@lnxvmoraract02 bin]# chown root:root oradism
[root@lnxvmoraract02 bin]# chmod 4755 oradism

Restart the database and query v$dnfs_server to verify that dNFS is actually being used.

SQL> col svrname format a60
SQL> col dirname format a80
SQL> set lines 200
SQL> select svrname,dirname,mntport,nfsversion,wtmax,rtmax,con_id
  2  from v$dnfs_servers;

SVRNAME                                                      DIRNAME                                              MNTPORT NFSVERSI      WTMAX       RTMAX     CON_ID
------------------------------------------------------------ -------------------------------------------------------------------------------- ---------- -------- ---------- ---------- ----------

Filer1                                          /vol/volnds001_oraclelinuxtest_db/nds001                4046 NFSv3.0       65536       65536          0

Upgrading EM12c Agents using EMCLI

Note to self on how to upgrade EM Agents using EMCLI

Find upgradeable agents:
$ emcli get_upgradable_agents
Upgradable Agents

Agent Installed Version Version After Upgrade Platform Oracle Home
----- ----------------- --------------------- -------- -----------
oradba10t:3872 12.1.0.2.0 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.2.0
oradba11t:3872 12.1.0.2.0 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.2.0

$ vi /tmp/oradba_agents.txt
$ cat /tmp/oradba_agents.txt
oradba10t:3872
oradba11t:3872
Upgrade agents using an input file 
$ emcli upgrade_agents -input_file="agents_file:/tmp/oradba_agents.txt" -job_name="labrac_agent_upgrade" -stage_location=/u02/tmp
The agent list size is 2

Upgradable Agents

Agent Installed Version Version After Upgrade Platform Oracle Home
----- ----------------- --------------------- -------- -----------
oradba10t:3872 12.1.0.2.0 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.2.0
oradba11t:3872 12.1.0.2.0 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.2.0

For the following Management Agents root.sh cannot be run. Run the script(/root.sh) manually post upgrade

Agent Reason
----- ------
oradba10t:3872 Preferred Privileged Credential for Oracle Home of Agent : Not Set | Privilege Delegation for Host : Not Set
oradba11t:3872 Preferred Privileged Credential for Oracle Home of Agent : Not Set | Privilege Delegation for Host : Not Set

Agent Upgrade Job submitted for Upgradable Agents shown above. Use emcli get_agent_upgrade_status command or goto EM CONSOLE -> Set Up -> Manage Cloud Control -> Upgrade Agents -> Agent Upgrade Results to see job status
Job Name : LABRAC_AGENT_UPGRADE
Check Status of agent upgrade
$ emcli get_agent_upgrade_status -job_name=LABRAC_AGENT_UPGRADE

Showing for each agent in the job LABRAC_AGENT_UPGRADE

Agent Status Started Ended
----- ------ ------- -----
oradba11t:3872 Running 2013-11-21 16:12:52 CST -
oradba10t:3872 Running 2013-11-21 16:12:52 CST -
Detailed steps are not available from the command-line. Use the GUI to get the exact steps that are being executed.
$ emcli get_agent_upgrade_status -job_name=LABRAC_AGENT_UPGRADE

Showing for each agent in the job LABRAC_AGENT_UPGRADE

Agent Status Started Ended
----- ------ ------- -----
oradba11t:3872 Success 2013-11-21 16:12:52 CST 2013-11-21 16:33:30 CST
oradba10t:3872 Success 2013-11-21 16:12:52 CST 2013-11-21 16:35:03 CST
Remove old agents. First check which agents have been successfully upgraded.
$ emcli get_signoff_agents

Agents available for Sign-off

Agent Installed Version Platform Oracle Home

----- ----------------- -------- -----------

lnxvmorat01 :3872 12.1.0.3.0 Linux x86-64 /u02/oracle/agent12c/core/12.1.0.3.0

smoora01 :3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12cR2/core/12.1.0.3.0

oradba10t:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

pocora30d:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

pocora31d:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

oem1 :3872 12.1.0.3.0 Linux x86-64 /u02/oracle/agent12c/core/12.1.0.3.0

pocora10d :3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12cR2/core/12.1.0.3.0

oradba11t:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

pocora11d :3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12cR2/core/12.1.0.3.0            
                       
The output can be re-directed to a file which can then be used as the input file for the signoff.
$ emcli get_signoff_agents -output_file="/tmp/signoff_agents.txt"

Agents available for Sign-off

Agent Installed Version Platform Oracle Home

----- ----------------- -------- -----------

lnxvmorat01:3872 12.1.0.3.0 Linux x86-64 /u02/oracle/agent12c/core/12.1.0.3.0

smoora01:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12cR2/core/12.1.0.3.0

oradba10t:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

pocora30d:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

pocora31d:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

oem1:3872 12.1.0.3.0 Linux x86-64 /u02/oracle/agent12c/core/12.1.0.3.0

pocora10d:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12cR2/core/12.1.0.3.0

oradba11t:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

pocora11d:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12cR2/core/12.1.0.3.0           
                        
View output of file
$ cat /tmp/signoff_agents.txt

lnxvmorat01:3872
smoora01:3872
oradba10t:3872
pocora30d:3872
pocora31d:3872
oem1:3872
pocora10d:3872
oradba11t:3872
pocora11d:3872
Remove agents matching string oradba%
$ emcli signoff_agents -agents="oradba%" -job_name=CLEANUP_12cR2_AGNTS

Agents available for Sign-off

Agent Installed Version Platform Oracle Home

----- ----------------- -------- -----------

oradba10t:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

oradba11t:3872 12.1.0.3.0 IBM AIX on POWER Systems (64-bit) /u02/oracle/agent12c/core/12.1.0.3.0

Agent Sign-off Job Submitted for the above agents. Use emcli get_signoff_status or goto EM CONSOLE -> Set Up -> Manage Cloud Control -> Upgrade Agents -> Post Agent Upgrade Tasks -> Sign-off Agent Results to see job status
Job Name : CLEANUP_12CR2_AGNTS
Check status of signoff job
$ emcli get_signoff_status

Showing for each job

Job Name Status Total Agents Started Ended

-------- ------ ------------ ------- -----

CLEANUP_12CR2_AGNTS Success 2 2013-11-21 17:08:30 CST    
                                             
2013-11-21 17:08:44 CST

Note that the sign-off doesn't remove the old agent home. You will have to manually remove the directory.
e.g. rm -rf /u01/oracle/agent12c/core/12.1.0.2.0

Configure and De-configure OMS with SLB



My OEM environment consists of two management servers (OMS) for high availability (level 3 HA). We also have a SLB configured for the OMSs but they haven't been configured with the SLB virtual hostname. I know, this sounds very confusing but there's a difference. Essentially they act as individual OMSs since agents can only upload to one OMS and console is pointed to a single OMS.
The load balancer is a F5 BIG-IP and all the configuration setting were done for using OEM. This whitepaper clearly describes how to do this.  

After upgrading my OEM environment from 12.1.0.2.0 to 12.1.0.3.0 I decided to reconfigure the OMSs to use the SLB virtual hostname. Reconfiguring the SLB involves securing the OMSs with the virtual hostname of the SLB.
[oracle@oms1[]-/u01/Oracle/Middleware/oms/bin >./emctl secure oms -host oemcc.example.com -secure_port 4899 -slb_port 4889 -slb_console_port 443
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Securing OMS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Enter Agent Registration Password :
Securing OMS... Successful
Restart OMS
Once the OMS has been secured you can confirm the configuration by checking with emctl status oms -details.
[oracle@oms1[]-/u01/Oracle/Middleware/oms/bin >./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oemcc.example.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7803
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4899
EM Instance Home           : /u01/Oracle/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/Oracle/gc_inst/em/EMGC_OMS1/sysman/log
SLB or virtual hostname: oemcc.example.com
HTTPS SLB Upload Port : 4899
HTTPS SLB Console Port : 443
Agent Upload is unlocked.
OMS Console is unlocked.
Active CA ID: 2
Console URL: https://oemcc.example.com:443/em
Upload URL: https://oemcc.example.com:4899/empbs/upload
WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : oms1.example.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oms1.example.com
WebTier is Up
Oracle Management Server is Up

From the output you can see the the SLB or virtual hostname is using the virtual hostname instead of the hostname of the OMS server - oms1 in this example.

After securing each OMS they need to be restarted.
$ ./emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
Once both OMSs have been secured and restarted, the agents also need to be re-secured with the SLB virtual hostname. To secure the agents use:
emctl secure agent -emdWalletSrcUrl https://oemcc.example.com:4899/em
It was this part that failed during my configuration. I'm not sure why it failed but I think it may have to do with the certificates generated for the SLB. I decided to revert the OMS changes until I could figure out the reason for the failure since all my agents were unable to reach the OMS. This is where I seemingly met a brick wall. I couldn't find any documentation that showed how to de-configure the SLB. Luckily emctl help provides a wealth of information about the commands verbs. This is where I found the solution - emctl secure -no_slb
[oracle@oms1[]-/u01/Oracle/Middleware/oms/bin >./emctl secure oms -no_slb
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Securing OMS... Started.
Enter Enterprise Manager Root (SYSMAN) Password :
Enter Agent Registration Password :
Securing OMS... SuccessfulRestart OMS
Verify that the SLB configuration is no longer present
[oracle@oms1[]-/u01/Oracle/Middleware/oms/bin >./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : oms1.example.com
HTTP Console Port : 7788
HTTPS Console Port : 7803
HTTP Upload Port : 4889
HTTPS Upload Port : 4899
EM Instance Home : /u01/Oracle/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/Oracle/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostnameAgent Upload is unlocked.
OMS Console is unlocked.
Active CA ID: 2
Console URL: https://oms1.example.com:7803/em
Upload URL: https://oms1.example.com:4899/empbs/upload

WLS Domain Information
Domain Name : GCDomain
Admin Server Host : oms1.example.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oms1.example.com
WebTier is Up
Oracle Management Server is Up
After re-securing both OMSs and restarting them my agents should now be able to upload to the OMSs. It turns out that I needed to secure the agents again without the slb. This was easily done through the OEM Console, Setup -> Manage Cloud Control -> Agents and then selecting the agents. Click the secure icon which will create a job for securing the agents. Enter the agent registration password and submit the job.

If I missed something in the documentation, please feel free to leave a comment below.