Wednesday, March 23, 2011

Grid Infrastructure 11.2.0.2 Hands-On: Automatic SSH Setup

While installing Oracle RAC it is important to setup SSH user equivalence for installation of of the binaries.  Oracle 11gR2 RAC now includes automatic ssh setup. While reading one of my favorite blogs by Martin Bach he describes using a utility called sshUserSetup.sh for setting up user equivalence for RAC. This takes out some of the repetitive steps needed to setup ssh user equivalence when installing a RAC cluster.  As Martin pointed out you can also use OUI to setup user equivalence but it helps if you can run the cluster verification utility (CVU) to check on many pre-requisites before starting the installation.  During installation using OUI on the AIX platform however, OUI is unable to perform automatic SSH setup. This means that SSH user equivalence will need to be setup manually prior to installation.

To ease the setup all you need to do is to run the sshUserSetup.sh script and follow the prompts directions. It takes as parameters the user (oracle or grid) cluster hosts and other options such as whether or not to use a passphrase.  You also need to specify the advanced flag to enable user equivalence to be setup from all nodes, otherwise it will only be done on the local node. Here is how I set up ssh user equivalence for the grid infrastructure owner 'grid' on a 2 node cluster.
[grid@gridora30t sshsetup]$ ./sshUserSetup.sh -user grid -hosts "gridora30t gridora31t" -noPromptPassphrase -advanced -exverify
The output of this script is also logged into /tmp/sshUserSetup_2011-01-25-10-13-10.log
Hosts are gridora30t gridora31t
user is grid
Platform:- AIX
Checking if the remote hosts are reachable
PING gridora30t (10.0.0.2.85): 56 data bytes
64 bytes from 10.0.0.2.85: icmp_seq=0 ttl=255 time=0 ms
64 bytes from 10.0.0.2.85: icmp_seq=1 ttl=255 time=0 ms
64 bytes from 10.0.0.2.85: icmp_seq=2 ttl=255 time=0 ms
64 bytes from 10.0.0.2.85: icmp_seq=3 ttl=255 time=0 ms
64 bytes from 10.0.0.2.85: icmp_seq=4 ttl=255 time=0 ms
--- gridora30t ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0/0/0 ms
PING gridora31t (10.0.0.2.86): 56 data bytes
64 bytes from 10.0.0.2.86: icmp_seq=0 ttl=255 time=1 ms
64 bytes from 10.0.0.2.86: icmp_seq=1 ttl=255 time=0 ms
64 bytes from 10.0.0.2.86: icmp_seq=2 ttl=255 time=0 ms
64 bytes from 10.0.0.2.86: icmp_seq=3 ttl=255 time=0 ms
64 bytes from 10.0.0.2.86: icmp_seq=4 ttl=255 time=0 ms
--- gridora31t ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0/0/1 ms
Remote host reachability check succeeded.
The following hosts are reachable: gridora30t gridora31t.
The following hosts are not reachable: .
All hosts are reachable. Proceeding further...
firsthost gridora30t
numhosts 2
The script will setup SSH connectivity from the host gridora30t to all
the remote hosts. After the script is executed, the user can use SSH to run
commands on the remote hosts or copy files between this host gridora30t
and the remote hosts without being prompted for passwords or confirmations.
NOTE 1:
As part of the setup procedure, this script will use ssh and scp to copy
files between the local host and the remote hosts. Since the script does not
store passwords, you may be prompted for the passwords during the execution of
the script whenever ssh or scp is invoked.
NOTE 2:
AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
directories.
Do you want to continue and let the script make the above mentioned changes (yes/no)?
yes
The user chose yes
User chose to skip passphrase related questions.
Creating .ssh directory on local host, if not present already
Creating authorized_keys file on local host
Changing permissions on authorized_keys to 644 on local host
Creating known_hosts file on local host
Changing permissions on known_hosts to 644 on local host
Creating config file on local host
If a config file exists already at /home/grid/.ssh/config, it would be backed up to /home/grid/.ssh/config.backup.
Removing old private/public keys on local host
Running SSH keygen on local host with empty passphrase
Generating public/private rsa key pair.
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
7f:d6:d5:25:6b:78:15:b6:da:e1:db:4e:91:d1:de:b1 grid@gridora30t
The key's randomart image is:
+--[ RSA 1024]----+
|               o |
|              . +|
|              .=+|
|             .+=X|
|        S   ..+E=|
|         .   + .+|
|          . o ..o|
|           o   o |
|                .|
+-----------------+
Creating .ssh directory and setting permissions on remote host gridora30t
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR grid. THIS IS AN SSH REQUIREMENT.
The script would create /home/grid/.ssh/config file on remote host gridora30t. If a config file exists already at /home/grid/.ssh/config, it would be backed up to /home/grid/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host gridora30t.
Warning: Permanently added 'gridora30t,10.0.0.2.85' (RSA) to the list of known hosts.
grid@gridora30t's password:
Done with creating .ssh directory and setting permissions on remote host gridora30t.
Creating .ssh directory and setting permissions on remote host gridora31t
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR grid. THIS IS AN SSH REQUIREMENT.
The script would create /home/grid/.ssh/config file on remote host gridora31t. If a config file exists already at /home/grid/.ssh/config, it would be backed up to /home/grid/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host gridora31t.
Warning: Permanently added 'gridora31t,10.0.0.2.86' (RSA) to the list of known hosts.
grid@gridora31t's password:
Done with creating .ssh directory and setting permissions on remote host gridora31t.
Copying local host public key to the remote host gridora30t
The user may be prompted for a password or passphrase here since the script would be using SCP for host gridora30t.
grid@gridora30t's password:
Done copying local host public key to the remote host gridora30t
Copying local host public key to the remote host gridora31t
The user may be prompted for a password or passphrase here since the script would be using SCP for host gridora31t.
grid@gridora31t's password:
Done copying local host public key to the remote host gridora31t
Creating keys on remote host gridora30t if they do not exist already. This is required to setup SSH on host gridora30t.
Creating keys on remote host gridora31t if they do not exist already. This is required to setup SSH on host gridora31t.
Generating public/private rsa key pair.
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
7d:a7:ac:59:1f:19:79:4f:7a:bf:a2:0e:ce:d8:c4:44 grid@gridora31t
The key's randomart image is:
+--[ RSA 1024]----+
|                 |
|                 |
|         E       |
|        ..     . |
|        S.. . + o|
|        o  o o *.|
|         +  + + o|
|        * .+ o o.|
|       . ++o. o.o|
+-----------------+
Updating authorized_keys file on remote host gridora30t
Updating known_hosts file on remote host gridora30t
Updating authorized_keys file on remote host gridora31t
Updating known_hosts file on remote host gridora31t
cat: cannot open /home/grid/.ssh/authorized_keys.tmp
cat: cannot open /home/grid/.ssh/known_hosts.tmp
SSH setup is complete.
------------------------------------------------------------------------
Verifying SSH setup
===================
The script will now run the date command on the remote nodes using ssh
to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
PASSWORDS. If you see any output other than date or are prompted for the
password, ssh is not setup correctly and you will need to resolve the
issue and set up ssh again.
The possible causes for failure could be:
1. The server settings in /etc/ssh/sshd_config file do not allow ssh
for user grid.
2. The server may have disabled public key based authentication.
3. The client public key on the server may be outdated.
4. /home/grid or /home/grid/.ssh on the remote host may not be owned by grid.
5. User may not have passed -shared option for shared remote users or
may be passing the -shared option for non-shared remote users.
6. If there is output in addition to the date, but no password is asked,
it may be a security alert shown as part of company policy. Append the
additional text to the /sysman/prov/resources/ignoreMessages.txt file.
------------------------------------------------------------------------
--gridora30t:--
Running /usr/bin/ssh -x -l grid gridora30t date to verify SSH connectivity has been setup from local host to gridora30t.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Tue Jan 25 10:15:48 CST 2011
------------------------------------------------------------------------
--gridora31t:--
Running /usr/bin/ssh -x -l grid gridora31t date to verify SSH connectivity has been setup from local host to gridora31t.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Tue Jan 25 10:15:49 CST 2011
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from gridora30t to gridora30t
------------------------------------------------------------------------
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Tue Jan 25 10:15:50 CST 2011
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from gridora30t to gridora31t
------------------------------------------------------------------------
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Tue Jan 25 10:15:51 CST 2011
------------------------------------------------------------------------
-Verification from gridora30t complete-
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from gridora31t to gridora30t
------------------------------------------------------------------------
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Tue Jan 25 10:15:51 CST 2011
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from gridora31t to gridora31t
------------------------------------------------------------------------
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Tue Jan 25 10:15:52 CST 2011
------------------------------------------------------------------------
-Verification from gridora31t complete-
SSH verification complete.
If RAC will also be installed, the procedure needs to be repeated for the RAC database owner as well e.g. oracle.

RAC RDA Setup on AIX

Remote Diagnostic Agent (RDA) is a Oracle utility used to collect information on RAC configurations. It collects a wealth of information that would otherwise be tedious to collect manually. This includes operating system, network, hardware, database, cluster configuration and logs. This is useful for troubleshooting RAC issues such as node evictions. Once collected this information can then be uploaded to My Oracle Support (MOS) as part of a SR. It also complements Oracle Configuration Manager (OCM) which is used to proactively do health checks and help with the speedy resolution of SRs. RDA can be downloaded from MOS.
Unzip the RDA archive
oracle@gridora31[gridprd2]-/u02 >cd /mnt/orasource/AIX/utils
oracle@gridora31[gridprd2]-/mnt/orasource/AIX/utils >ls
RDA422.zip ocm-Production-AIX-ppc.zip swingbench230422.zip
cpumonitor02.91.zip orion_aix_ppc64.gz swingbench240688.zip
datagenerator040198.zip osw3b.tar traceanalyzer010100.zip
oracle@gridora31[gridprd2]-/mnt/orasource/AIX/utils >cp RDA422.zip /u02/
oracle@gridora31[gridprd2]-/mnt/orasource/AIX/utils >cd /u02
oracle@gridora31[gridprd2]-/u02 >ls
RDA422.zip datapump lost+found orion tmp
oracle@gridora31[gridprd2]-/u02 >unzip RDA422.zip
Verify the RAC nodes status using olsnodes
grid@gridora30[+ASM1]-/home/grid >olsnodes
gridora30
gridora31
 
Ensure SSH/SCP connectivity as the RDA owner e.g. oracle from each server
oracle@gridora30[gridprd1]-/u02/rda >ssh gridora31 date
Thu Feb 24 12:45:07 CST 2011
oracle@gridora31[gridprd2]-/u02/rda >ssh gridora30 date
Thu Feb 24 12:47:54 CST 2011
 
Perform initial setup to determine which information to collect.
oracle@gridora30[gridprd1]-/u02/rda >./rda.pl -vX RDA::Remote setup_cluster /
------------------------------------------------------------------------------
Requesting common information
------------------------------------------------------------------------------
Enter the absolute path of the directory where RDA will be installed on the
remote nodes.
Hit 'Return' to accept the default (/u02/rda)
>
Enter the absolute path of the directory where setup files and reports will be
stored on the remote nodes?
Hit 'Return' to accept the default (/u02/rda)
>
Do you want RDA to use an alternative login to execute remote requests (Y/N)?
Hit 'Return' to accept the default (N)
>
------------------------------------------------------------------------------
Requesting information for node gridora30
------------------------------------------------------------------------------
Enter the Oracle home to be analyzed on the node gridora30
Hit 'Return' to accept the default (/u01/oracle/11.2.0/dbhome_1)
>
------------------------------------------------------------------------------
Requesting information for node gridora31
------------------------------------------------------------------------------
Enter the Oracle home to be analyzed on the node gridora31
Hit 'Return' to accept the default (/u01/oracle/11.2.0/dbhome_1)
>
------------------------------------------------------------------------------
RAC Setup Summary
------------------------------------------------------------------------------
Nodes:
. NOD001 gridora30/gridprd1
. NOD002 gridora31/gridprd2
2 nodes found
-------------------------------------------------------------------------------
S000INI: Initializes the Data Collection
-------------------------------------------------------------------------------
RDA uses the output file prefix to identify all files belonging to the same
data collection. The prefix must start with a letter and must contain only
alphanumeric characters.
Enter the prefix to be used for all the generated files
Hit 'Return' to accept the default (RDA)
>
Enter the directory used for all the files to be generated
Hit 'Return' to accept the default (/u02/rda/output)
>
Do you want to keep report packages from previous runs (Y/N)?
Hit 'Return' to accept the default (N)
>
Is a fresh collection performed as the default action (Y/N)?
Hit 'Return' to accept the default (Y)
>
Enter the Oracle home to be used for data analysis
Hit 'Return' to accept the default (/u01/oracle/11.2.0/dbhome_1)
>
Enter the network domain name for this server
Hit 'Return' to accept the default (griddba.com)
>
-------------------------------------------------------------------------------
S010CFG: Collects Key Configuration Information
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
S090OCM: Set up the Configuration Manager Interface
-------------------------------------------------------------------------------
RDA benefits from Oracle Configuration Manager by improving its data
collection. Oracle Configuration Manager (OCM) collects the configuration
information and uploads it to the OCM Repository in connected mode. When OCM
is configured in disconnected mode, it adds the OCM payload to the RDA report
packages in future RDA runs.

Select the mode that you want to install Oracle Configuration Manager in
/u01/oracle/11.2.0/dbhome_1
C Connected Mode
D Disconnected Mode
Enter the letter corresponding to your choice
Hit 'Return' to accept the default (C)
>
Do you want to install OCM Database collection scripts (Y/N)?
Hit 'Return' to accept the default (Y)
>
Enter a SYSDBA user for installing the OCM Database collection scripts
Hit 'Return' to accept the default (sys)
>
Is /u01/oracle/11.2.0/dbhome_1 an Oracle E-Business Suite Database Oracle home
(Y/N)?
Hit 'Return' to accept the default (N)
>
-------------------------------------------------------------------------------
S919LOAD: Produces the External Collection Reports
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
S999END: Finalizes the Data Collection
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
S909RDSP: Produces the Remote Data Collection Reports
-------------------------------------------------------------------------------
Updating the setup file ...
 
Review setup.cfg file to make any changes as necessary
oracle@gridora30[gridprd1]-/u02/rda >ls
Convert README_Windows.txt hcve rda.cmd
DISCLAIM.txt README_irda.txt irda.pl rda.com
IRDA README_upgrade.txt lNOD001_setup.cfg rda.pl
RDA ccr lNOD002_setup.cfg rda.sh
README_Unix.txt dfw modules setup.cfg
README_VMS.txt engine output
oracle@gridora30[gridprd1]-/u02/rda >vi setup.cfg
 
Check Remote Nodes accessibility
oracle@gridora30[gridprd1]-/u02/rda >./rda.pl -vX RDA::Remote list
Nodes:
. NOD001 gridora30/gridprd1
. NOD002 gridora31/gridprd2
2 nodes found
 
Start RDA collection on all nodes
./rda.pl -v -e REMOTE_TRACE=1
 
The RDA reports are generated in the /output directory

oracle@gridora30[gridprd1]-/u02/rda/output >ls
JOB_RDA RDA_END_system.htm RDA_S909RDSP.txt
LCK_RDA RDA_END_system.txt RDA_S909RDSP_A.fil
RDA.RDA_gridora30.zip RDA_RDSP_overview.htm RDA_S999END.htm
RDA.log RDA_RDSP_overview.txt RDA_S999END.toc
RDA_CFG_S.fil RDA_RDSP_results.htm RDA_S999END.txt
RDA_CFG_homes.htm RDA_RDSP_results.txt RDA_S999END_A.fil
RDA_CFG_homes.txt RDA_S010CFG.htm RDA__blank.htm
RDA_CFG_oh_inv.htm RDA_S010CFG.toc RDA__index.htm
RDA_CFG_oh_inv.txt RDA_S010CFG.txt RDA__index.txt
RDA_END_files.htm RDA_S010CFG_A.fil RDA__start.htm
RDA_END_files.txt RDA_S090OCM_A.fil RDA_mrc.fil
RDA_END_report.htm RDA_S909RDSP.htm RDA_rda.css
RDA_END_report.txt RDA_S909RDSP.toc remote

The RDA.RDA_nodename.zip file can be uploaded to Oracle Support to assist with a SR.