DevTech101

DevTech101

Contents

Oracle CRS installation

Install oracle CRS 11G from CD

Oracle CRS Notes

  • Make sure /etc/issue is empty other wise Oracle OCR wont install.
  • Make sure ssh works without a password even local mentioned in part 1

Sun Cluster and QFS configuration

Create an mcf file

create mcf file on both servers (each file system needs an entry in the mcf file)

cat /etc/opt/SUNWsamfs/mcf
 
# device                ord     type    familyset       on      other parameters
#--------------------------------------------------------------------------------
archivelogs             10      ma      archivelogs     on      shared
/dev/md/oralogs/dsk/d0  11      mm      archivelogs     on
/dev/md/oralogs/dsk/d10 15      mr      archivelogs     on
#
# device                ord     type    familyset       on      other parameters
#--------------------------------------------------------------------------------
flashback                       20      ma      flashback       on      shared
/dev/md/oralogs/dsk/d20         21      mm      flashback       on
/dev/md/oralogs/dsk/d30         25      mr      flashback       on

Create QFS file system hosts file

Create hosts.[familyset name]
Note: server should be the same server as the metaset master

cat /etc/opt/SUNWsamfs/hosts.archivelogs
# Host          Host IP                 Server          Not     Server
# Name          Addresses               Priority        Used    Host
# -----         ---------               --------        ----    ----
db1         clusternode1-priv       1               -       server
db2         clusternode2-priv       2               -
 
cat /etc/opt/SUNWsamfs/hosts.flashback
# Host          Host IP                 Server          Not     Server
# Name          Addresses               Priority        Used    Host
# -----         ---------               --------        ----    ----
db1         clusternode1-priv       1               -       server
db2         clusternode2-priv       2               -

Create samfs mount option file

Create /etc/opt/SUNWsamfs/samfs.cmd for each file system

cat /etc/opt/SUNWsamfs/samfs.cmd
fs = archivelogs
 stripe=1
 sync_meta=1
 qwrite
 forcedirectio
 notrace
 rdlease=600
 wrlease=600
 aplease=600
 nstreams=1024
 dmr
 abr
fs = flashback
 stripe=1
 sync_meta=1
 qwrite
 forcedirectio
 notrace
 rdlease=600
 wrlease=600
 aplease=600
 dmr
 abr

Create the QFS file system

First run sam-fsd to check for errors (the errors column should show 0)
If no errors run

QFS to refresh all his config files with the new configuration:

samd config

Create QFS file system:

sammkfs -S archivelogs (run on one node only)
sammkfs -S flashback (run on one node only)
 
mkdir /archive_logs
mkdir /flash_back

Add the mounts to vfstab

archivelogs    -       /archive_logs    samfs   -       no      shared,notrace
flashback    -       /flash_back     samfs   -       no      shared,notrace

Note: Run first on QFS Master then on client

mount /archive_logs
mount /flash_back

Note: To force a meta data server take over and re-create of the file system you could run

sammkfs [-R]-S db1 archivelogs
sammkfs [-R]-S db1 flashback

Sun Cluster QFS configuration

To add QFS to Sun Cluster and metasets master failover

ln -s /opt/SUNWsamfs/sc/etc/SUNW.qfs /usr/cluster/lib/rgm/rtreg/SUNW.qfs

Register the QFS resource with Sun cluster:

scrgadm -a -t SUNW.qfs

Setup the QFS resource:

scrgadm -a -g QFS-rg -h db1,db2 -y RG_DEPENDENCIES="rac-fmwk-rg"
scrgadm -a -g QFS-rg -t SUNW.qfs -j QFS-fs-rs -x QFSFileSystem=/archive_logs,/flash_back

Turn the QFS resource online:

scswitch -Z -g QFS-rg

If you add another file system at a latter time the cluster needs to know about it run:

clrs set -p QFSFileSystem=/archive_logs,/flash_back QFS-fs-rs

Complete the Sun Cluster and Oracle RAC configuration

Note:

Before you use this wizard, ensure that the following prerequistes are met:

* The RAC framework resource group is created and is online.

* Storage resources for Oracle files are configured.

* The Oracle Real Application Clusters software is installed.

* The installation of the Oracle Real Application Clusters software is verified.

* The Oracle database is created.

* Node-specific files and directories for a shared file system are created.

To complete the install run clsetup select the options below

  • 3) Data Services – then
  • 4) Oracle Real Application Clusters – then
  • Answer yes for the next question
  • 1) Oracle RAC Create Configuration – then
  • 3) Oracle RAC Database Resources – then continue

Answer all the reset of questions
For a complete example Click Here
Sun Cluster and crs references is available Here
To manuel complete the Sun cluster and CRS setup run use the process below

/usr/cluster/bin/clresourcegroup online -emM rac-fmwk-rg
 
/usr/cluster/bin/clresourcetype register SUNW.crs_framework
 
/usr/cluster/bin/clresource create -t SUNW.crs_framework -g rac-fmwk-rg -p \
Resource_dependencies=rac-fmwk-rs -p \
resource_dependencies_offline_restart=scalocrdg-rs{local_node},scalvotingdg-rs{local_node} crs_framework-rs
 
/usr/cluster/bin/clresourcegroup online -emM rac-fmwk-rg
 
/usr/cluster/bin/clresourcegroup create -p Desired_primaries=2 -p \
nodelist=db1,db2 -p RG_mode=Scalable -p RG_affinities=++rac-fmwk-rg,++scaldg-rg -p \
Maximum_primaries=2 rac_server_proxy-rg
 
/usr/cluster/bin/clresource create -t SUNW.scalable_rac_server_proxy -g \
rac_server_proxy-rg -p Resource_dependencies=rac-fmwk-rs -p \
resource_dependencies_offline_restart=scaloradg-rs,scalvotingdg-rs,scalocrdg-rs,crs_framework-rs -p \
CRS_HOME=/oracle/product/11.1.0/crs_1 -p \
DB_NAME=DB -p ORACLE_HOME=/oracle/product/11.1.0/db_1 -p ORACLE_SID{db2}=DB2 -p \
ORACLE_SID{db1}=DB1 rac_server_proxy-rs
 
/usr/cluster/bin/clresourcegroup online -emM rac_server_proxy-rg

Attempting to launch command:

/opt/SUNWscor/dsconfig/bin/scproxy_crs NO /oracle/product/11.1.0/crs_1 scaloradg-rs

DB Proxy resource sun.db1.scaloradg-rs:
Creating Oracle CRS profile in /var/cluster/ucmm/profile.

/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db1.scaloradg-rs \
-t application -d "Proxy for Sun Cluster resource scaloradg-rs" \
-dir /var/cluster/ucmm/profile -a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p \
restricted -h db1 -f -o ra=60,st=20

Proxy resource sun.db1.scaloradg-rs: Registering with Oracle CRS.

/oracle/product/11.1.0/crs_1/bin/crs_register sun.db1.scaloradg-rs -dir /var/cluster/ucmm/profile

Proxy resource sun.db1.scaloradg-rs: Registered with Oracle CRS.

Adding dependency of ora.DB.DB1.inst on Sun Cluster proxy resource sun. db1.scaloradg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB1.inst -update -r "sun.db1.scaloradg-rs"

Dependency of ora.DB.DB1.inst now includes sun.db1.scaloradg-rs.

Proxy resource sun.db2.scaloradg-rs: Creating Oracle CRS profile in /var/cluster/ucmm/profile.

/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db2.scaloradg-rs \
-t application -d "Proxy for Sun Cluster resource scaloradg-rs" -dir /var/cluster/ucmm/profile \
-a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p \
restricted -h db2 -f -o ra=60,st=20

Proxy resource sun.db2.scaloradg-rs: Registering with Oracle CRS.

/oracle/product/11.1.0/crs_1/bin/crs_register sun.db2.scaloradg-rs -dir /var/cluster/ucmm/profile

Proxy resource sun.db2.scaloradg-rs: Registered with Oracle CRS.

Adding dependency of ora.DB.DB2.inst on Sun Cluster proxy resource sun. db2.scaloradg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.BDB2.inst -update -r "sun.db2.scaloradg-rs"

Dependency of ora.DB.DB2.inst now includes sun.db2.scaloradg-rs.

Attempting to launch command:

/opt/SUNWscor/dsconfig/bin/scproxy_crs NO
/oracle/product/11.1.0/crs_1 scalvotingdg-rs DB

Proxy resource sun.db1.scalvotingdg-rs:
Creating Oracle CRS profile in /var/cluster/ucmm/profile.

/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db1.scalvotingdg-rs -t application \
-d "Proxy for Sun Cluster resource scalvotingdg-rs" -dir /var/cluster/ucmm/profile -a \
/opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p restricted -h db1 -f -o ra=60,st=20

Proxy resource sun.db1.scalvotingdg-rs: Registering with Oracle CRS.

/oracle/product/11.1.0/crs_1/bin/crs_register sun.db1.scalvotingdg-rs -dir /var/cluster/ucmm/profile

Proxy resource sun.db1.scalvotingdg-rs: Registered with Oracle CRS.

Starting proxy resource sun.db1.scalvotingdg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_start sun.db1.scalvotingdg-rs

Attempting to start `sun.db1.scalvotingdg-rs` on member `db1` Start of `sun.db1.scalvotingdg-rs` on member `db1` succeeded.
Proxy resource sun.db1.scalvotingdg-rs started successfully.

Adding dependency of ora.DB.DB1.inst on Sun Cluster proxy resource sun.db1.scalvotingdg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB1.inst -update -r "sun.v.scaloradg-rs sun.db1.scalvotingdg-rs"

Dependency of ora.DB.DB1.inst now includes sun.db1.scalvotingdg-rs.

Proxy resource sun.db2.scalvotingdg-rs: Creating Oracle CRS profile in /var/cluster/ucmm/profile.

/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db2.scalvotingdg-rs \
-t application -d "Proxy for Sun Cluster resource scalvotingdg-rs" \
-dir /var/cluster/ucmm/profile -a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p \
restricted -h db2 -f -o ra=60,st=20

Proxy resource sun.db2.scalvotingdg-rs: Registering with Oracle CRS.

/oracle/product/11.1.0/crs_1/bin/crs_register sun.v.scalvotingdg-rs -dir /var/cluster/ucmm/profile

Proxy resource sun.db2.scalvotingdg-rs: Registered with Oracle CRS.

Starting proxy resource sun.db2.scalvotingdg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_start sun.v.scalvotingdg-rs

Attempting to start `sun.db2.scalvotingdg-rs` on member `db2` Start of `sun.v.scalvotingdg-rs` on member `v` succeeded.
Proxy resource sun.db2.scalvotingdg-rs started successfully.

Adding dependency of ora.DB.DB2.inst on Sun Cluster proxy resource sun.
db2.scalvotingdg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB2.inst -update -r \
"sun.db2.scaloradg-rs sun.db2.scalvotingdg-rs"

Dependency of ora.DB.DB2.inst now includes sun.db2.scalvotingdg-rs.

Attempting to launch command:

/opt/SUNWscor/dsconfig/bin/scproxy_crs NO /oracle/product/11.1.0/crs_1 scalocrdg-rs DB

Proxy resource sun.db1.scalocrdg-rs: Creating Oracle CRS profile in /var/cluster/ucmm/profile.

/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db1.scalocrdg-rs \
-t application -d "Proxy for Sun Cluster resource scalocrdg-rs" -dir /var/cluster/ucmm/profile \
-a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p restricted -h db1 -f -o ra=60,st=20

Proxy resource sun.db1.scalocrdg-rs: Registering with Oracle CRS.

/oracle/product/11.1.0/crs_1/bin/crs_register sun.db1.scalocrdg-rs -dir /var/cluster/ucmm/profile

Proxy resource sun.db1.scalocrdg-rs: Registered with Oracle CRS.

Starting proxy resource sun.db1.scalocrdg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_start sun.db1.scalocrdg-rs

Attempting to start `sun.db1.scalocrdg-rs` on member `db1` Start of `sun.db1.scalocrdg-rs` on member `db1` succeeded.
Proxy resource sun.db1.scalocrdg-rs started successfully.

Adding db1 of ora.DB.DB1.inst on Sun Cluster proxy resource sun. db1.scalocrdg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB1.inst -update \
-r "sun.db1.scaloradg-rs sun.db1.scalvotingdg-rs sun.db1.scalocrdg-rs"

Dependency of ora.DB.DB1.inst now includes sun.db1.scalocrdg-rs.

Proxy resource sun.db2.scalocrdg-rs: Creating Oracle CRS profile in /var/cluster/ucmm/profile.

/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db2.scalocrdg-rs \
-t application -d "Proxy for Sun Cluster resource scalocrdg-rs" -dir /var/cluster/ucmm/profile \
-a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p restricted -h db2 -f -o ra=60,st=20

Proxy resource sun.db2.scalocrdg-rs: Registering with Oracle CRS.

/oracle/product/11.1.0/crs_1/bin/crs_register sun.db2.scalocrdg-rs -dir /var/cluster/ucmm/profile

Proxy resource sun.db2.scalocrdg-rs: Registered with Oracle CRS.

Starting proxy resource sun.db2.scalocrdg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_start sun.db2.scalocrdg-rs

Attempting to start `sun.db2.scalocrdg-rs` on member `db2` Start of `sun.db2.scalocrdg-rs` on member `db2` succeeded.
Proxy resource sun.db2.scalocrdg-rs started successfully.

Adding dependency of ora.DB.DB2.inst on Sun Cluster proxy resource sun.db2.scalocrdg-rs.

/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB2.inst -update \
-r "sun.db2.scaloradg-rs sun.db2.scalvotingdg-rs sun.db2.scalocrdg-rs"

Dependency of ora.DB.DB2.inst now includes sun.db2.scalocrdg-rs.

Sun Cluster 3.2 tweeks

Increase Rac FramWork timeout

Its recommended to increase the RAC FramWork Time Out

/usr/cluster/bin/clresource set -p Svm_step4_timeout=360 rac-svm-rs

Remove wrongly set dependencies

Fix the QFS-rg dependence that was wrongly set
Remove QFS-rg dependence:

/usr/cluster/bin/clresourcegroup set -p RG_dependencies= QFS-rg

ReAdd the correct dependencies:

/usr/cluster/bin/clresourcegroup set -p RG_affinities=++scaldg-rg QFS-rg

Add QFS Resource restart dependencies

Add a restart Resource dependence on the QFS-fs-rs resource

/usr/cluster/bin/clresource set -p Resource_dependencies_offline_restart=scaloradg-rs,scalocrdg-rs,scalvotingdg-rs QFS-fs-rs

Fix Cluster cron jobs

The cluster crontab jobs are times are wrong and needs to change

crontab -e

Change From:
20 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/eventlog
20 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/DS
20 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/commandlog

Change To:
20 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/eventlog
25 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/DS
30 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/commandlog

Setup Cluster NTP to our servers

Modify the /etc/inet/ntp.conf.cluster like this

#server 127.127.1.0
server ntp.pool.org

Enable dynamic pools

Make sure dynamic pools are enabled (for dynamic hardware configuration)

svcadm enable /system/pools/dynamic

Chnage cluster to reboot if all disks fail

By default Sun Cluster 3.2 will not reboot if all disk failed, to change that run

Get device listing:
scdpm -p all:all

Set the reboot flag to on:
clnode set -p reboot_on_path_failure=enabled db1 db2

Verify the change:
scdpm -p all:all

UnSet the cluster monitor for local disks

First get the local disk did, then unset the cluster monitor on this did’s (run this on all cluster nodes)

scdidadm -l
cldevice unmonitor d1 d2 d3

Verify the change:
scdpm -p all:all

Set and change the monitor to local only

Check the setting:
cldg show dsk/d3

Device Groups ===

Device Group Name:                              dsk/d3
Type:                                            Disk
failback:                                        false
Node List:                                       db1
preferenced:                                     false
localonly:                                       false
autogen:                                         true
numsecondaries:                                  1
device names:                                    /dev/did/rdsk/d3s2

Change localonly to true for all local disks:
cldg set -p localonly=true dsk/d3

For extra protection add the clusterpriv address

Note: This step is not needed but could be done for extra protection
Add all clusternode1-priv to the /etc/hosts,/etc/netmasks file

/etc/hosts
Cluster private IP's:
10.10.5.129    clusternode1-priv
10.10.5.9      clusternode1-priv-physical1
10.10.5.17     clusternode1-priv-physical2
10.10.5.130    clusternode2-priv
10.10.5.10     clusternode2-priv-physical1
10.10.5.18     clusternode2-priv-physical2

Sun Cluster and GW Gateway configuration

Add GW VIP address to the local host files

cat /etc/hosts
10.10.10.12       gw-vip

Crated bea user and group accounts in /etc/password
Copy /usr/local/bea from joel/hosea to the new cluster

Configure GW Gateway to be managed by Sun Cluster

clresourcetype register SUNW.gds
 
/usr/cluster/bin/clresourcegroup create -p nodelist=db1,db2 gw-vip-rg
/usr/cluster/bin/clressharedaddress create -g gw-vip-rg -h gw-vip -N db1_gp@db1,wedb2bdb2_gp@db2 gw-vip-rs
 
/usr/cluster/bin/clresourcegroup online -eM gw-vip-rg
 
/usr/cluster/bin/clresourcegroup create -p Desired_primaries=2 -p Maximum_primaries=2 -p Nodelist=db1,db2 -p RG_SLM_type=manual \
-p Global_resources_used=false GW-rg
 
/usr/cluster/bin/clresource create -t SUNW.gds:6 -g GW-rg -p Resource_project_name= \
-p Resource_dependencies=gw-vip-rs \
-p Load_balancing_policy=Lb_weighted -p Network_resources_used=gw-vip-rs -p Scalable=TRUE -p Load_balancing_weights=1@db1,1@db2 \
-p R_description="Scalable data service resource for SUNW.gds:6" -p Port_list=9999/tcp -p Validate_command= \
-p Start_command=/usr/cluster/GW/Start.sh -p Stop_command=/usr/cluster/GW/Stop.sh \
-p Probe_command=/usr/cluster/GW/Probe.sh GW-rs
 
/usr/cluster/bin/clresourcegroup online -eM GW-rg

DR Configuration:

clresourcetype register SUNW.gds
 
/usr/cluster/bin/clresourcegroup create -p nodelist=dbdr1,dbdr2 gwdr-vip-rg
/usr/cluster/bin/clressharedaddress create -g gwdr-vip-rg -h gwdr-vip -N dbdr1_gp@dbdr1,dbdr2_gp@dbdr2 gwedr-vip-rs
 
/usr/cluster/bin/clresourcegroup online -eM gwdr-vip-rg
 
/usr/cluster/bin/clresourcegroup create -p Desired_primaries=2 -p Maximum_primaries=2 -p Nodelist=dbdr1,dbdr2 -p RG_SLM_type=manual \
-p Global_resources_used=false GW-rg
 
/usr/cluster/bin/clresource create -t SUNW.gds:6 -g GW-rg -p Resource_project_name= \
-p Resource_dependencies=gwdr-vip-rs \
-p Load_balancing_policy=Lb_weighted -p Network_resources_used=gwdr-vip-rs -p Scalable=TRUE -p Load_balancing_weights=1@dbdr1,1@dbdr2 \
-p R_description="Scalable data service resource for SUNW.gds:6" -p Port_list=9999/tcp -p Validate_command= \
-p Start_command=/usr/cluster/GW/Start.sh -p Stop_command=/usr/cluster/GW/Stop.sh \
-p Probe_command=/usr/cluster/GW/Probe.sh GW-rs
 
/usr/cluster/bin/clresourcegroup online -eM GW-rg

Sun Cluster Help

Sun Cluster commands

To show all resource groups and list there status or just resources

List resource groups status
clrg status

List resources status:
clrs status

To stop or start a resource groups or multiple resource groups

To turn on:
clrg switch -n db1,db2 resource-group1 resource-group2

To turn off:
clrg ofline -n db1,db2 resource-group1 resource-group2

To disable or enable a resource or multiple resources

To enable a resource:
clrs disable resource-1 resource-2

To disable a resource:
clrs enable resource-1 resource-2

Oracle Commands

To check the crs status

Check to Oracle Clusterware and DB health:
crs_stat -t

Check CRS health:
crsctl check crs

To restart the Oracle crs clean

If crs_stop -all doesn’t work then run this on each node that has a problem.

cd /oracle/product/11.1.0/crs_1/bin
# ./crsctl stop crs
Stopping resources. 
This could take several minutes.
Error while stopping resources. Possible cause: CRSD is down.
Stopping Cluster Synchronization Services. 
Unable to communicate with the Cluster Synchronization Services daemon.
 
# To force a crs stop just run
crsctl stop crs -force
 
# ./crsctl start crs
Attempting to start Oracle Clusterware stack 
The CRS stack will be started shortly
 
# ./crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy

Issues and resolution doing Sun Cluster and Oracle RAC configuration

Problem 1

Problem: Cluster panic after reboot

  • Problem: Cluster panic after reboot
Aug 28 08:59:40 svc.startd[8]: svc:/system/cluster/rpc-pmf:default: Method "/usr/cluster/lib/svc/method/svc_rpc_pmf start" failed with exit status 96.
Aug 28 08:59:40 svc.startd[8]: system/cluster/rpc-pmf:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details)
Aug 28 08:59:43 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door
Aug 28 08:59:43 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door
Aug 28 08:59:44 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door
Aug 28 08:59:44 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door
Aug 28 08:59:45 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door
Notifying cluster that this node is panicking

panic[cpu0]/thread=2a100047ca0: Failfast: Aborting zone "global" (zone ID 0) because "sc_zonesd" died 35 seconds ago.

000002a1000473f0 cl_runtime:__1cZsc_syslog_msg_log_no_args6Fpviipkc0_nZsc_syslog_msg_status_enum__+30 (6002c0c0000, 3, 0, 46, 2a1000475f0, 706c3461)
 %l0-3: 0000000000000000 0000000000010000 0000000000000000 0000000000000000
 %l4-7: 0000000000000000 0000000000000001 0000000000000000 000000000180c000
000002a1000474a0 cl_runtime:__1cCosNsc_syslog_msgDlog6MiipkcE_nZsc_syslog_msg_status_enum__+1c (6002c0a73b8, 3, 0, 706c3461, 60032da8d4a, 70400)
 %l0-3: 0000060032da8c40 0000000000070400 00000000000706be 0000000000070400
 %l4-7: 00000000706c3452 00000000706c3000 00000000706c2000 0000000000000001
000002a100047550 cl_haci:__1cHff_implNunit_timedout6M_v_+70 (60032da8cd8, 193c400, 16, 60032da8c40, 180c000, 1)
 %l0-3: 000000007aa8585c 0000000000000000 000000000121a000 0000000000000001
 %l4-7: 0000000000000001 0000000000000001 0000000000000000 00000000018b8f58

Solution: Install Sun Cluster patch

Solution: Install Sun Cluster patch 126106-15 fixed the problem
Note: If you run in to this bug after the install of this patch you might need to uninstall and reinstall the patch
Another problem related to uninstall / reinstall of the patch

cd /etc/opt/SUNWcluster/conf

/etc/opt/SUNWcluster/conf #  grep oracle.dba.gid *
udlm.conf:oracle.dba.gid  : y
udlm.conf.template:oracle.dba.gid  : dba 

Change in udlm.conf the oracle.dba.gid : y to oracle.dba.gid : dba
reboot when done

Problem 2

Problem: metaset metainit will refuse to work

  • Problem: After adding disks to a multi-owner metaset metainit will refuse to work with errors.

Solution: A reboot is needed

Solution: A reboot is needed for the metadb to work properly

Before a re-boot:
[root@db1] / # metadb -s oradg -i
flags           first blk       block count
u            16              32768           /dev/did/dsk/d11s7
u            16              32768           /dev/did/dsk/d22s7
u            16              32768           /dev/did/dsk/d10s7
u            16              32768           /dev/did/dsk/d21s7
u            16              32768           /dev/did/dsk/d9s7
u            16              32768           /dev/did/dsk/d20s7
u            16              32768           /dev/did/dsk/d8s7

After a reboot:
[root@db1] / # metadb -s oradg -i
flags           first blk       block count
a m     luo   r    16              32768           /dev/did/dsk/d11s7
a       luo   r    16              32768           /dev/did/dsk/d22s7
a       luo   r    16              32768           /dev/did/dsk/d10s7
a       luo   r    16              32768           /dev/did/dsk/d21s7
a       luo   r    16              32768           /dev/did/dsk/d9s7
a       luo   r    16              32768           /dev/did/dsk/d20s7
a       luo   r    16              32768           /dev/did/dsk/d8s7

Problem 3

Problem: Not All IO components work in a M5000

  • Problem: All IO components add to the 2nd IO boat dose not show up on the system

Solution: Moving system boards is needed

Solution:
The IO Boards add to the 2nd IO boat will only show up if one CPU and one Memory board is moved to the other IO Path (Slot3/Slot4)

Before the board was moved:
XSCF> showhardconf
SPARC Enterprise M5000;
+ Serial:BEF08265F0; Operator_Panel_Switch:Locked;
+ Power_Supply_System:Single; SCF-ID:XSCF#0;
+ System_Power:On; System_Phase:Cabinet Power On;
Domain#0 Domain_Status:Running;

MBU_B Status:Normal; Ver:0201h; Serial:BE081813E2  ;
+ FRU-Part-Number:CF00541-0478 05   /541-0478-05          ;
+ Memory_Size:16 GB;
CPUM#0-CHIP#0 Status:Normal; Ver:0201h; Serial:PP074003RV  ;
+ FRU-Part-Number:CA06761-D105 A3   /375-3477-01          ;
+ Freq:2.150 GHz; Type:16;
+ Core:2; Strand:2;
CPUM#0-CHIP#1 Status:Normal; Ver:0201h; Serial:PP074003RV  ;
+ FRU-Part-Number:CA06761-D105 A3   /375-3477-01          ;
+ Freq:2.150 GHz; Type:16;
+ Core:2; Strand:2;
*       CPUM#2-CHIP#0 Status:Deconfigured; Ver:0201h; Serial:PP0740030W  ;
+ FRU-Part-Number:CA06761-D105 A3   /375-3477-01          ;
+ Freq:2.150 GHz; Type:16;
+ Core:2; Strand:2;
*       CPUM#2-CHIP#1 Status:Deconfigured; Ver:0201h; Serial:PP0740030W  ;
+ FRU-Part-Number:CA06761-D105 A3   /375-3477-01          ;
+ Freq:2.150 GHz; Type:16;
+ Core:2; Strand:2;
MEMB#0 Status:Normal; Ver:0101h; Serial:BF080998WR  ;
+ FRU-Part-Number:CF00541-0545 05   /541-0545-05          ;
MEM#0A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-dc19390c;
+ Type:1A; Size:1 GB;
MEM#0B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-dc19390a;
+ Type:1A; Size:1 GB;
MEM#1A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-dc19390b;
+ Type:1A; Size:1 GB;
MEM#1B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-dc19390d;
+ Type:1A; Size:1 GB;
MEM#2A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-dc193906;
+ Type:1A; Size:1 GB;
MEM#2B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-dc193907;
+ Type:1A; Size:1 GB;
MEM#3A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-dc193908;
+ Type:1A; Size:1 GB;
MEM#3B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-dc193909;
+ Type:1A; Size:1 GB;
MEMB#1 Status:Normal; Ver:0101h; Serial:BF0813AH3C  ;
+ FRU-Part-Number:CF00541-0545 05   /541-0545-05          ;
MEM#0A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-e709709b;
+ Type:1A; Size:1 GB;
MEM#0B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-d421b34e;
+ Type:1A; Size:1 GB;
MEM#1A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-d421b34f;
+ Type:1A; Size:1 GB;
MEM#1B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-d421b350;
+ Type:1A; Size:1 GB;
MEM#2A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-e70970a4;
+ Type:1A; Size:1 GB;
MEM#2B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-e70970a5;
+ Type:1A; Size:1 GB;
MEM#3A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-e70970a6;
+ Type:1A; Size:1 GB;
MEM#3B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-e70970a7;
+ Type:1A; Size:1 GB;
DDC_A#0 Status:Normal;
DDC_A#1 Status:Normal;
DDC_A#2 Status:Normal;
DDC_A#3 Status:Normal;
DDC_B#0 Status:Normal;
DDC_B#1 Status:Normal;
IOU#0 Status:Normal; Ver:0101h; Serial:BF0815B2EK  ;
+ FRU-Part-Number:CF00541-2240 03   /541-2240-03          ;
DDC_A#0 Status:Normal;
DDCR Status:Normal;
DDC_B#0 Status:Normal;
PCI#1 Name_Property:network; Card_Type:Other;
PCI#2 Name_Property:SUNW,qlc; Card_Type:Other;
IOU#1 Status:Normal; Ver:0101h; Serial:BF0816BFL3  ;
+ FRU-Part-Number:CF00541-2240 03   /541-2240-03          ;
DDC_A#0 Status:Normal;
DDCR Status:Normal;
DDC_B#0 Status:Normal;
XSCFU Status:Normal,Active; Ver:0101h; Serial:BF08109GQ1  ;
+ FRU-Part-Number:CF00541-0481 04   /541-0481-04          ;
OPNL Status:Normal; Ver:0101h; Serial:BF0813AFKG  ;
+ FRU-Part-Number:CF00541-0850 06   /541-0850-06          ;
PSU#0 Status:Normal; Serial:0017527-0748016175;
+ FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50;
+ Power_Status:On; AC:200 V;
PSU#1 Status:Normal; Serial:0017527-0748016184;
+ FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50;
+ Power_Status:On; AC:200 V;
PSU#2 Status:Normal; Serial:0017527-0803010075;
+ FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50;
+ Power_Status:On; AC:200 V;
PSU#3 Status:Normal; Serial:0017527-0803010108;
+ FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50;
+ Power_Status:On; AC:200 V;
FANBP_C Status:Normal; Ver:0301h; Serial:BF0812A7E2  ;
+ FRU-Part-Number:CF00541-0848 05   /541-0848-05          ;
FAN_A#0 Status:Normal;
FAN_A#1 Status:Normal;
FAN_A#2 Status:Normal;
FAN_A#3 Status:Normal;

Notice "IOU#1" Now shows network and qlc.
After the board was moved:
XSCF> showhardconf
SPARC Enterprise M5000;
+ Serial:BEF0825530; Operator_Panel_Switch:Service;
+ Power_Supply_System:Single; SCF-ID:XSCF#0;
+ System_Power:On; System_Phase:Cabinet Power On;
Domain#0 Domain_Status:Running;

MBU_B Status:Normal; Ver:0201h; Serial:BE0812107C  ;
+ FRU-Part-Number:CF00541-0478 05   /541-0478-05          ;
+ Memory_Size:16 GB;
CPUM#0-CHIP#0 Status:Normal; Ver:0201h; Serial:PP0816015K  ;
+ FRU-Part-Number:CA06761-D105 A4   /375-3477-01          ;
+ Freq:2.150 GHz; Type:16;
+ Core:2; Strand:2;
CPUM#0-CHIP#1 Status:Normal; Ver:0201h; Serial:PP0816015K  ;
+ FRU-Part-Number:CA06761-D105 A4   /375-3477-01          ;
+ Freq:2.150 GHz; Type:16;
+ Core:2; Strand:2;
CPUM#2-CHIP#0 Status:Normal; Ver:0201h; Serial:PP08160159  ;
+ FRU-Part-Number:CA06761-D105 A4   /375-3477-01          ;
+ Freq:2.150 GHz; Type:16;
+ Core:2; Strand:2;
CPUM#2-CHIP#1 Status:Normal; Ver:0201h; Serial:PP08160159  ;
+ FRU-Part-Number:CA06761-D105 A4   /375-3477-01          ;
+ Freq:2.150 GHz; Type:16;
+ Core:2; Strand:2;
MEMB#0 Status:Normal; Ver:0101h; Serial:BF0814AQLP  ;
+ FRU-Part-Number:CF00541-0545 05   /541-0545-05          ;
MEM#0A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-da189d92;
+ Type:1A; Size:1 GB;
MEM#0B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-da189d8a;
+ Type:1A; Size:1 GB;
MEM#1A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-da189d93;
+ Type:1A; Size:1 GB;
MEM#1B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-da189d94;
+ Type:1A; Size:1 GB;
MEM#2A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-da189d8e;
+ Type:1A; Size:1 GB;
MEM#2B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-da189d90;
+ Type:1A; Size:1 GB;
MEM#3A Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-da189d8b;
+ Type:1A; Size:1 GB;
MEM#3B Status:Normal;
+ Code:2c000000000000000818HTF12872PY-667D20200-da189d91;
+ Type:1A; Size:1 GB;
MEMB#4 Status:Normal; Ver:0101h; Serial:BF08078HL7  ;
+ FRU-Part-Number:CF00541-0545 05   /541-0545-05          ;
MEM#0A Status:Normal;
+ Code:2c000000000000000818HTF12872Y-53EB3 0300-d41e17d2;
+ Type:1A; Size:1 GB;
MEM#0B Status:Normal;
+ Code:2c000000000000000818HTF12872Y-53EB3 0300-d41e1793;
+ Type:1A; Size:1 GB;
MEM#1A Status:Normal;
+ Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa51;
+ Type:1A; Size:1 GB;
MEM#1B Status:Normal;
+ Code:2c000000000000000818HTF12872Y-53EB3 0300-d41e1736;
+ Type:1A; Size:1 GB;
MEM#2A Status:Normal;
+ Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa52;
+ Type:1A; Size:1 GB;
MEM#2B Status:Normal;
+ Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa54;
+ Type:1A; Size:1 GB;
MEM#3A Status:Normal;
+ Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa53;
+ Type:1A; Size:1 GB;
MEM#3B Status:Normal;
+ Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa56;
+ Type:1A; Size:1 GB;
DDC_A#0 Status:Normal;
DDC_A#1 Status:Normal;
DDC_A#2 Status:Normal;
DDC_A#3 Status:Normal;
DDC_B#0 Status:Normal;
DDC_B#1 Status:Normal;
IOU#0 Status:Normal; Ver:0101h; Serial:BF0742530G  ;
+ FRU-Part-Number:CF00541-2240 02   /541-2240-02          ;
DDC_A#0 Status:Normal;
DDCR Status:Normal;
DDC_B#0 Status:Normal;
PCI#1 Name_Property:network; Card_Type:Other;
PCI#2 Name_Property:SUNW,qlc; Card_Type:Other;
IOU#1 Status:Normal; Ver:0101h; Serial:BF0820CQD5  ;
+ FRU-Part-Number:CF00541-2240 03   /541-2240-03          ;
DDC_A#0 Status:Normal;
DDCR Status:Normal;
DDC_B#0 Status:Normal;
PCI#1 Name_Property:network; Card_Type:Other;
PCI#2 Name_Property:SUNW,qlc; Card_Type:Other;
XSCFU Status:Normal,Active; Ver:0101h; Serial:BF07455P4J  ;
+ FRU-Part-Number:CF00541-0481 04   /541-0481-04          ;
OPNL Status:Normal; Ver:0101h; Serial:BF073943C0  ;
+ FRU-Part-Number:CF00541-0850 06   /541-0850-06          ;
PSU#0 Status:Normal; Serial:0017527-0811015311;
+ FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50;
+ Power_Status:On; AC:200 V;
PSU#1 Status:Normal; Serial:0017527-0811015504;
+ FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50;
+ Power_Status:On; AC:200 V;
PSU#2 Status:Normal; Serial:0017527-0811015503;
+ FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50;
+ Power_Status:On; AC:200 V;
PSU#3 Status:Normal; Serial:0017527-0811015511;
+ FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50;
+ Power_Status:On; AC:200 V;
FANBP_C Status:Normal; Ver:0301h; Serial:BF08088QAR  ;
+ FRU-Part-Number:CF00541-0848 04   /541-0848-04          ;
FAN_A#0 Status:Normal;
FAN_A#1 Status:Normal;
FAN_A#2 Status:Normal;
FAN_A#3 Status:Normal;

Problem 4

Problem: SSH X-forwarding dose not work work

  • Problem: SSH Xforwarding won’t work (you could see this with env|grep -i display)

Solution: You will have to create a hostname6.lo0

Solution: You will have to create a hostname6.lo0 and reboot to get ssh Xforwarding to work

touch hostname6.lo0
reboot

After reboot the fowling error will come and may be ignored.
t_optmgmt: System error: Cannot assign requested address

Problem 5

Problem: Sun cluster reports online faulted for rac proxy startup

  • Problem: The Sun Cluster reports rac_server_proxy-rg “Online faulted” Becuse it takes to long to start

Solution: Extend the cluster start time to 900 seconds

Solution: You will have to Extend the cluster start time to 900 seconds

/usr/cluster/bin/clresource set -p START_TIMEOUT=900 rac_server_proxy-rs

Problem 6

Problem: Oracle Rac hangs / waits

  • Problem: The Oracle Data Base will wait for the private network to recover if failed

Solution: The Oracle clusterwere should only use the Sun cluster virtual private interface

Solution: You will have to remove the physical interfaces from the Oralce CRS database

/oracle/product/11.1.0/crs_1/bin/oifcfg getif
nxge0  10.10.20.0  global  public
nxge3  10.11.5.8  global  cluster_interconnect   <- to be removed
nxge4  10.10.20.0  global  public
nxge7  10.11.5.16  global  cluster_interconnect  <- to be removed
clprivnet0  10.11.5.128  global  cluster_interconnect

/oracle/product/11.1.0/crs_1/bin/oifcfg delif  -global nxge3
/oracle/product/11.1.0/crs_1/bin/oifcfg delif  -global nxge7

/oracle/product/11.1.0/crs_1/bin/oifcfg getif
nxge0  10.10.20.0  global  public
nxge4  10.10.20.0  global  public
clprivnet0  10.10.5.128  global  cluster_interconnect

Rman And NetBackup San Backup

NetBackup Rman SAN Client Installation and configuration

References and help

List of references on how to setup Sun Cluster and Oracle RAC configuration

Sun Cluster References

QFS References

To Get to Sun Cluster 3.2 and Oracle Real Application Cluster (RAC) Installation / configuration Part 1 Click Here

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x
%d bloggers like this: