Configuring RAC to use SCAN
The set-up for SCAN is implemented as follows:
STEP 1 - Add a new DNS entry for SCAN (e.g.
/etc/hosts). This will create a single DNS domain name that resolves to
all of the IP addresses in your RAC cluster (one for each node). This
will use a round-robin algorithm
root> cat /etc/hosts
myscan.mydomain.com IN A 122.22.22.22 IN A 122.22.22.23 IN
A 122.22.22.24
STEP 2 - Create a
SCAN VIP for each IP address in step 1.
STEP 3 - Create a SCAN listener for each
VIP that was created in step 2.
STEP 4 - Change the tnsnames.ora file.
This entry will point SCAN to the single DNS hostname created in step 1.
Prior to the introduction of SCAN (11g r1 and earlier using TAF), the tnsnames.ora
file contained a list of hostnames, one for each node in the cluster:
SCANDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = myscan1.mydomain.com)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = myscan2.mydomain.com)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = myscan3.mydomain.com)(PORT = 1522))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = SCANDB)
)
)
STEP 5 - Set the remote_listener and local_listener parameters:
You want to set your remote_listener parameter to point to a
tnsnames.ora entry for SCAN as defined in step 4.
remote_listener=myscan.mydomain.com:1522
The SCAN
listener for RAC
This single domain addressing is implemented via a "scan
listener" and the status can be checked with the standard lsnrctl command,
passing the SCAN listener name as an argument:
oracle> lsnrctl
service
oracle> lsnrctl stat
myscan_listener
LSNRCTL for UNIX: Version 11.2.0.1.0
. .
Connecting to
(DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_MYSCAN1)))
STATUS of the LISTENER . . .
Convert Single Instance Databases to RAC
Convert Single Instance Databases to RAC
1.1 Overview
We have Oracle grid infrastructure 11.2.0.3 with ASM installed but we most databases running as standalones instead of RAC databases. Recently we decide to convert all databases to RAC databases. The database version is 11.2.0.3. They are already running on the RAC home and on the grid cluster.
Oracle provide 4 ways to convert a standalone database to a RAC database: DBCA, manual, RConfig and OEM.
With DBCA, we can backup the database as a template and then create a RAC database with the template. This way is simple but the long down time is not acceptable in most environments so we quickly excluded as a choice.
I found the manual way is most reliable. You know exactly what each step does. It has the minimum downtime. The drawback is you have to do the manual steps for each database. If you have 100 database, you might not like to repeat the steps again, again and again.
The RConfig is the most applicable way. All we need to do is to create an xml file and run a command. The benefit is once you create the first xml file, you can reuse it with very minimum modification. Sometimes you only need to modify the name of the database. You can run multiple rconfig at the same time.
OEM may appears like the easiest way as simple as a few clicks. But it turned out that I never make it to work. I actually never spend the time to fix the issue. It's not worth the time to troubleshoot OEM. I just went with RConfig as it's easy enough.
In this article, we provides 2 samples. First one with manual steps and the second one use RConfig. Our assumption is that the database already running in the RAC home. If your database is not currently in the RAC home, you need to move it there first. You can do it with RMAN or datapump, or manually copy the files and recreate the control file. If your database is not in the same version, you might want to upgrade or patch to the same patch level before you move your database.
1.2 Manual Conversion of Standalone database to RAC
1. Create a temporary PFILE from the SPFILE and add cluster parameters.
CREATE PFILE='/u03/oracle/wangm/pfile' from spfile;
Edit the pfile created and add the following RAC related parameters:
cluster_database_instances=2
cluster_database=true
E91DEV1.instance_number=1
E91DEV2.instance_number=2
E91DEV1.thread=1
E91DEV2.thread=2
E91DEV1.undo_tablespace='PSUNDO01'
E91DEV2.undo_tablespace='PSUNDO02'
E91DEV1.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr1-vip)(PORT=1521))'
E91DEV2.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr2-vip)(PORT=1521))'
Note you need to delete the two parameters:
undo_tablespace='PSUNDO01'
local_listener=’ DEVCLSTR1_LOCAL’
1. Create additional UNDO tablespace.
create undo tablespace psundo02 datafile '+DATA';
1. Create redo threads and enable the second thread.
E91DEV> alter database add logfile thread 2 group 5 ('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 6 ('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 7 ('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 8 ('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database enable public thread 2;
Database altered.
E91DEV>
1. Copy the Oracle password file and init file from the initial node, or from the node from which you are working, to the corresponding location on the additional nodes on which the cluster database will have an instance. Replace the ORACLE_SID name in each password file appropriately for each additional instance.
devclstr1-/u03/oracle/wangm>cd $ORACLE_HOME/dbs
devclstr1-/u01/app/oracle/product/11.2/dbs>mv orapwE91DEV orapwE91DEV1
devclstr1-/u01/app/oracle/product/11.2/dbs>mv initE91DEV.ora initE91DEV1.ora
devclstr1-/u01/app/oracle/product/11.2/dbs>scp -p orapwE91DEV1 devclstr2:/u01/app/oracle/product/11.2/dbs/orapwE91DEV2
orapwE91DEV1 100% 2560 2.5KB/s 00:00
devclstr1-/u01/app/oracle/product/11.2/dbs>scp -p initE91DEV1.ora devclstr2:/u01/app/oracle/product/11.2/dbs/initE91DEV2.ora
initE91DEV1.ora
1. Create the SPFILE from the updated PFILE.
E91DEV> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
E91DEV> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
devclstr1-/u03/oracle/wangm>export ORACLE_SID=E91DEV1
devclstr1-/u03/oracle/wangm>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Wed Jun 25 15:52:16 2014
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance.
none> create spfile='+DATA/E91DEV/spfileE91DEV.ora' from pfile='/u03/oracle/wangm/pfile';
File created.
none> startup
ORACLE instance started.
Total System Global Area 2137886720 bytes
Fixed Size 2230072 bytes
Variable Size 654313672 bytes
Database Buffers 1442840576 bytes
Redo Buffers 38502400 bytes
Database mounted.
Database opened.
none>
1. On the local node, run catclust.sql. This command creates the dictionary views needed for Oracle RAC databases. For example:
SQL> start ?/rdbms/admin/catclust.sql
1. Add the configuration for the Oracle RAC database and its instance-to-node mapping using SRVCTL.
devclstr1-/u03/oracle/wangm>srvctl add database -d E91DEV -o /u01/app/oracle/product/11.2
devclstr1-/u03/oracle/wangm>srvctl add instance -d E91DEV -i E91DEV1 -n devclstr1
devclstr1-/u03/oracle/wangm>srvctl add instance -d E91DEV -i E91DEV2 -n devclstr2
devclstr1-/u03/oracle/wangm>
1. Start the database using SRVCTL.
devclstr1-/u03/oracle/wangm>srvctl start database -d E91DEV
devclstr1-/u03/oracle/wangm>srvctl status database -d E91DEV
Instance E91DEV1 is running on node devclstr1
Instance E91DEV2 is running on node devclstr2
devclstr1-/u03/oracle/wangm>
After starting the database with SRVCTL, your conversion process is complete and, for example, you can run the following SQL statement to see the status of all the instances in your Oracle RAC database:
E91DEV> select * from v$active_instances;
INST_NUMBER INST_NAME
----------- ------------------------------------------------------------
1 devclstr1:E91DEV1
2 devclstr2:E91DEV2
E91DEV>
1.3 Converting with RConfig
1. As the user, copy the file $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC_AdminManaged.xml to a working directory and edit the file.
Here is the file E91TRP.xml after modification.
<?xml version="1.0" encoding="UTF-8"?>
<n:RConfig xmlns:n="http://www.oracle.com/rconfig"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.oracle.com/rconfig rconfig.xsd">
<n:ConvertToRAC>
<!-- Verify does a precheck to ensure all pre-requisites are met, before the conversion is attempted. Allowable values are: YES|NO|ONLY -->
<n:Convert verify="ONLY">
<!--Specify current OracleHome of non-rac database for SourceDBHome -->
<n:SourceDBHome>/u01/app/oracle/product/11.2</n:SourceDBHome>
<!--Specify OracleHome where the rac database should be configured. It can be same as SourceDBHome -->
<n:TargetDBHome>/u01/app/oracle/product/11.2</n:TargetDBHome>
<!--Specify SID of non-rac database and credential. User with sysdba role is required to perform conversion -->
<n:SourceDBInfo SID="E91TPR">
<n:Credentials>
<n:User>sys</n:User>
<n:Password>password</n:Password>
<n:Role>sysdba</n:Role>
</n:Credentials>
</n:SourceDBInfo>
<!--Specify the list of nodes that should have rac instances running for the Admin Managed Cluster Database. LocalNode should be the first node in this nodelist. --> <n:NodeList>
<n:Node name="devclstr3"/>
<n:Node name="devclstr2"/>
</n:NodeList>
<!--Specify RacOneNode along with servicename to convert database to RACOne Node -->
<!--n:RacOneNode servicename="salesrac1service"/-->
<!--Instance Prefix tag is optional starting with 11.2. If left empty, it is derived from db_unique_name.-->
<n:InstancePrefix></n:InstancePrefix>
<!-- Listener details are no longer needed starting 11.2. Database is registered with default listener and SCAN listener running from Oracle Grid Infrastructure home. -->
<!--Specify the type of storage to be used by rac database. Allowable values are CFS|ASM. The non-rac database should have same storage type. ASM credentials are no needed for conversion. -->
<n:SharedStorage type="ASM">
<!--Specify Database Area Location to be configured for rac database.If this field is left empty, current storage will be used for rac database. For CFS, this field will have directory path. -->
<n:TargetDatabaseArea></n:TargetDatabaseArea>
<!--Specify Fast Recovery Area to be configured for rac database. If this field is left empty, current recovery area of non-rac database will be configured for rac database. If current database is not using recovery Area, the resulting rac database will not have a recovery area. -->
<n:TargetFlashRecoveryArea></n:TargetFlashRecoveryArea>
</n:SharedStorage>
</n:Convert>
</n:ConvertToRAC>
</n:RConfig>
1. Run :
First we set the verify in the xml to “ONLY” to make sure we can convert.
<n:Convert verify="ONLY">
devclstr1-/u03/oracle/wangm>rconfig E91TPR.xml
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
<Convert>
<Response>
<Result code="0" >
Operation Succeeded
</Result>
</Response>
<ReturnValue type="object">
There is no return value for this step </ReturnValue>
</Convert>
</ConvertToRAC></RConfig>
devclstr1-/u03/oracle/wangm>
Then we set it to “YES” to perform the convert.
<n:Convert verify="YES">
devclstr1-/u03/oracle/wangm>rconfig E91TPR.xml
Converting Database "E91TPR" to Cluster Database. Target Oracle Home: /u01/app/oracle/product/11.2. Database Role: PRIMARY.
Setting Data Files and Control Files
Adding Database Instances
Adding Redo Logs
Enabling threads for all Database Instances
Setting TEMP tablespace
Adding UNDO tablespaces
Adding Trace files
Setting Fast Recovery Area
Updating Oratab
Creating Password file(s)
Configuring Listeners
Configuring related CRS resources
Starting Cluster Database
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
<Convert>
<Response>
<Result code="0" >
Operation Succeeded
</Result>
</Response>
<ReturnValue type="object">
<Oracle_Home>
/u01/app/oracle/product/11.2
</Oracle_Home>
<Database type="ADMIN_MANAGED" >
<InstanceList>
<Instance SID="E91TPR1" Node="devclstr1" >
</Instance>
<Instance SID="E91TPR2" Node="devclstr2" >
</Instance>
</InstanceList>
</Database> </ReturnValue>
</Convert>
</ConvertToRAC></RConfig>
devclstr1-/u03/oracle/wangm>
1.4 Post Conversion steps.
1) Update crontab if applicable.
2) Remove and add the database target in OEM.
3) Modify local listener setting.
alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr1-vip)(PORT=1521))' sid='E91TPR1';
alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr2-vip)(PORT=1521))' sid='E91TPR2';
4) If using RConfig, the tool automatically name the new undo tablespace to UNDOTBS2. To change it, do the following.
sys@E91TPR2> alter tablespace UNDOTBS2 rename to PSUNDO02;
Tablespace altered.
sys@E91TPR2> alter system set undo_tablespace=PSUNDO02 sid='E91TPR2';
System altered.
sys@E91TPR2> show parameter undo
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
undo_management string AUTO
undo_retention integer 10800
undo_tablespace string PSUNDO02
sys@E91TPR2>
1.5 Notes
1) Oracle supplies two sample xml files, one for Admin managed and one for policy managed. Since we are using the Admin managed xml here. The policy managed one is similiar.
2) We can create a new undo table space with desired name, swith the undo_tablespace parameter and then drop the undo tablespace created by RConfig. I've noticed if the psundo02 table space exists before running RConfig, then it's smart enough to use it without creating the default undo tablespace.
Convert Single Instance Databases to RAC
1.1 Overview
We have Oracle grid infrastructure 11.2.0.3 with ASM installed but we most databases running as standalones instead of RAC databases. Recently we decide to convert all databases to RAC databases. The database version is 11.2.0.3. They are already running on the RAC home and on the grid cluster.
Oracle provide 4 ways to convert a standalone database to a RAC database: DBCA, manual, RConfig and OEM.
With DBCA, we can backup the database as a template and then create a RAC database with the template. This way is simple but the long down time is not acceptable in most environments so we quickly excluded as a choice.
I found the manual way is most reliable. You know exactly what each step does. It has the minimum downtime. The drawback is you have to do the manual steps for each database. If you have 100 database, you might not like to repeat the steps again, again and again.
The RConfig is the most applicable way. All we need to do is to create an xml file and run a command. The benefit is once you create the first xml file, you can reuse it with very minimum modification. Sometimes you only need to modify the name of the database. You can run multiple rconfig at the same time.
OEM may appears like the easiest way as simple as a few clicks. But it turned out that I never make it to work. I actually never spend the time to fix the issue. It's not worth the time to troubleshoot OEM. I just went with RConfig as it's easy enough.
In this article, we provides 2 samples. First one with manual steps and the second one use RConfig. Our assumption is that the database already running in the RAC home. If your database is not currently in the RAC home, you need to move it there first. You can do it with RMAN or datapump, or manually copy the files and recreate the control file. If your database is not in the same version, you might want to upgrade or patch to the same patch level before you move your database.
1.2 Manual Conversion of Standalone database to RAC
1. Create a temporary PFILE from the SPFILE and add cluster parameters.
CREATE PFILE='/u03/oracle/wangm/pfile' from spfile;
Edit the pfile created and add the following RAC related parameters:
cluster_database_instances=2
cluster_database=true
E91DEV1.instance_number=1
E91DEV2.instance_number=2
E91DEV1.thread=1
E91DEV2.thread=2
E91DEV1.undo_tablespace='PSUNDO01'
E91DEV2.undo_tablespace='PSUNDO02'
E91DEV1.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr1-vip)(PORT=1521))'
E91DEV2.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr2-vip)(PORT=1521))'
Note you need to delete the two parameters:
undo_tablespace='PSUNDO01'
local_listener=’ DEVCLSTR1_LOCAL’
1. Create additional UNDO tablespace.
create undo tablespace psundo02 datafile '+DATA';
1. Create redo threads and enable the second thread.
E91DEV> alter database add logfile thread 2 group 5 ('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 6 ('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 7 ('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 8 ('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database enable public thread 2;
Database altered.
E91DEV>
1. Copy the Oracle password file and init file from the initial node, or from the node from which you are working, to the corresponding location on the additional nodes on which the cluster database will have an instance. Replace the ORACLE_SID name in each password file appropriately for each additional instance.
devclstr1-/u03/oracle/wangm>cd $ORACLE_HOME/dbs
devclstr1-/u01/app/oracle/product/11.2/dbs>mv orapwE91DEV orapwE91DEV1
devclstr1-/u01/app/oracle/product/11.2/dbs>mv initE91DEV.ora initE91DEV1.ora
devclstr1-/u01/app/oracle/product/11.2/dbs>scp -p orapwE91DEV1 devclstr2:/u01/app/oracle/product/11.2/dbs/orapwE91DEV2
orapwE91DEV1 100% 2560 2.5KB/s 00:00
devclstr1-/u01/app/oracle/product/11.2/dbs>scp -p initE91DEV1.ora devclstr2:/u01/app/oracle/product/11.2/dbs/initE91DEV2.ora
initE91DEV1.ora
1. Create the SPFILE from the updated PFILE.
E91DEV> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
E91DEV> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
devclstr1-/u03/oracle/wangm>export ORACLE_SID=E91DEV1
devclstr1-/u03/oracle/wangm>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Wed Jun 25 15:52:16 2014
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance.
none> create spfile='+DATA/E91DEV/spfileE91DEV.ora' from pfile='/u03/oracle/wangm/pfile';
File created.
none> startup
ORACLE instance started.
Total System Global Area 2137886720 bytes
Fixed Size 2230072 bytes
Variable Size 654313672 bytes
Database Buffers 1442840576 bytes
Redo Buffers 38502400 bytes
Database mounted.
Database opened.
none>
1. On the local node, run catclust.sql. This command creates the dictionary views needed for Oracle RAC databases. For example:
SQL> start ?/rdbms/admin/catclust.sql
1. Add the configuration for the Oracle RAC database and its instance-to-node mapping using SRVCTL.
devclstr1-/u03/oracle/wangm>srvctl add database -d E91DEV -o /u01/app/oracle/product/11.2
devclstr1-/u03/oracle/wangm>srvctl add instance -d E91DEV -i E91DEV1 -n devclstr1
devclstr1-/u03/oracle/wangm>srvctl add instance -d E91DEV -i E91DEV2 -n devclstr2
devclstr1-/u03/oracle/wangm>
1. Start the database using SRVCTL.
devclstr1-/u03/oracle/wangm>srvctl start database -d E91DEV
devclstr1-/u03/oracle/wangm>srvctl status database -d E91DEV
Instance E91DEV1 is running on node devclstr1
Instance E91DEV2 is running on node devclstr2
devclstr1-/u03/oracle/wangm>
After starting the database with SRVCTL, your conversion process is complete and, for example, you can run the following SQL statement to see the status of all the instances in your Oracle RAC database:
E91DEV> select * from v$active_instances;
INST_NUMBER INST_NAME
----------- ------------------------------------------------------------
1 devclstr1:E91DEV1
2 devclstr2:E91DEV2
E91DEV>
1.3 Converting with RConfig
1. As the user, copy the file $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC_AdminManaged.xml to a working directory and edit the file.
Here is the file E91TRP.xml after modification.
<?xml version="1.0" encoding="UTF-8"?>
<n:RConfig xmlns:n="http://www.oracle.com/rconfig"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.oracle.com/rconfig rconfig.xsd">
<n:ConvertToRAC>
<!-- Verify does a precheck to ensure all pre-requisites are met, before the conversion is attempted. Allowable values are: YES|NO|ONLY -->
<n:Convert verify="ONLY">
<!--Specify current OracleHome of non-rac database for SourceDBHome -->
<n:SourceDBHome>/u01/app/oracle/product/11.2</n:SourceDBHome>
<!--Specify OracleHome where the rac database should be configured. It can be same as SourceDBHome -->
<n:TargetDBHome>/u01/app/oracle/product/11.2</n:TargetDBHome>
<!--Specify SID of non-rac database and credential. User with sysdba role is required to perform conversion -->
<n:SourceDBInfo SID="E91TPR">
<n:Credentials>
<n:User>sys</n:User>
<n:Password>password</n:Password>
<n:Role>sysdba</n:Role>
</n:Credentials>
</n:SourceDBInfo>
<!--Specify the list of nodes that should have rac instances running for the Admin Managed Cluster Database. LocalNode should be the first node in this nodelist. --> <n:NodeList>
<n:Node name="devclstr3"/>
<n:Node name="devclstr2"/>
</n:NodeList>
<!--Specify RacOneNode along with servicename to convert database to RACOne Node -->
<!--n:RacOneNode servicename="salesrac1service"/-->
<!--Instance Prefix tag is optional starting with 11.2. If left empty, it is derived from db_unique_name.-->
<n:InstancePrefix></n:InstancePrefix>
<!-- Listener details are no longer needed starting 11.2. Database is registered with default listener and SCAN listener running from Oracle Grid Infrastructure home. -->
<!--Specify the type of storage to be used by rac database. Allowable values are CFS|ASM. The non-rac database should have same storage type. ASM credentials are no needed for conversion. -->
<n:SharedStorage type="ASM">
<!--Specify Database Area Location to be configured for rac database.If this field is left empty, current storage will be used for rac database. For CFS, this field will have directory path. -->
<n:TargetDatabaseArea></n:TargetDatabaseArea>
<!--Specify Fast Recovery Area to be configured for rac database. If this field is left empty, current recovery area of non-rac database will be configured for rac database. If current database is not using recovery Area, the resulting rac database will not have a recovery area. -->
<n:TargetFlashRecoveryArea></n:TargetFlashRecoveryArea>
</n:SharedStorage>
</n:Convert>
</n:ConvertToRAC>
</n:RConfig>
1. Run :
First we set the verify in the xml to “ONLY” to make sure we can convert.
<n:Convert verify="ONLY">
devclstr1-/u03/oracle/wangm>rconfig E91TPR.xml
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
<Convert>
<Response>
<Result code="0" >
Operation Succeeded
</Result>
</Response>
<ReturnValue type="object">
There is no return value for this step </ReturnValue>
</Convert>
</ConvertToRAC></RConfig>
devclstr1-/u03/oracle/wangm>
Then we set it to “YES” to perform the convert.
<n:Convert verify="YES">
devclstr1-/u03/oracle/wangm>rconfig E91TPR.xml
Converting Database "E91TPR" to Cluster Database. Target Oracle Home: /u01/app/oracle/product/11.2. Database Role: PRIMARY.
Setting Data Files and Control Files
Adding Database Instances
Adding Redo Logs
Enabling threads for all Database Instances
Setting TEMP tablespace
Adding UNDO tablespaces
Adding Trace files
Setting Fast Recovery Area
Updating Oratab
Creating Password file(s)
Configuring Listeners
Configuring related CRS resources
Starting Cluster Database
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
<Convert>
<Response>
<Result code="0" >
Operation Succeeded
</Result>
</Response>
<ReturnValue type="object">
<Oracle_Home>
/u01/app/oracle/product/11.2
</Oracle_Home>
<Database type="ADMIN_MANAGED" >
<InstanceList>
<Instance SID="E91TPR1" Node="devclstr1" >
</Instance>
<Instance SID="E91TPR2" Node="devclstr2" >
</Instance>
</InstanceList>
</Database> </ReturnValue>
</Convert>
</ConvertToRAC></RConfig>
devclstr1-/u03/oracle/wangm>
1.4 Post Conversion steps.
1) Update crontab if applicable.
2) Remove and add the database target in OEM.
3) Modify local listener setting.
alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr1-vip)(PORT=1521))' sid='E91TPR1';
alter system set local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr2-vip)(PORT=1521))' sid='E91TPR2';
4) If using RConfig, the tool automatically name the new undo tablespace to UNDOTBS2. To change it, do the following.
sys@E91TPR2> alter tablespace UNDOTBS2 rename to PSUNDO02;
Tablespace altered.
sys@E91TPR2> alter system set undo_tablespace=PSUNDO02 sid='E91TPR2';
System altered.
sys@E91TPR2> show parameter undo
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
undo_management string AUTO
undo_retention integer 10800
undo_tablespace string PSUNDO02
sys@E91TPR2>
1.5 Notes
1) Oracle supplies two sample xml files, one for Admin managed and one for policy managed. Since we are using the Admin managed xml here. The policy managed one is similiar.
2) We can create a new undo table space with desired name, swith the undo_tablespace parameter and then drop the undo tablespace created by RConfig. I've noticed if the psundo02 table space exists before running RConfig, then it's smart enough to use it without creating the default undo tablespace.
Local Listener & Remote Listener
Suppose we have 2-node cluster: host1 and host2, with VIP address host1-vip and host2-vip respectively.
and one RAC database (orcl) running on this cluster; instace 1 (orcl1) on host1, and instance 2 (orcl2) on host2
we have listener_host1 running on host1, and listener_host2 running on host2.
listener_host1 is considered local listener for orcl1 instance, while listener_host2 is considered remote listener for that same orcl1 instance (because the listener in not running on the same machine as the database instance).
similarly, listener_host2 is considered local listener for orcl2 instance, and considered as remote listener for orcl1.
to make this consideration a real configuration, we configure the 2 parameters local_listener and remote_listener for both instances as below:
orcl1.local_listener=(address of listener_host1)
orcl1.remote_listener=(addresses of both listener_host1 and listener_host2)
orcl2.local_listener=(address of listener_host2)
orcl2.remote_listener=(addresses of both listener_host1 and listener_host2)
(as you see, we can simply use both listeners for the remote listener, as a simple configuration. But of course you could have configured orcl1.remote_listener=(addres of listener_host2) only.)
with such configuration, both listeners in the cluster knows about both instances, and about both hosts (statistics about host load, and instance load). and can make decision about forwarding a client connection request to the other node if it's less loaded. Which is the mechanism behind server-side load balancing.
clients are generally configured with tnsnames with both VIP addresses of the 2 hosts (i.e. can connect to either listener). so if a client attempts the connection to the database with the first IP (which is listener_host1), and suppose host1 is a bit loaded that host2, in such case the listener_host1 knows there is another instance orcl2 running on host2 that's less loaded. in such case, listener_host1 sends a redirect packet to the client asking him to transparently reconnect to listener_host2 to establish the database connection.
without such configuration of remote listener, each listener knows only about the local instance, and have nothing to do but connecting the client to the instance running on the same host as the listener. in such case you have only what's called client-side load balancing.
Courtesy : https://forums.oracle.com/thread/1103141
Thanks to Moustafa
11gR2 Clusterware and Grid Home - Need to Know
Purpose |
Scope |
Details |
11gR2 Clusterware Key Facts |
Clusterware Startup Sequence |
Important Log Locations |
Clusterware Resource Status Check |
Clusterware Resource Administration |
OCRCONFIG Options: |
OLSNODES Options |
Cluster Verification Options |
Database - RAC/Scalability Community |
References |
APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.1 [Release 11.2]Information in this document applies to any platform.
PURPOSE
The 11gR2 Clusterware has undergone numerous changes since the previous release. For information on the previous release(s), see Note: 259301.1"CRS and 10g Real Application Clusters". This document is intended to go over the 11.2 Clusterware which has some similarities and some differences from the previous version(s).
SCOPE
This document is intended for RAC Database Administrators and Oracle support engineers.
DETAILS
11gR2 Clusterware Key Facts
- 11gR2 Clusterware is required to be up and running prior to installing a 11gR2 Real Application Clusters database.
- The GRID home consists of the Oracle Clusterware and ASM. ASM should not be in a separate home.
- The 11gR2 Clusterware can be installed in "Standalone" mode for ASM and/or "Oracle Restart" single node support. This clusterware is a subset of the full clusterware described in this document.
- The 11gR2 Clusterware can be run by itself or on top of vendor clusterware. See the certification matrix for certified combinations. Ref: Note: 184875.1 "How To Check The Certification Matrix for Real Application Clusters"
- The GRID Home and the RAC/DB Home must be installed in different locations.
- The 11gR2 Clusterware requires a shared OCR files and voting files. These can be stored on ASM or a cluster filesystem.
- The OCR is backed up automatically every 4 hours to <GRID_HOME>/cdata/<clustername>/ and can be restored via ocrconfig.
- The voting file is backed up into the OCR at every configuration change and can be restored via crsctl.
- The 11gR2 Clusterware requires at least one private network for inter-node communication and at least one public network for external communication. Several virtual IPs need to be registered with DNS. This includes the node VIPs (one per node), SCAN VIPs (three). This can be done manually via your network administrator or optionally you could configure the "GNS" (Grid Naming Service) in the Oracle clusterware to handle this for you (note that GNS requires its own VIP).
- A SCAN (Single Client Access Name) is provided to clients to connect to. For more informantion on SCAN see Note: 887522.1
- The root.sh script at the end of the clusterware installation starts the clusterware stack. For information on troubleshooting root.sh issues seeNote: 1053970.1
- Only one set of clusterware daemons can be running per node.
- On Unix, the clusterware stack is started via the init.ohasd script referenced in /etc/inittab with "respawn".
- A node can be evicted (rebooted) if a node is deemed to be unhealthy. This is done so that the health of the entire cluster can be maintained. For more information on this see: Note: 1050693.1 "Troubleshooting 11.2 Clusterware Node Evictions (Reboots)"
- Either have vendor time synchronization software (like NTP) fully configured and running or have it not configured at all and let CTSS handle time synchonization. See Note: 1054006.1 for more information.
- If installing DB homes for a lower version, you will need to pin the nodes in the clusterware or you will see ORA-29702 errors. See Note 946332.1 and Note:948456.1 for more information.
- The clusterware stack can be started by either booting the machine, running "crsctl start crs" to start the clusterware stack, or by running "crsctl start cluster" to start the clusterware on all nodes. Note that crsctl is in the <GRID_HOME>/bin directory. Note that "crsctl start cluster" will only work if ohasd is running.
- The clusterware stack can be stopped by either shutting down the machine, running "crsctl stop crs" to stop the clusterware stack, or by running "crsctl stop cluster" to stop the clusterware on all nodes. Note that crsctl is in the <GRID_HOME>/bin directory.
- Killing clusterware daemons is not supported.
- Instance is now part of .db resources in "crsctl stat res -t" output, there is no separate .inst resource for 11gR2 instance.
Note that it is also a good idea to follow the RAC Assurance best practices in Note: 810394.1
Clusterware Startup Sequence
The following is the Clusterware startup sequence (image from the "Oracle Clusterware Administration and Deployment Guide):
INIT
-> OHASD-> orarootagent,
oraagent, cssdagent,
cssdmonitor
Orarootagent -> crsd, ctssd, diskmon,
Oraagent -> mdnsd, gipcd, gpnpd, evmd, asm
Cssdagent -> cssd
Cssdmonitor -> cssdmonitorDon't let this picture scare you too much. You aren't responsible for managing all of these processes, that is the Clusterware's job!
Short summary of the startup sequence: INIT spawns init.ohasd (with respawn) which in turn starts the OHASD process (Oracle High Availability Services Daemon). This daemon spawns 4 processes.
Level 1: OHASD Spawns:
- cssdagent - Agent responsible for spawning CSSD.
- orarootagent - Agent responsible for managing all root owned ohasd resources.
- oraagent - Agent responsible for managing all oracle owned ohasd resources.
- cssdmonitor - Monitors CSSD and node health (along wth the cssdagent).
Level 2: OHASD rootagent spawns:
- CRSD - Primary daemon responsible for managing cluster resources.
- CTSSD - Cluster Time Synchronization Services Daemon
- Diskmon
- ACFS (ASM Cluster File System) Drivers
Level 2: OHASD oraagent spawns:
- MDNSD - Used for DNS lookup
- GIPCD - Used for inter-process and inter-node communication
- GPNPD - Grid Plug & Play Profile Daemon
- EVMD - Event Monitor Daemon
- ASM - Resource for monitoring ASM instances
Level 3: CRSD spawns:
- orarootagent - Agent responsible for managing all root owned crsd resources.
- oraagent - Agent responsible for managing all oracle owned crsd resources.
Level 4: CRSD rootagent spawns:
- Network resource - To monitor the public network
- SCAN VIP(s) - Single Client Access Name Virtual IPs
- Node VIPs - One per node
- ACFS Registery - For mounting ASM Cluster File System
- GNS VIP (optional) - VIP for GNS
Level 4: CRSD oraagent spawns:
- ASM Resouce - ASM Instance(s) resource
- Diskgroup - Used for managing/monitoring ASM diskgroups.
- DB Resource - Used for monitoring and managing the DB and instances
- SCAN Listener - Listener for single client access name, listening on SCAN VIP
- Listener - Node listener listening on the Node VIP
- Services - Used for monitoring and managing services
- ONS - Oracle Notification Service
- eONS - Enhanced Oracle Notification Service
- GSD - For 9i backward compatibility
- GNS (optional) - Grid Naming Service - Performs name resolution
This image shows the various levels more clearly:
Startup Sequence:
RAC 11gR2 BackGround process startup sequence
================================
1. START / STOP CLUSTERWARE CRS:
================================
To start and stop CRS when the machine starts or shutdown, on unix there are rc scripts in place.
/etc/init.d/init.crs start
/etc/init.d/init.crs stop
/etc/init.d/init.crs enable
/etc/init.d/init.crs disable
You can also, as root, manually start, stop, enable or disable the services with:
# crsctl start crs
# crsctl stop crs
# crsctl enable crs
# crsctl disable crs
On a unix system, you may find the following in the /etc/inittab file.
# cat /etc/inittab | grep crs
h3:35:respawn:/etc/init.d/init.crsd run > /dev/null 2>&1 </dev/null
# cat /etc/inittab | grep evmd
h1:35:respawn:/etc/init.d/init.evmd run > /dev/null 2>&1 </dev/null
# cat /etc/inittab | grep css
h2:35:respawn:/etc/init.d/init.cssd fatal > /dev/null 2>&1 </dev/null
/etc/init.d> ls -al *init*
init.crs
init.crsd
init.cssd
init.evmd
# cat /etc/inittab
..
..
h1:35:respawn:/etc/init.d/init.evmd run > /dev/null 2>&1 </dev/null
h2:35:respawn:/etc/init.d/init.cssd fatal > /dev/null 2>&1 </dev/null
h3:35:respawn:/etc/init.d/init.crsd run > /dev/null 2>&1 </dev/null
===================================
2. STARTING / STOPPING THE CLUSTER:
===================================
-- Stopping the Cluster:
Before you shut down any processes that are monitored by Enterprise Manager Grid Control, set a blackout in
Grid Control for the processes that you intend to shut down. This is necessary so that the availability records
for these processes indicate that the shutdown was planned downtime, rather than an unplanned system outage.
Shut down all Oracle RAC instances on all nodes. To shut down all Oracle RAC instances for a database,
enter the following command, where db_name is the name of the database:
$ ORACLE_HOME/bin/srvctl stop database -d db_name
Shut down all ASM instances on all nodes. To shut down an ASM instance, enter the following command,
where node is the name of the node where the ASM instance is running:
$ ORACLE_HOME/bin/srvctl stop asm -n node
Stop all node applications on all nodes. To stop node applications running on a node, enter the following command,
where node is the name of the node where the applications are running
$ ORACLE_HOME/bin/srvctl stop nodeapps -n node
Log in as the root user, and shut down the Oracle Clusterware or CRS process by entering the following command
on all nodes:
# CRS_HOME/bin/crsctl stop crs # as root
-- Starting the Cluster:
# CRS_HOME/bin/crsctl start crs # as root
$ ORACLE_HOME/bin/srvctl start nodeapps -n node
$ ORACLE_HOME/bin/srvctl start asm -n node
$ ORACLE_HOME/bin/srvctl start database -d db_name # will start all instances of the Database
================================================
3. CRS_STAT -t command OR SRVCTL STATUS command:
================================================
CRS_STAT example:
-----------------
Viewing the status of instances, database, nodeapps:
For example, to list the status of the apps in the cluster, use crs_stat:
/home/oracle-->$CRS_HOME/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE aix1
ora....x1.lsnr application ONLINE ONLINE aix1
ora....ix1.gsd application ONLINE ONLINE aix1
ora....ix1.ons application ONLINE ONLINE aix1
ora....ix1.vip application ONLINE ONLINE aix1
ora....SM2.asm application ONLINE ONLINE aix2
ora....x2.lsnr application ONLINE ONLINE aix2
ora....ix2.gsd application ONLINE ONLINE aix2
ora....ix2.ons application ONLINE ONLINE aix2
ora....ix2.vip application ONLINE ONLINE aix2
ora....test.db application ONLINE ONLINE aix1
ora....x1.inst application ONLINE ONLINE aix1
ora....x2.inst application ONLINE ONLINE aix2
/home/oracle-->
SRVCTL example:
---------------
$ srvctl status nodeapps -n mynewserver
VIP is running on node: mynewserver
GSD is running on node: mynewserver
Listener is not running on node: mynewserver
ONS daemon is running on node: mynewserver
CRSCTL example:
---------------
Checking crs on the node:
# crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy
Checking crs clusterwide:
crsctl check cluster
# crsctl check cluster
node1-pub ONLINE
node2-pub ONLINE
===================
4. OTHER EXAMPLES:
===================
Example 1. Bring up the MYSID1 instance of the MYSID database.
$ srvctl start instance -d MYSID -i MYSID1
Example 2. Stop the MYSID database: all its instances and all its services, on all nodes.
$ srvctl stop database -d MYSID
Example 3. Stop the nodeapps on the myserver node. NB: Instances and services also stop.
$ srvctl stop nodeapps -n myserver
Example 4. Add the MYSID3 instance, which runs on the myserver node, to the MYSID
clustered database.
$ srvctl add instance -d MYSID -i MYSID3 -n myserver
Example 4. Add a new node, the mynewserver node, to a cluster.
$ srvctl add nodeapps -n mynewserver -o $ORACLE_HOME -A 10.10.10.100/255.255.255.0/eth1
(The -A flag precedes an address specification.)
Example 5. To change the VIP (virtual IP) on a RAC node, use the command
$ srvctl modify nodeapps -A new_address
Example 6. Disable the ASM instance on myserver for maintenance.
$ srvctl disable asm -n myserver
Some other examples:
--------------------
$ srvctl add instance -d -i -n
$ srvctl add service -d -s -r “” [-a ""] [-P ]
$ srvctl add service -d -s -u {-r “” | -a “”}
$ srvctl add nodeapps -n -o -A /netmask[/if1[|if2|...]]
$ srvctl add asm -n -i -o [-p ]
$ srvctl config database
$ srvctl config database -d [-a] [-t]
$ srvctl config service -d [-s ] [-a] [-S ]
$ srvctl config nodeapps -n [-a] [-g] [-o] [-s] [-l]
$ srvctl config asm -n
$ srvctl config listener -n
$ srvctl disable database -d
$ srvctl disable instance -d -i “”
$ srvctl disable service -d -s “” [-i ]
$ srvctl disable asm -n [-i ]
$ srvctl enable database -d
$ srvctl enable instance -d -i “”
$ srvctl enable service -d -s “” [-i ]
$ srvctl enable asm -n [-i ]
$ srvctl getenv database -d [-t ""]
$ srvctl getenv instance -d -i [-t ""]
$ srvctl getenv service -d -s [-t ""]
$ srvctl getenv nodeapps -n [-t ""]
$ srvctl modify database -d [-n <db_name] [-o ] [-m ] [-p ] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}] [-s ] [-y {AUTOMATIC | MANUAL}]
$ srvctl modify instance -d -i -n
$ srvctl modify instance -d -i {-s | -r}
$ srvctl modify service -d -s -i -t [-f]
$ srvctl modify service -d -s -i -r [-f]
$ srvctl modify service -d -s -n -i [-a ] [-f]
$ srvctl modify asm -n -i [-o ] [-p ]
$ srvctl relocate service -d -s -i -t [-f]
$ srvctl remove database -d [-f]
$ srvctl remove instance -d -i [-f]
$ srvctl remove service -d -s [-i ] [-f]
$ srvctl remove nodeapps -n “” [-f]
$ srvctl remove asm -n [-i ] [-f]
$ srvctl setenv database -d {-t =[,=,...] | -T =}
$ srvctl setenv instance -d [-i ] {-t “=[,=,...]” | -T “=”}
$ srvctl setenv service -d [-s ] {-t “=[,=,...]” | -T “=”}
$ srvctl setenv nodeapps -n {-t “=[,=,...]” | -T “=”}
$ srvctl start database -d [-o ] [-c | -q]
$ srvctl start instance -d -i “” [-o ] [-c | -q]
$ srvctl start service -d [-s "" [-i ]] [-o ] [-c | -q]
$ srvctl start nodeapps -n
$ srvctl start asm -n [-i ] [-o ] [-c | -q]
$ srvctl start listener -n [-l ]
$ srvctl status database -d [-f] [-v] [-S ]
$ srvctl status instance -d -i “” [-f] [-v] [-S ]
$ srvctl status service -d [-s ""] [-f] [-v] [-S ]
$ srvctl status nodeapps -n
$ srvctl status asm -n
$ srvctl stop database -d [-o ] [-c | -q]
$ srvctl stop instance -d -i “” [-o ] [-c | -q]
$ srvctl stop service -d [-s "" [-i ]] [-c | -q] [-f]
$ srvctl stop nodeapps -n
$ srvctl stop asm -n [-i ] [-o ] [-c | -q]
$ srvctl stop listener -n [-l ]
$ srvctl unsetenv database -d -t “”
$ srvctl unsetenv instance -d [-i ] -t “”
$ srvctl unsetenv service -d [-s ] -t “”
$ srvctl unsetenv nodeapps -n -t “”
==========================================
OCR & VOTING DISK LOCATION
==========================================
oracle@b11-stg1dbms-101:~->ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2928
Available space (kbytes) : 259192
ID : 2064459122
Device/File Name : /oracle/ocr/ocr
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
oracle@b11-stg1dbms-101:~->crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 9ed4c391f64d4f07bf3fd473f5b03473 (/oracle/ocr/vdsk) []
Located 1 voting disk(s).
oracle@b11-stg1dbms-101:~->cd /oracle/ocr/ocr
bash: cd: /oracle/ocr/ocr: Not a directory
oracle@b11-stg1dbms-101:~->cd /oracle/ocr/
oracle@b11-stg1dbms-101:/oracle/ocr->ls -ltr ocr
-rw-r----- 1 oracle dba 272756736 Sep 23 20:47 ocr
oracle@b11-stg1dbms-101:/oracle/ocr->cd /oracle/ocr/
oracle@b11-stg1dbms-101:/oracle/ocr->ls -ltr vdsk*
-rw-r----- 1 oracle dba 21004288 Sep 23 23:11 vdsk
oracle@b11-stg1dbms-101:/oracle/ocr->
********************************************************************************
Courtesy to Apun's Blog (http://apunhiran.blogspot.in/2009/04/what-is-voting-disk-split-brain.html)
********************************************************************************
What is Voting Disk & Split Brain Syndrome in RAC
Voting Disk
Oracle Clusterware uses the voting disk to determine which instances are members of a cluster. The voting disk must reside on a shared disk. Basically all nodes in the RAC cluster register their heart-beat information on thes voting disks. The number decides the number of active nodes in the RAC cluster. These are also used for checking the availability of instances in RAC and remove the unavailable nodes out of the cluster. It helps in preventing split-brain condition and keeps database information intact. The split brain syndrome and its affects and how it has been managed in oracle is mentioned below.
For high availability, Oracle recommends that you have a minimum of three voting disks. If you configure a single voting disk, then you should use external mirroring to provide redundancy. You can have up to 32 voting disks in your cluster. What I could understand about the odd value of the number of voting disks is that a noe should see maximun number of voting disk to continue to function, so with 2, if it can see only 1, its not the maximum value but a half value of voting disk. I am still trying to search more on this concept.
Split Brain Syndrome:
In a Oracle RAC environment all the instances/servers communicate with each other using high-speed interconnects on the private network. This private network interface or interconnect are redundant and are only used for inter-instance oracle data block transfers. Now talking about split-brain concept with respect to oracle rac systems, it occurs when the instance members in a RAC fail to ping/connect to each other via this private interconnect, but the servers are all pysically up and running and the database instance on each of these servers is also running. These individual nodes are running fine and can conceptually accept user connections and work independently. So basically due to lack of commincation the instance thinks that the other instance that it is not able to connect is down and it needs to do something about the situation. The problem is if we leave these instance running, the sane block might get read, updated in these individual instances and there would be data integrity issue, as the blocks changed in one instance, will not be locked and could be over-written by another instance. Oracle has efficiently implemented check for the split brain syndrome.
What does RAC do incase node becomes inactive:
In RAC if any node becomes inactive, or if other nodes are unable to ping/connect to a node in the RAC, then the node which first detects that one of the node is not accessible, it will evict that node from the RAC group. e.g. there are 4 nodes in a rac instance, and node 3 becomes unavailable, and node 1 tries to connect to node 3 and finds it not responding, then node 1 will evict node 3 out of the RAC groups and will leave only Node1, Node2 & Node4 in the RAC group to continue functioning.
The split brain concepts can become more complicated in large RAC setups. For example there are 10 RAC nodes in a cluster. And say 4 nodes are not able to communicate with the other 6. So there are 2 groups formed in this 10 node RAC cluster ( one group of 4 nodes and other of 6 nodes). Now the nodes will quickly try to affirm their membership by locking controlfile, then the node that lock the controlfile will try to check the votes of the other nodes. The group with the most number of active nodes gets the preference and the others are evicted. Moreover, I have seen this node eviction issue with only 1 node getting evicted and the rest function fine, so I cannot really testify that if thats how it work by experience, but this is the theory behind it.
When we see that the node is evicted, usually oracle rac will reboot that node and try to do a cluster reconfiguration to include back the evicted node.
You will see oracle error: ORA-29740, when there is a node eviction in RAC. There are many reasons for a node eviction like heart beat not received by the controlfile, unable to communicate with the clusterware etc.
A good metalink note on understanding node eviction and how to address is Note ID: 219361.1
The CSS (Cluster Synchronization Service) daemon in the clusterware maintains the heart beat to the voting disk.
===================================
Thanks Osama : http://osamamustafa.blogspot.in/2012/06/dealing-with-ocr-oracle-cluster.html
Oracle Clusterware uses the voting disk to determine which instances are members of a cluster. The voting disk must reside on a shared disk. Basically all nodes in the RAC cluster register their heart-beat information on thes voting disks. The number decides the number of active nodes in the RAC cluster. These are also used for checking the availability of instances in RAC and remove the unavailable nodes out of the cluster. It helps in preventing split-brain condition and keeps database information intact. The split brain syndrome and its affects and how it has been managed in oracle is mentioned below.
For high availability, Oracle recommends that you have a minimum of three voting disks. If you configure a single voting disk, then you should use external mirroring to provide redundancy. You can have up to 32 voting disks in your cluster. What I could understand about the odd value of the number of voting disks is that a noe should see maximun number of voting disk to continue to function, so with 2, if it can see only 1, its not the maximum value but a half value of voting disk. I am still trying to search more on this concept.
Split Brain Syndrome:
In a Oracle RAC environment all the instances/servers communicate with each other using high-speed interconnects on the private network. This private network interface or interconnect are redundant and are only used for inter-instance oracle data block transfers. Now talking about split-brain concept with respect to oracle rac systems, it occurs when the instance members in a RAC fail to ping/connect to each other via this private interconnect, but the servers are all pysically up and running and the database instance on each of these servers is also running. These individual nodes are running fine and can conceptually accept user connections and work independently. So basically due to lack of commincation the instance thinks that the other instance that it is not able to connect is down and it needs to do something about the situation. The problem is if we leave these instance running, the sane block might get read, updated in these individual instances and there would be data integrity issue, as the blocks changed in one instance, will not be locked and could be over-written by another instance. Oracle has efficiently implemented check for the split brain syndrome.
What does RAC do incase node becomes inactive:
In RAC if any node becomes inactive, or if other nodes are unable to ping/connect to a node in the RAC, then the node which first detects that one of the node is not accessible, it will evict that node from the RAC group. e.g. there are 4 nodes in a rac instance, and node 3 becomes unavailable, and node 1 tries to connect to node 3 and finds it not responding, then node 1 will evict node 3 out of the RAC groups and will leave only Node1, Node2 & Node4 in the RAC group to continue functioning.
The split brain concepts can become more complicated in large RAC setups. For example there are 10 RAC nodes in a cluster. And say 4 nodes are not able to communicate with the other 6. So there are 2 groups formed in this 10 node RAC cluster ( one group of 4 nodes and other of 6 nodes). Now the nodes will quickly try to affirm their membership by locking controlfile, then the node that lock the controlfile will try to check the votes of the other nodes. The group with the most number of active nodes gets the preference and the others are evicted. Moreover, I have seen this node eviction issue with only 1 node getting evicted and the rest function fine, so I cannot really testify that if thats how it work by experience, but this is the theory behind it.
When we see that the node is evicted, usually oracle rac will reboot that node and try to do a cluster reconfiguration to include back the evicted node.
You will see oracle error: ORA-29740, when there is a node eviction in RAC. There are many reasons for a node eviction like heart beat not received by the controlfile, unable to communicate with the clusterware etc.
A good metalink note on understanding node eviction and how to address is Note ID: 219361.1
The CSS (Cluster Synchronization Service) daemon in the clusterware maintains the heart beat to the voting disk.
===================================
Thanks Osama : http://osamamustafa.blogspot.in/2012/06/dealing-with-ocr-oracle-cluster.html
Dealing with OCR Backup ( Oracle Cluster Registery)
OCR calls Oracle Cluster Registry. It stores cluster configuration information. It is also shared disk component. It must be accessed by all nodes in cluster environment.It also keeps information of Which database instance run on which nodes and which service runs on which database.The process daemon OCSSd manages the configuration info in OCR and maintains the changes to cluster in the registry.
There's Two Kind Of OCR Backup :
1-Automatic Backup By Oracle CRS , You can check
$ORA_CRS_HOME/cdata
Or
ocrconfig -showbackup
Want to change The Default Location for automatic backup :
ocrconfig -backuploc
2- Manual Backup
$ocrconfig -export /u04/crs_backup/ocrbackup/exports/OCRFile_expBackup.dmp
$ocrconfig -export /u04/crs_backup/ocrbackup/exports/OCRFile_expBackup.dmp
Recover OCR from Physical Backup(AUTOMATIC):
$ocrconfig -showbackup
$srvctl -stop database -d ORCL (Shutdown all RAC instances and RAC database)
$crsctl stop crs (Shutdown Cluster)
#ocrconfig -restore /u02/apps/crs/cdata/crs/backup00.ocr
$crsctl start crs (After issuing start cluster check status of cluster using 'crs_stat -t')
$srvctl start database -d ORCL (Start Oracle RAC database and RAC instances)
Recover OCR from Physical Backup (MANUAL):
$srvctl -stop database -d ORCL (Shutdown all RAC instances and RAC database)
$crsctl stop crs (Shutdown Cluster)
SAME process should need to repeat for OCR mirror also.
ocrconfig -import /u04/crs_backup/ocrbackup/exports/OCRFile_exp_Backup.dmp (Import metadata of OCR using command)
$crsctl start crs (After issuing start cluster check status of cluster using 'crs_stat -t')
$srvctl start database -d ORCL (Start Oracle RAC database and RAC instances)
Some Important Notes :
- Oracle takes physical backup of OCR automatically.
- No Cluster downtime or RAC database down time requires for PHYSICAL backup of OCR.
- No Cluster downtime or RAC database down time requires for MANUAL export backup of OCR.
- For recovery of OCR from any of above backup it should need to down ALL.
- All procedure requires ROOT login.
================================================================================
Restore and Recover OCR from backup (Oracle Cluster Registry)
Step 1: Locate physical the OCR backups using –showbackup command.
#ocrconfig – showbackup
Step 2: Review the contents
#ocrdump –backupfile backup_file_name
Step 3: Stop the Oracle clusterware on all the nodes.
#crsctl stop crs
Step 4: Restore the OCR backup
# ocrconfig –restore $CRS_HOME/cdata/crs/day.ocr
OR
Restore the OCR from export/logical backup.
# ocrconfig –import export_file_name
For ex: - # ocrconfig –import /backup/oracle/exp_ocrbackup.dmp
Step 5: Restart the Clusterware on all nodes.
#crsctl start crs
Step 6: Check the OCR integrity
# cluvfy comp ocr –n all
Click here to learn How to backup and recover OCR.
Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com/
#ocrconfig – showbackup
Step 2: Review the contents
#ocrdump –backupfile backup_file_name
Step 3: Stop the Oracle clusterware on all the nodes.
#crsctl stop crs
Step 4: Restore the OCR backup
# ocrconfig –restore $CRS_HOME/cdata/crs/day.ocr
OR
Restore the OCR from export/logical backup.
# ocrconfig –import export_file_name
For ex: - # ocrconfig –import /backup/oracle/exp_ocrbackup.dmp
Step 5: Restart the Clusterware on all nodes.
#crsctl start crs
Step 6: Check the OCR integrity
# cluvfy comp ocr –n all
Click here to learn How to backup and recover OCR.
Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com/
Convert Single Instance
Databases to RAC
1.1
Overview
We have Oracle grid infrastructure 11.2.0.3 with ASM installed
but we most databases running as standalones instead of RAC databases. Recently
we decide to convert all databases to RAC databases. The database version is
11.2.0.3. They are already running on the RAC home and on the grid cluster.
Oracle provide 4 ways to convert a standalone database to a RAC
database: DBCA, manual, RConfig and OEM.
With DBCA, we can backup the database as a template and then
create a RAC database with the template. This way is simple but the long down
time is not acceptable in most environments so we quickly excluded as a choice.
I found the manual way is most reliable. You know exactly what
each step does. It has the minimum downtime. The drawback is you have to do the
manual steps for each database. If you have 100 database, you might not like to
repeat the steps again, again and again.
The RConfig is the most applicable way. All we need to do is to
create an xml file and run a command. The benefit is once you create the first xml
file, you can reuse it with very minimum modification. Sometimes you only need
to modify the name of the database. You can run multiple rconfig at the same
time.
OEM may appears like the easiest way as simple as a few clicks.
But it turned out that I never make it to work. I actually never spend the time
to fix the issue. It's not worth the time to troubleshoot OEM. I just went with
RConfig as it's easy enough.
In this article, we provides 2 samples. First one with manual
steps and the second one use RConfig. Our assumption is that the database
already running in the RAC home. If your database is not currently in the RAC
home, you need to move it there first. You can do it with RMAN or datapump, or
manually copy the files and recreate the control file. If your database is not
in the same version, you might want to upgrade or patch to the same patch level
before you move your database.
1.2
Manual Conversion of Standalone database to RAC
1. Create a temporary
PFILE from the SPFILE and add cluster parameters.
CREATE PFILE='/u03/oracle/wangm/pfile' from spfile;
Edit the pfile created and add the following RAC related
parameters:
cluster_database_instances=2
cluster_database=true
E91DEV1.instance_number=1
E91DEV2.instance_number=2
E91DEV1.thread=1
E91DEV2.thread=2
E91DEV1.undo_tablespace='PSUNDO01'
E91DEV2.undo_tablespace='PSUNDO02'
E91DEV1.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr1-vip)(PORT=1521))'
E91DEV2.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr2-vip)(PORT=1521))'
Note you need to delete the two parameters:
undo_tablespace='PSUNDO01'
local_listener=’ DEVCLSTR1_LOCAL’
1. Create additional
UNDO tablespace.
create undo tablespace psundo02 datafile '+DATA';
1. Create redo threads
and enable the second thread.
E91DEV> alter database add logfile thread 2 group 5
('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 6
('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 7
('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database add logfile thread 2 group 8
('+DATA','+FLASH') size 128m;
Database altered.
E91DEV> alter database enable public thread 2;
Database altered.
E91DEV>
1. Copy the Oracle
password file and init file from the initial node, or from the node from which
you are working, to the corresponding location on the additional nodes on which
the cluster database will have an instance. Replace the ORACLE_SID name in each
password file appropriately for each additional instance.
devclstr1-/u03/oracle/wangm>cd $ORACLE_HOME/dbs
devclstr1-/u01/app/oracle/product/11.2/dbs>mv orapwE91DEV
orapwE91DEV1
devclstr1-/u01/app/oracle/product/11.2/dbs>mv initE91DEV.ora
initE91DEV1.ora
devclstr1-/u01/app/oracle/product/11.2/dbs>scp -p
orapwE91DEV1 devclstr2:/u01/app/oracle/product/11.2/dbs/orapwE91DEV2
orapwE91DEV1 100% 2560 2.5KB/s 00:00
devclstr1-/u01/app/oracle/product/11.2/dbs>scp -p
initE91DEV1.ora devclstr2:/u01/app/oracle/product/11.2/dbs/initE91DEV2.ora
initE91DEV1.ora
1. Create the SPFILE
from the updated PFILE.
E91DEV> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
E91DEV> exit
Disconnected from Oracle Database 11g Enterprise Edition Release
11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
Data Mining and Real Application Testing options
devclstr1-/u03/oracle/wangm>export ORACLE_SID=E91DEV1
devclstr1-/u03/oracle/wangm>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Wed Jun 25 15:52:16
2014
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance.
none> create spfile='+DATA/E91DEV/spfileE91DEV.ora' from
pfile='/u03/oracle/wangm/pfile';
File created.
none> startup
ORACLE instance started.
Total System Global Area 2137886720 bytes
Fixed Size 2230072 bytes
Variable Size 654313672 bytes
Database Buffers 1442840576 bytes
Redo Buffers 38502400 bytes
Database mounted.
Database opened.
none>
1. On the local node,
run catclust.sql. This command creates the dictionary views needed for Oracle
RAC databases. For example:
SQL> start ?/rdbms/admin/catclust.sql
1. Add the configuration
for the Oracle RAC database and its instance-to-node mapping using SRVCTL.
devclstr1-/u03/oracle/wangm>srvctl add database -d E91DEV -o
/u01/app/oracle/product/11.2
devclstr1-/u03/oracle/wangm>srvctl add instance -d E91DEV -i
E91DEV1 -n devclstr1
devclstr1-/u03/oracle/wangm>srvctl add instance -d E91DEV -i
E91DEV2 -n devclstr2
devclstr1-/u03/oracle/wangm>
1. Start the database
using SRVCTL.
devclstr1-/u03/oracle/wangm>srvctl start database -d E91DEV
devclstr1-/u03/oracle/wangm>srvctl status database -d E91DEV
Instance E91DEV1 is running on node devclstr1
Instance E91DEV2 is running on node devclstr2
devclstr1-/u03/oracle/wangm>
After starting the database with SRVCTL, your conversion process
is complete and, for example, you can run the following SQL statement to see
the status of all the instances in your Oracle RAC database:
E91DEV> select * from v$active_instances;
INST_NUMBER INST_NAME
-----------
------------------------------------------------------------
1 devclstr1:E91DEV1
2 devclstr2:E91DEV2
E91DEV>
1.3
Converting with RConfig
1. As the user, copy the
file $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC_AdminManaged.xml
to a working directory and edit the file.
Here is the file E91TRP.xml after modification.
<?xml version="1.0" encoding="UTF-8"?>
<n:RConfig xmlns:n="http://www.oracle.com/rconfig"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.oracle.com/rconfig
rconfig.xsd">
<n:ConvertToRAC>
<!-- Verify does a precheck to ensure all pre-requisites are
met, before the conversion is attempted. Allowable values are: YES|NO|ONLY
-->
<n:Convert verify="ONLY">
<!--Specify current OracleHome of non-rac database for
SourceDBHome -->
<n:SourceDBHome>/u01/app/oracle/product/11.2</n:SourceDBHome>
<!--Specify OracleHome where the rac database should be
configured. It can be same as SourceDBHome -->
<n:TargetDBHome>/u01/app/oracle/product/11.2</n:TargetDBHome>
<!--Specify SID of non-rac database and credential. User with
sysdba role is required to perform conversion -->
<n:SourceDBInfo SID="E91TPR">
<n:Credentials>
<n:User>sys</n:User>
<n:Password>password</n:Password>
<n:Role>sysdba</n:Role>
</n:Credentials>
</n:SourceDBInfo>
<!--Specify the list of nodes that should have rac instances
running for the Admin Managed Cluster Database. LocalNode should be the first
node in this nodelist. --> <n:NodeList>
<n:Node name="devclstr3"/>
<n:Node name="devclstr2"/>
</n:NodeList>
<!--Specify RacOneNode along with servicename to convert
database to RACOne Node -->
<!--n:RacOneNode
servicename="salesrac1service"/-->
<!--Instance Prefix tag is optional starting with 11.2. If
left empty, it is derived from db_unique_name.-->
<n:InstancePrefix></n:InstancePrefix>
<!-- Listener details are no longer needed starting 11.2.
Database is registered with default listener and SCAN listener running from
Oracle Grid Infrastructure home. -->
<!--Specify the type of storage to be used by rac database.
Allowable values are CFS|ASM. The non-rac database should have same storage
type. ASM credentials are no needed for conversion. -->
<n:SharedStorage type="ASM">
<!--Specify Database Area Location to be configured for rac
database.If this field is left empty, current storage will be used for rac
database. For CFS, this field will have directory path. -->
<n:TargetDatabaseArea></n:TargetDatabaseArea>
<!--Specify Fast Recovery Area to be configured for rac
database. If this field is left empty, current recovery area of non-rac
database will be configured for rac database. If current database is not using
recovery Area, the resulting rac database will not have a recovery area. -->
<n:TargetFlashRecoveryArea></n:TargetFlashRecoveryArea>
</n:SharedStorage>
</n:Convert>
</n:ConvertToRAC>
</n:RConfig>
1. Run :
First we set the verify in the xml to “ONLY” to make sure we can
convert.
<n:Convert verify="ONLY">
devclstr1-/u03/oracle/wangm>rconfig E91TPR.xml
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
<Convert>
<Response>
<Result code="0" >
Operation Succeeded
</Result>
</Response>
<ReturnValue type="object">
There is no return value for this step </ReturnValue>
</Convert>
</ConvertToRAC></RConfig>
devclstr1-/u03/oracle/wangm>
Then we set it to “YES” to perform the convert.
<n:Convert verify="YES">
devclstr1-/u03/oracle/wangm>rconfig E91TPR.xml
Converting Database "E91TPR" to Cluster Database.
Target Oracle Home: /u01/app/oracle/product/11.2. Database Role: PRIMARY.
Setting Data Files and Control Files
Adding Database Instances
Adding Redo Logs
Enabling threads for all Database Instances
Setting TEMP tablespace
Adding UNDO tablespaces
Adding Trace files
Setting Fast Recovery Area
Updating Oratab
Creating Password file(s)
Configuring Listeners
Configuring related CRS resources
Starting Cluster Database
<?xml version="1.0" ?>
<RConfig version="1.1" >
<ConvertToRAC>
<Convert>
<Response>
<Result code="0" >
Operation Succeeded
</Result>
</Response>
<ReturnValue type="object">
<Oracle_Home>
/u01/app/oracle/product/11.2
</Oracle_Home>
<Database type="ADMIN_MANAGED" >
<InstanceList>
<Instance SID="E91TPR1" Node="devclstr1"
>
</Instance>
<Instance SID="E91TPR2" Node="devclstr2"
>
</Instance>
</InstanceList>
</Database> </ReturnValue>
</Convert>
</ConvertToRAC></RConfig>
devclstr1-/u03/oracle/wangm>
1.4
Post Conversion steps.
1) Update crontab if applicable.
2) Remove and add the database target in OEM.
3) Modify local listener setting.
alter system set
local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr1-vip)(PORT=1521))'
sid='E91TPR1';
alter system set
local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=devclstr2-vip)(PORT=1521))'
sid='E91TPR2';
4) If using RConfig, the tool automatically name the new undo
tablespace to UNDOTBS2. To change it, do the following.
sys@E91TPR2> alter tablespace UNDOTBS2 rename to PSUNDO02;
Tablespace altered.
sys@E91TPR2> alter system set undo_tablespace=PSUNDO02
sid='E91TPR2';
System altered.
sys@E91TPR2> show parameter undo
NAME TYPE VALUE
------------------------------------ -----------
------------------------------
undo_management string AUTO
undo_retention integer 10800
undo_tablespace string PSUNDO02
sys@E91TPR2>
1.5
Notes
1) Oracle supplies two sample xml files, one for Admin managed
and one for policy managed. Since we are using the Admin managed xml here. The
policy managed one is similiar.
2) We can create a new undo table space with desired name, swith
the undo_tablespace parameter and then drop the undo tablespace created by
RConfig. I've noticed if the psundo02 table space exists before running
RConfig, then it's smart enough to use it without creating the default undo
tablespace.
No comments:
Post a Comment