Sunday, July 7, 2013

ASM 12101 Normal Redundancy Testing

We will be testing normal ASM redundancy using ASM 12c. The test will simulate a device failure for one of the devices used by ASM disk.

Iomage HDD Drive will be Disk A Side
Seagate Backup Plus Drive will be Disk  B Side


Note: When using ASM mirroring all disks should be even in all failgroups. If you want to see what happens when the disk are uneven in the failgroups review the Oracle Support doc ID below.

How To Reorganize A Normal or High Redundancy ASM Diskgroup With An Uneven Number Of Disks Members In The Failgroups (ORA-15041). [ID 1526819.1]

My Setup

RedHat 5.6
8GB Memory
I already have external redundancy diskgroups for my other databases on my VM machine

For my testing I configured the following disk in my VM machine

Setup and Configure Disks

DG_MIRROR

Side A Disk- 3 vdi SATA drives 10G each on Iomage HDD Drive
Side B Disk- 3 vdi SATA drives 10G each on Seagate Backup Plus Drive

FRA_MIRROR

Side A Disk- 1 vdi SATA drives 15G each on Iomage HDD Drive
Side B Disk- 1 vdi SATA drives 15G each on Seagate Backup Plus Drive

How to add new VDI disk to VM Machine



Note: I made a mistake on the b side FRA disk I created as 10G and it needed to be 15G


Partition Disks

The disk we added are as followed which you can retriever from /dev
sdt sds sdr sdq sdp sdo sdn sdm

As root
#cd /dev
#fdisk sdt
sequence of answers "n", "p", "1", "Return", "Return", "p" and "w"
#fdisk sds
sequence of answers "n", "p", "1", "Return", "Return", "p" and "w"
#fdisk sdr
sequence of answers "n", "p", "1", "Return", "Return", "p" and "w"
#fdisk sdq
sequence of answers "n", "p", "1", "Return", "Return", "p" and "w"
#fdisk sdp
sequence of answers "n", "p", "1", "Return", "Return", "p" and "w"
#fdisk sdo
sequence of answers "n", "p", "1", "Return", "Return", "p" and "w"
#fdisk sdn
sequence of answers "n", "p", "1", "Return", "Return", "p" and "w"
#fdisk sdm
sequence of answers "n", "p", "1", "Return", "Return", "p" and "w"

Configure Disks for ASM

As root
#/usr/sbin/oracleasm createdisk DG_DISK1a /dev/sdm1
#/usr/sbin/oracleasm createdisk DG_DISK2a /dev/sdn1
#/usr/sbin/oracleasm createdisk DG_DISK3a /dev/sdo1
#/usr/sbin/oracleasm createdisk DG_DISK1b /dev/sdp1
#/usr/sbin/oracleasm createdisk DG_DISK2b /dev/sdq1
#/usr/sbin/oracleasm createdisk DG_DISK3b /dev/sdr1
#/usr/sbin/oracleasm createdisk FRA_DISK1a /dev/sds1
#/usr/sbin/oracleasm createdisk FRA_DISK1b /dev/sdt1

Scan ASM Disks

As root
#/usr/sbin/oracleasm scandisks

List ASM Disks

As root
#/usr/sbin/oracleasm listdisks

Output
[root@alpddbs002 dev]# /usr/sbin/oracleasm listdisks
DG_DISK1A
DG_DISK1B
DG_DISK2A
DG_DISK2B
DG_DISK3A
DG_DISK3B
DISK1
DISK2
DISK3
DISK4
FRA1
FRA_DISK1A
FRA_DISK1B

Check ASM Disks

As root
#/etc/init.d/oracleasm querydisk -d `/etc/init.d/oracleasm listdisks -d` | \
cut -f2,10,11 -d" " | \
perl -pe 's/"(.*)".*\[(.*), *(.*)\]/$1 $2 $3/g;' | \
while read v_asmdisk v_minor v_major
do
v_device=`ls -la /dev | grep " $v_minor, *$v_major " | awk '{print $10}'`
echo "ASM disk $v_asmdisk based on /dev/$v_device [$v_minor, $v_major]"
done

Output
[root@alpddbs002 dev]# /etc/init.d/oracleasm querydisk -d `/etc/init.d/oracleasm listdisks -d` | \
> cut -f2,10,11 -d" " | \
> perl -pe 's/"(.*)".*\[(.*), *(.*)\]/$1 $2 $3/g;' | \
> while read v_asmdisk v_minor v_major
> do
> v_device=`ls -la /dev | grep " $v_minor, *$v_major " | awk '{print $10}'`
> echo "ASM disk $v_asmdisk based on /dev/$v_device [$v_minor, $v_major]"
> done
ASM disk DG_DISK1A based on /dev/sdm1 [8, 193]
ASM disk DG_DISK1B based on /dev/sdp1 [8, 241]
ASM disk DG_DISK2A based on /dev/sdn1 [8, 209]
ASM disk DG_DISK2B based on /dev/sdq1 [65, 1]
ASM disk DG_DISK3A based on /dev/sdo1 [8, 225]
ASM disk DG_DISK3B based on /dev/sdr1 [65, 17]
ASM disk DISK1 based on /dev/sdg1 [8, 97]
ASM disk DISK2 based on /dev/sdh1 [8, 113]
ASM disk DISK3 based on /dev/sdi1 [8, 129]
ASM disk DISK4 based on /dev/sdj1 [8, 145]
ASM disk FRA1 based on /dev/sdk1 [8, 161]
ASM disk FRA_DISK1A based on /dev/sds1 [65, 33]
ASM disk FRA_DISK1B based on /dev/sdt1 [65, 49]

Create ASM Diskgroups

DG_MIRROR

Login to ASM
#sqlplus / as sysasm
SQL>CREATE DISKGROUP DG_MIRROR NORMAL REDUNDANCY  FAILGROUP FG_2 DISK 'ORCL:DG_DISK1B' SIZE 10236 M ,'ORCL:DG_DISK2B' SIZE 10236 M ,'ORCL:DG_DISK3B' SIZE 10236 M  FAILGROUP FG_1 DISK 'ORCL:DG_DISK1A' SIZE 10236 M ,'ORCL:DG_DISK2A' SIZE 10236 M ,'ORCL:DG_DISK3A' SIZE 10236 M  ATTRIBUTE 'compatible.rdbms' = '12.1', 'compatible.asm' = '12.1';

FRA_MIRROR

Login to ASM
#sqlplus / as sysasm
SQL>CREATE DISKGROUP FRA_MIRROR NORMAL REDUNDANCY  FAILGROUP FG_2 DISK 'ORCL:FRA_DISK1B' SIZE 15358 M  FAILGROUP FG_1 DISK 'ORCL:FRA_DISK1A' SIZE 15358 M  ATTRIBUTE 'compatible.rdbms' = '12.1', 'compatible.asm' = '12.1';

If you want to see a demo on how to setup a ASM diskgroup with normal redundancy using EM12c see my YouTube video below if not continue on. 

Database Layout

The database I will be using is called DBTEST3 which is a 12.1.0.1 database version. Below are the datafile, redolog, controlfile and tempfile location.









Test


Steps:

1. Issue the below command to simulate failed device:


  • #echo 1 > /sys/block/sdm/device/delete

DB Alert Log shows the following when device failure:
Sun Jul 07 13:18:28 2013
NOTE: updating disk modes to 0x5 from 0x7 for disk 3 (DG_DISK1A) in group 3 (DG_MIRROR): lflags 0x0  
NOTE: disk 3 (DG_DISK1A) in group 3 (DG_MIRROR) is offline for reads
Sun Jul 07 13:18:28 2013
NOTE: ospid 10251 initiating cluster wide offline of disk 3 in group 3
Sun Jul 07 13:18:28 2013
NOTE: disk 3 (DG_DISK1A) in group 3 (DG_MIRROR) is locally offline for writes
Sun Jul 07 13:18:29 2013
NOTE: disk 3 (DG_DISK1A) in group 3 (DG_MIRROR) is offline for writes

ASM Alert Log shows the following when device failure:
Sun Jul 07 13:18:28 2013
NOTE: process _user15614_+asm (15614) initiating offline of disk 3.3915927247 (DG_DISK1A) with mask 0x7e in group 3 (DG_MIRROR) with client assisting
NOTE: checking PST: grp = 3
Sun Jul 07 13:18:28 2013
GMON checking disk modes for group 3 at 428 for pid 26, osid 15614
Sun Jul 07 13:18:28 2013
NOTE: checking PST for grp 3 done.
NOTE: initiating PST update: grp 3 (DG_MIRROR), dsk = 3/0xe9684ecf, mask = 0x6a, op = clear
Sun Jul 07 13:18:28 2013
GMON updating disk modes for group 3 at 429 for pid 26, osid 15614
NOTE: group DG_MIRROR: updated PST location: disk 0000 (PST copy 0)
NOTE: group DG_MIRROR: updated PST location: disk 0004 (PST copy 1)
Sun Jul 07 13:18:28 2013
NOTE: PST update grp = 3 completed successfully 
Sun Jul 07 13:18:29 2013
NOTE: process _b000_+asm (15616) initiating offline of disk 3.3915927247 (DG_DISK1A) with mask 0x7e in group 3 (DG_MIRROR) without client assisting
NOTE: checking PST: grp = 3
Sun Jul 07 13:18:29 2013
GMON checking disk modes for group 3 at 430 for pid 26, osid 15616
Sun Jul 07 13:18:29 2013
NOTE: checking PST for grp 3 done.
Sun Jul 07 13:18:29 2013
NOTE: sending set offline flag message (4056126759) to 1 disk(s) in group 3
Sun Jul 07 13:18:29 2013
WARNING: Disk 3 (DG_DISK1A) in group 3 mode 0x15 is now being offlined
Sun Jul 07 13:18:29 2013
NOTE: initiating PST update: grp 3 (DG_MIRROR), dsk = 3/0xe9684ecf, mask = 0x6a, op = clear
Sun Jul 07 13:18:29 2013
GMON updating disk modes for group 3 at 431 for pid 26, osid 15616
Sun Jul 07 13:18:29 2013
NOTE: PST update grp = 3 completed successfully 
NOTE: initiating PST update: grp 3 (DG_MIRROR), dsk = 3/0xe9684ecf, mask = 0x7e, op = clear
Sun Jul 07 13:18:29 2013
GMON updating disk modes for group 3 at 432 for pid 26, osid 15616
NOTE: group DG_MIRROR: updated PST location: disk 0000 (PST copy 0)
NOTE: group DG_MIRROR: updated PST location: disk 0004 (PST copy 1)
Sun Jul 07 13:18:29 2013
NOTE: cache closing disk 3 of grp 3: DG_DISK1A
Sun Jul 07 13:18:29 2013
NOTE: PST update grp = 3 completed successfully 

2. Issue the below command to simulate fixing the device failure:
  • #echo "0 0 0" > /sys/class/scsi_host/host12/scan
Note: To find the SCSI ID that you need for the above statement us the following

  • #ll /sys/block/sdm/device

3. Login to ASM using SYSASM use the below ASM command to online the disk:
SQL>ALTER DISKGROUP DG_MIRROR ONLINE  DISK DG_DISK1A;

DB Alert Log shows the following when placing disk back online:
Sun Jul 07 13:22:11 2013
NOTE: Found ORCL:DG_DISK1A for disk DG_DISK1A
SUCCESS: disk DG_DISK1A (3.3915927247) replaced in diskgroup DG_MIRROR path: ORCL:DG_DISK1A
NOTE: updating disk modes to 0x5 from 0x1 for disk 3 (DG_DISK1A) in group 3 (DG_MIRROR): lflags 0x0    
NOTE: disk 3 (DG_DISK1A) in group 3 (DG_MIRROR) is online for writes

ASM Alert Log shows the following when placing disk back online:
Sun Jul 07 13:22:11 2013
NOTE: initiating PST update: grp 3 (DG_MIRROR), dsk = 3/0x0, mask = 0x19, op = assign
Sun Jul 07 13:22:11 2013
GMON updating disk modes for group 3 at 455 for pid 30, osid 5093
NOTE: group DG_MIRROR: updated PST location: disk 0000 (PST copy 0)
NOTE: group DG_MIRROR: updated PST location: disk 0004 (PST copy 1)
Sun Jul 07 13:22:11 2013
NOTE: PST update grp = 3 completed successfully 
Sun Jul 07 13:22:11 2013
NOTE: membership refresh pending for group 3/0xdf78be60 (DG_MIRROR)
Sun Jul 07 13:22:12 2013
GMON querying group 3 at 456 for pid 7, osid 6263
NOTE: cache opening disk 3 of grp 3: DG_DISK1A label:DG_DISK1A
Sun Jul 07 13:22:12 2013
SUCCESS: refreshed membership for 3/0xdf78be60 (DG_MIRROR)
Sun Jul 07 13:22:12 2013
NOTE: initiating PST update: grp 3 (DG_MIRROR), dsk = 3/0x0, mask = 0x5d, op = assign
Sun Jul 07 13:22:12 2013
GMON updating disk modes for group 3 at 457 for pid 30, osid 5093
NOTE: group DG_MIRROR: updated PST location: disk 0000 (PST copy 0)
NOTE: group DG_MIRROR: updated PST location: disk 0004 (PST copy 1)
Sun Jul 07 13:22:13 2013
NOTE: PST update grp = 3 completed successfully 
Sun Jul 07 13:22:14 2013
NOTE: Attempting voting file refresh on diskgroup DG_MIRROR
Sun Jul 07 13:22:20 2013
NOTE: initiating PST update: grp 3 (DG_MIRROR), dsk = 3/0x0, mask = 0x7d, op = assign
Sun Jul 07 13:22:20 2013
GMON updating disk modes for group 3 at 458 for pid 30, osid 5093
NOTE: group DG_MIRROR: updated PST location: disk 0000 (PST copy 0)
NOTE: group DG_MIRROR: updated PST location: disk 0004 (PST copy 1)
Sun Jul 07 13:22:20 2013
NOTE: PST update grp = 3 completed successfully 
Sun Jul 07 13:22:20 2013
NOTE: Voting File refresh pending for group 3/0xdf78be60 (DG_MIRROR)
Sun Jul 07 13:22:20 2013
SUCCESS: ALTER DISKGROUP DG_MIRROR ONLINE  DISK DG_DISK1A /* EM */
NOTE: Attempting voting file refresh on diskgroup DG_MIRROR
Sun Jul 07 13:22:23 2013
NOTE: starting rebalance of group 3/0xdf78be60 (DG_MIRROR) at power 1
Starting background process ARB0
Sun Jul 07 13:22:23 2013
ARB0 started with pid=28, OS id=15868 
NOTE: assigning ARB0 to group 3/0xdf78be60 (DG_MIRROR) with 1 parallel I/O
Sun Jul 07 13:28:40 2013
NOTE: initiating PST update: grp 3 (DG_MIRROR), dsk = 3/0x0, mask = 0x7f, op = assign
Sun Jul 07 13:28:40 2013
GMON updating disk modes for group 3 at 495 for pid 28, osid 15868
NOTE: group DG_MIRROR: updated PST location: disk 0000 (PST copy 0)
NOTE: group DG_MIRROR: updated PST location: disk 0004 (PST copy 1)
Sun Jul 07 13:28:40 2013
NOTE: PST update grp = 3 completed successfully 
NOTE: reset timers for disk: 3
NOTE: completed online of disk group 3 disks
DG_DISK1A (3) 
Sun Jul 07 13:28:43 2013
NOTE: requesting all-instance membership refresh for group=3
Sun Jul 07 13:28:43 2013
NOTE: membership refresh pending for group 3/0xdf78be60 (DG_MIRROR)
Sun Jul 07 13:28:44 2013
GMON querying group 3 at 496 for pid 7, osid 6263
Sun Jul 07 13:28:44 2013
SUCCESS: refreshed membership for 3/0xdf78be60 (DG_MIRROR)
NOTE: Attempting voting file refresh on diskgroup DG_MIRROR
NOTE: stopping process ARB0
Sun Jul 07 13:28:50 2013
SUCCESS: rebalance completed for group 3/0xdf78be60 (DG_MIRROR)

The same online operations can be completed using Enterprise Manager 12c see demo below:

Posted by at

5 comments:

  1. ahmetmelihbasbugMarch 15, 2018 at 12:49 AM

    This comment has been removed by the author.

    ReplyDelete
  2. ahmetmelihbasbugMarch 15, 2018 at 12:49 AM

    This comment has been removed by the author.

    ReplyDelete
  3. ahmetmelihbasbugMarch 15, 2018 at 12:50 AM

    Hello.
    I have Exadata X3v2 machine. One disk was broken down and i can't replace it. I want to drop that disk from data diskgroup in ASMCMD. Could it be a problem?
    I plan to drop Data diskgroup from asmcmd and create Data diskgroup in asmcmd without a disk. Should i recreate database? Thanks for your help.

    ReplyDelete
  4. gopipatelNovember 10, 2019 at 9:02 PM

    Wow! amazing post.. Thanks for sharing!
    How to Protect Yourself From Cybersecurity Threats

    ReplyDelete
  5. luckysFebruary 3, 2021 at 1:24 AM

    singer fashion mate 3342

    ReplyDelete

Subscribe to: Post Comments (Atom)