一点一滴努力,满仓满谷收成

1 现象
11.2.0.3的ORACLE RAC，因一DG空间紧张，故加一新盘到这个DG,命令如下：

SQL> alter diskgroup SALEDB_04
add disk ‘/dev/rdisk/disk2565’ name SALEDB_04_0012;

却引发了这个DG的dismount,进而引发了这个结点的ORACLE实例宕。

2 查原因
检查grid用户下的alert_+ASM1.log,发现如下信息：

SQL> alter diskgroup SALEDB_04 add disk ‘/dev/rdisk/disk2565’
NOTE: GroupBlock outside rolling migration privileged region
NOTE: Assigning number (6,12) to disk (/dev/rdisk/disk2565)
NOTE: requesting all-instance membership refresh for group=6
NOTE: initializing header on grp 6 disk SALEDB_04_0012
NOTE: requesting all-instance disk validation for group=6
Wed Jan 14 16:31:27 2015
NOTE: skipping rediscovery for group 6/0x6547b3f5 (SALEDB_04) on local instance.
Wed Jan 14 16:31:31 2015
NOTE: requesting all-instance disk validation for group=6
NOTE: skipping rediscovery for group 6/0x6547b3f5 (SALEDB_04) on local instance.
NOTE: initiating PST update: grp = 6
Wed Jan 14 16:31:35 2015
GMON updating group 6 at 55 for pid 37, osid 19499
NOTE: PST update grp = 6 completed successfully
Errors in file /grid/app/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_18724.trc:
ORA-27070: async read/write failed
HPUX-ia64 Error: 15: Block device required
WARNING: Read Failed. group:6 disk:12 AU:0 offset:0 size:4096
Errors in file /grid/app/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_18724.trc:
ORA-27070: async read/write failed
HPUX-ia64 Error: 15: Block device required
WARNING: Read Failed. group:6 disk:11 AU:0 offset:0 size:4096
Errors in file /grid/app/crs_base/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_18724.trc:
ORA-27070: async read/write failed
HPUX-ia64 Error: 15: Block device required
WARNING: Read Failed. group:6 disk:10 AU:0 offset:0 size:4096
GMON querying group 6 at 56 for pid 18, osid 18724
NOTE: process _rbal_+asm1 (18724) initiating offline of disk 12.4163322780 (SALEDB_04_0012) with mask 0x7e in group 6
WARNING: Disk 12 (SALEDB_04_0012) in group 6 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 6, dsk = 12/0xf827439c, mask = 0x6a, op = clear
GMON updating disk modes for group 6 at 57 for pid 18, osid 18724
ERROR: Disk 12 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 6)
WARNING: Offline of disk 12 (SALEDB_04_0012) in group 6 and mode 0x7f failed on ASM inst 1

主要的信息为:

ORA-27070: async read/write failed
HPUX-ia64 Error: 15: Block device required

这个报错的主要原因为：BUG:17264575,在于maxfiles_lim参数值比较低
3 处理
maxfiles_lim参数：由98304提升至1048576
maxfiles参数：由1024提升到最佳实践的4096
调整后持续观察中。