11.2 RAC 修改了目录权限(u01)后crs不能启动的解决方法-1-手工修复错误的权限

联系:QQ(5163721)

标题:11.2 RAC 修改了目录权限(u01)后crs不能启动的解决方法-1-手工修复错误的权限

作者:Lunar©版权所有[文章允许转载,但必须以链接方式注明源地址,否则追究法律责任.]

在11.2RAC中,如果修改了gird的安装目录(类似chown -R xxx /u01),比如通常我们会使用/u01,则crs会出现不能启动的状态,启动时,mdnsd进程会首先卡主,crs日志会有如下信息:

2014-09-26 20:56:25.895
[ohasd(16366)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.mdnsd'. Details at (:CRSPE00111:) {0:0:199} in /u01/app/11.2.0/grid/log/lunardb1/ohasd/ohasd.log.
2014-09-26 20:58:28.984
[/u01/app/11.2.0/grid/bin/oraagent.bin(15422)]CRS-5818:Aborted command 'start' for resource 'ora.mdnsd'. Details at (:CRSAGF00113:) {0:0:228} in /u01/app/11.2.0/grid/log/lunardb1/agent/ohasd/oraagent_grid//oraagent_grid.log.
2014-09-26 20:58:32.994
[ohasd(16366)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.mdnsd'. Details at (:CRSPE00111:) {0:0:228} in /u01/app/11.2.0/grid/log/lunardb1/ohasd/ohasd.log.
2014-09-26 23:36:05.848
[/u01/app/11.2.0/grid/bin/orarootagent.bin(8064)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/acfsload" spawned by agent "/u01/app/11.2.0/grid/bin/orarootagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/lunardb1/agent/ohasd/orarootagent_root/orarootagent_root.log"

下面我们尝试使用3种方法来修复该问题。
方法1————直接修改/u01和其他相关文件或者目录的权限:
注意: 此方法,仅仅用于紧急启动数据库或者ASM的不得已的做法,在生产环境下,官方建议的做法是删除节点和添加节点(后面会在方法3中详细描述)。

首先修改/u01目录为grid:oinstall,并修改/u01/app/oracle为oracle:oinstall

chown -R grid:oinstall /u01
chown -R oracle:oinstall /u01/app/oracle

如果进行上述目录权限的修改,那么crs表面可以启动,但是可以看到后台重要的agent进程都是错误的状态:

2014-10-04 19:56:16.828
[/u01/app/11.2.0/grid/bin/oraagent.bin(19898)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/q9db01/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-10-04 19:56:16.832
[/u01/app/11.2.0/grid/bin/oraagent.bin(19898)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/q9db01/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-10-04 19:56:16.848
[/u01/app/11.2.0/grid/bin/oraagent.bin(19898)]CRS-5016:Process "/u01/app/11.2.0/grid/opmn/bin/onsctli" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/q9db01/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-10-04 19:56:23.733
[/u01/app/11.2.0/grid/bin/oraagent.bin(20116)]CRS-5010:Update of configuration file "/u01/app/oracle/product/11.2.0/db_1/srvm/admin/oratab.bak.q9db01" failed: details at "(:CLSN00013:)" in "/u01/app/11.2.0/grid/log/q9db01/agent/crsd/oraagent_oracle//oraagent_oracle.log"

还有一些对ohasd和crsd比较关键的文件的权限,也一并修改了:

[grid@lunardb1 ~]$ env|grep ORA
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=/u01/app/11.2.0/grid
[grid@lunardb1 ~]$ exit
logout
[root@lunardb1 app]# export GRID_HOME=/u01/app/11.2.0/grid
[root@lunardb1 app]# cd $GRID_HOME/log/`hostname`/crsd ; 
-bash: cd: /u01/app/11.2.0/grid/log/lunardb1.800best.com/crsd: No such file or directory
[root@lunardb1 app]# cd /u01/app/11.2.0/grid/log/lunardb1/crsd
[root@lunardb1 crsd]# chown root:root *
[root@lunardb1 crsd]# cd ../ohasd
[root@lunardb1 ohasd]# chown root:root *
[root@lunardb1 ohasd]# cd ..
[root@lunardb1 lunardb1]# ll
total 2312
drwxr-xr-x 2 grid oinstall    4096 Jun  7  2013 acfs
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 acfslog
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 acfsrepl
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 acfsreplroot
drwxr-xr-x 2 grid oinstall    4096 Jun  7  2013 acfssec
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 admin
drwxrwxr-t 4 grid oinstall    4096 Jun  7  2013 agent
-rw-rw-r-- 1 grid oinstall 2266872 Oct  4 17:48 alertlunardb1.log
drwxr-x--x 2 grid oinstall    4096 Jun 17 14:24 client
drwxr-x--- 2 grid oinstall    4096 Aug  6 15:40 crflogd
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 crfmond
drwxr-x--- 2 grid oinstall    4096 Sep  2 01:02 crsd
drwxr-x--- 2 grid oinstall    4096 Sep 27 01:54 cssd
drwxr-x--- 2 grid oinstall    4096 Sep 26 10:02 ctssd
drwxr-x--- 4 grid oinstall    4096 Jun  7  2013 cvu
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 diskmon
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 evmd
drwxr-x--- 2 grid oinstall    4096 Oct  4 17:47 gipcd
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 gnsd
drwxr-x--- 2 grid oinstall    4096 Oct  4 17:47 gpnpd
drwxr-x--- 2 grid oinstall    4096 Sep 27 01:30 mdnsd
drwxr-x--- 2 grid oinstall    4096 Sep 27 01:49 ohasd
drwxrwxr-t 5 grid oinstall    4096 Jun  7  2013 racg
drwxr-x--- 2 grid oinstall    4096 Jun  7  2013 srvm
[root@lunardb1 lunardb1]# cd /crsd
-bash: cd: /crsd: No such file or directory
[root@lunardb1 lunardb1]# cd agent/crsd/orarootagent_root 
[root@lunardb1 orarootagent_root]# chown root:root *
[root@lunardb1 orarootagent_root]# 
[root@lunardb1 orarootagent_root]# cd /u01/app/11.2.0/grid/log/lunardb1/agent/ohasd/orarootagent_root
[root@lunardb1 orarootagent_root]# ll
total 104528
-rw-r--r-- 1 grid oinstall 10564752 Sep 26 14:18 orarootagent_root.l01
-rw-r--r-- 1 grid oinstall 10565738 Sep 24 04:23 orarootagent_root.l02
-rw-r--r-- 1 grid oinstall 10563920 Sep 21 18:28 orarootagent_root.l03
-rw-r--r-- 1 grid oinstall 10565310 Sep 19 08:40 orarootagent_root.l04
-rw-r--r-- 1 grid oinstall 10565749 Sep 16 22:41 orarootagent_root.l05
-rw-r--r-- 1 grid oinstall 10563754 Sep 14 12:41 orarootagent_root.l06
-rw-r--r-- 1 grid oinstall 10563226 Sep 12 02:49 orarootagent_root.l07
-rw-r--r-- 1 grid oinstall 10561202 Sep  9 17:03 orarootagent_root.l08
-rw-r--r-- 1 grid oinstall 10543893 Sep  7 07:09 orarootagent_root.l09
-rw-r--r-- 1 grid oinstall 10566373 Sep  4 21:51 orarootagent_root.l10
-rw-r--r-- 1 grid oinstall  1213705 Oct  4 17:57 orarootagent_root.log
-rw-r--r-- 1 grid oinstall        0 Jun  7  2013 orarootagent_rootOUT.log
-rw-r--r-- 1 grid oinstall        5 Oct  4 17:48 orarootagent_root.pid
[root@lunardb1 orarootagent_root]# chown root:root *
[root@lunardb1 orarootagent_root]# 

[root@lunardb1 orarootagent_root]# cd $GRID_HOME
[root@lunardb1 grid]# cd bin
[root@lunardb1 bin]# ll oradism
-rwxr-x--- 1 grid oinstall 71758 Sep 17  2011 oradism
[root@lunardb1 bin]# 
[root@lunardb1 bin]# chown root:oinstall oradism
[root@lunardb1 bin]# chmod 4750 oradism
[root@lunardb1 bin]# ll oradism
-rwsr-x--- 1 root oinstall 71758 Sep 17  2011 oradism
[root@lunardb1 bin]# 

此时启动可以crs可以启动了。
但是,可以看到目录权限有问题的节点,数据库没有正常启动:

[root@lunardb1 app]# ps -ef|grep d.bin
root      3722     1  3 17:47 ?        00:00:02 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
grid      3938     1  0 17:47 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
grid      3950     1  0 17:47 ?        00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
grid      4003     1  0 17:47 ?        00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
grid      4024     1  1 17:47 ?        00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
root      4071     1  0 17:47 ?        00:00:00 /u01/app/11.2.0/grid/bin/cssdmonitor
root      4086     1  0 17:47 ?        00:00:00 /u01/app/11.2.0/grid/bin/cssdagent
grid      4117     1  2 17:47 ?        00:00:01 /u01/app/11.2.0/grid/bin/ocssd.bin 
root      4508     1  1 17:48 ?        00:00:00 /u01/app/11.2.0/grid/bin/orarootagent.bin
root      4531     1  0 17:48 ?        00:00:00 /u01/app/11.2.0/grid/bin/octssd.bin reboot
grid      4571     1  0 17:48 ?        00:00:00 /u01/app/11.2.0/grid/bin/evmd.bin
root      5322     1  5 17:48 ?        00:00:01 /u01/app/11.2.0/grid/bin/crsd.bin reboot
grid      5600  4571  0 17:48 ?        00:00:00 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log
grid      5646     1  6 17:48 ?        00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
root      5650     1  3 17:48 ?        00:00:00 /u01/app/11.2.0/grid/bin/orarootagent.bin
grid      5847     1  2 17:48 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      5864     1  0 17:49 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_DG -inherit
root      5869   429  0 17:49 pts/1    00:00:00 grep d.bin
[root@lunardb1 app]# crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.DATA.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.DATA1.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.LISTENER.lsnr
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.LISTENER_DG.lsnr
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.OCR_VOTE.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.REDODG.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.asm
               ONLINE  ONLINE       lunardb1                   Started             
               ONLINE  ONLINE       lunardb2                   Started             
ora.gsd
               OFFLINE OFFLINE      lunardb1                                       
               OFFLINE OFFLINE      lunardb2                                       
ora.net1.network
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.net2.network
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.ons
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.registry.acfs
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       lunardb2                                       
ora.cvu
      1        ONLINE  ONLINE       lunardb2                                       
ora.oc4j
      1        ONLINE  ONLINE       lunardb2                                       
ora.lunardb.db
      1        ONLINE  OFFLINE                                           
      2        ONLINE  ONLINE       lunardb2                   Open,Readonly       
ora.lunardb1-dg-vip.vip
      1        ONLINE  ONLINE       lunardb1                                       
ora.lunardb1.vip
      1        ONLINE  ONLINE       lunardb1                                       
ora.lunardb2-dg-vip.vip
      1        ONLINE  ONLINE       lunardb2                                       
ora.lunardb2.vip
      1        ONLINE  ONLINE       lunardb2                                       
ora.scan1.vip
      1        ONLINE  ONLINE       lunardb2                                       
[root@lunardb1 app]#

手工启动数据库,报错信息如下:

[oracle@lunardb1 ~]$ ss

SQL*Plus: Release 11.2.0.3.0 Production on Sat Oct 4 17:52:09 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

17:52:09 @>startup mount
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/lunardb/spfilelunardb.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/lunardb/spfilelunardb.ora
ORA-12547: TNS:lost contact
17:52:15 @>exit
Disconnected
[oracle@lunardb1 ~]$ 

这个错误通常意味着oracle二进制文件权限不对,尝试修改:

[oracle@lunardb1 trace]$ ss

SQL*Plus: Release 11.2.0.3.0 Production on Sat Oct 4 18:01:53 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

18:01:53 @>startup   
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/lunardb/spfilelunardb.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/lunardb/spfilelunardb.ora
ORA-12547: TNS:lost contact
18:01:58 @>

正常情况下,$GRID_HOME/bin/oracle和$ORACLE_HOME/bin/oracle的权限都应该是6751,即“-rwsr-s–x”
对比下节点2(正常节点):

[grid@lunardb2 ~]$ cd $ORACLE_HOME
[grid@lunardb2 grid]$ cd bin
[grid@lunardb2 bin]$ ll oracle
-rwsr-s--x 1 grid oinstall 204113496 Jun  7  2013 oracle
[grid@lunardb2 bin]$ 

再看看节点1(问题节点):

[root@lunardb1 bin]# ll oracle
-rwxr-x--x 1 grid oinstall 204113496 Jun  7  2013 oracle
[root@lunardb1 bin]# 

手工修改$GRID_HOME/bin/oracle文件权限:

[root@lunardb1 bin]# chmod 6751 oracle
[root@lunardb1 bin]# ll oracle
-rwsr-s--x 1 grid oinstall 204113496 Jun  7  2013 oracle
[root@lunardb1 bin]# 

顺便检查一下$ORACLE_HOME/bin/oracle文件权限:

[root@lunardb1 bin]# su - oracle
[oracle@lunardb1 ~]$ cd $ORACLE_HOME
[oracle@lunardb1 db_1]$ cd bin
[oracle@lunardb1 bin]$ ll oracle
-rwxr-s--x 1 oracle asmadmin 221332085 Jun  7  2013 oracle
[oracle@lunardb1 bin]$ 

现在,再重新启动数据库:

[oracle@lunardb1 ~]$ ss

SQL*Plus: Release 11.2.0.3.0 Production on Sat Oct 4 18:06:27 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

18:06:27 @>startup
ORACLE instance started.

Total System Global Area 1.6034E+11 bytes
Fixed Size                  2236968 bytes
Variable Size            3.0602E+10 bytes
Database Buffers         1.2939E+11 bytes
Redo Buffers              352468992 bytes
Database mounted.
Database opened.
18:07:19 @>exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Data Mining
and Real Application Testing options
[oracle@lunardb1 ~]$ exit
logout
You have new mail in /var/spool/mail/root
[root@lunardb1 bin]# crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.DATA.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.DATA1.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.LISTENER.lsnr
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.LISTENER_DG.lsnr
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.OCR_VOTE.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.REDODG.dg
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.asm
               ONLINE  ONLINE       lunardb1                   Started             
               ONLINE  ONLINE       lunardb2                   Started             
ora.gsd
               OFFLINE OFFLINE      lunardb1                                       
               OFFLINE OFFLINE      lunardb2                                       
ora.net1.network
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.net2.network
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.ons
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
ora.registry.acfs
               ONLINE  ONLINE       lunardb1                                       
               ONLINE  ONLINE       lunardb2                                       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       lunardb2                                       
ora.cvu
      1        ONLINE  ONLINE       lunardb2                                       
ora.oc4j
      1        ONLINE  ONLINE       lunardb2                                       
ora.lunardb.db
      1        ONLINE  ONLINE       lunardb1                   Open,Readonly       
      2        ONLINE  ONLINE       lunardb2                   Open,Readonly       
ora.lunardb1-dg-vip.vip
      1        ONLINE  ONLINE       lunardb1                                       
ora.lunardb1.vip
      1        ONLINE  ONLINE       lunardb1                                       
ora.lunardb2-dg-vip.vip
      1        ONLINE  ONLINE       lunardb2                                       
ora.lunardb2.vip
      1        ONLINE  ONLINE       lunardb2                                       
ora.scan1.vip
      1        ONLINE  ONLINE       lunardb2                                       
[root@lunardb1 bin]# 

目前该数据库貌似可以启动了,如果在很多异常情况下,目前的情况,已经可以尝试导出数据库或者备份数据库等等了。
但是这种状态的crs和数据库是存在很大隐患的,比如很可能会异常宕机或者出现其他莫名其妙的损坏等情况。
因此,一旦权限出现问题,要么使用rootcrs.pl -init修复(通常这种情况下,这种修复是徒劳的,后面的测试会证明这一点)
否则官方不支持任何手工手工修改权限的做法。就这一点,官方有明确的:

 The permissions can be reverted back to original values with rootcrs.pl or roothas.pl . There is a option -init  :     
Reset the permissions of all files and directories under Oracle CRS/HA home.

for GRID:
rootcrs.pl -init

For Standalone Grig:
roothas.pl  -init

If that does not work then permissions can be altered manually with information found from crsconfig_fileperms and crsconfig_dirs files.

Please note that changing the permissions manually is last resort and shouldn't be done unless re
此条目发表在 RAC 分类目录,贴了 , , 标签。将固定链接加入收藏夹。

发表评论

电子邮件地址不会被公开。 必填项已用 * 标注

您可以使用这些 HTML 标签和属性: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>