解决Exadata上IB检查脚本infinicheck的报错过程

联系:QQ(5163721)

标题:解决Exadata上IB检查脚本infinicheck的报错过程

作者:Lunar©版权所有[文章允许转载,但必须以链接方式注明源地址,否则追究法律责任.]

今天检查Exadata的IB网络时,使用 infinicheck 检查,发现db节点有报错,cell节点正常。
当前主机是Exadata X5-2:

[root@dm01db01 ibdiagtools]# /opt/oracle.SupportTools/CheckHWnFWProfile -c loose
[SUCCESS] The hardware and firmware matches supported profile for server=ORACLE_SERVER_X5-2
[root@dm01db01 ibdiagtools]# 

infinicheck的执行结果(该命令可以有很丰富的参数,但是也可以不带任何参数,缺省就可以):

                        INFINICHECK
                [Network Connectivity, Configuration and Performance]
                [Version IBD VER 2.d ]

 Verifying User Equivalance of user=root to all hosts.
(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)

 Verifying User Equivalance of user=root to all cells.
(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)


                    ####  CONNECTIVITY TESTS  ####
                    [COMPUTE NODES -> STORAGE CELLS]
                           (30 seconds approx.)
[SUCCESS]..............Results OK

[SUCCESS]....... All  can talk to all storage cells

        Verifying Subnet Masks on all nodes

[SUBNET MASKS DIFFER].....2 entries found

         Prechecking for uniformity of rds-tools on all nodes
[SUCCESS].... rds-tools version is the same across the cluster

        Checking for bad links in the fabric
[SUCCESS].......... No bad fabric links found

                    [COMPUTE NODES -> COMPUTE NODES]
                           (30 seconds approx.)
[SUCCESS]..............Results OK

[SUCCESS]....... All hosts can talk to all other nodes


                    ####  PERFORMANCE TESTS  ####

                    [(1) Storage Cell to Compute Node]
                                 (375 seconds approx)
[  INFO  ].............Performance test between 192.168.10.5 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.6 and 192.168.10.4 has been started.
[  INFO  ].............Performance test between 192.168.10.7 and 192.168.10.2 has been started.
[  INFO  ].............Performance test between 192.168.10.8 and 192.168.10.1 has been started.
[  INFO  ].............Performance test between 192.168.10.9 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.10 and 192.168.10.4 has been started.
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off

                    [(2) Every COMPUTE NODE to another COMPUTE NODE]
                                 (195 seconds approx)
[  INFO  ].............Performance test between 192.168.10.2 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.1 and 192.168.10.4 has been started.
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off

                    [(3) Every COMPUTE NODE to ALL STORAGE CELLS]
                        (looking for SymbolErrors)
                                 (195 seconds approx)
[  INFO  ].............Performance test between 192.168.10.5 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.6 and 192.168.10.4 has been started.
[  INFO  ].............Performance test between 192.168.10.7 and 192.168.10.2 has been started.
[  INFO  ].............Performance test between 192.168.10.8 and 192.168.10.1 has been started.
[  INFO  ].............Performance test between 192.168.10.9 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.10 and 192.168.10.4 has been started.
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off

[SUCCESS]....... No port errors found
Infinicheck failures reported.. please check log files

从这里我们看到,凡是到db节点的都报错。

infinicheck命令底层是调用的rds-stress命令,例如: rds-stress -r 192.168.10.1 -p 10584

当然,除了infinicheck意外,还有其他很多检查方法,比如rds-ping(ExaWatcher和OSWatcher中调用的这个命令)。
很奇怪,为什么就db节点报错?
于是,使用infinicheck 带参数-b -g 来检查和配置一下DB节点的IB的SSH连通性:
这里我犯了个错误:这个命令需要配置IB的基于IP的SSH(root),而不是主机名


                        INFINICHECK
                [Network Connectivity, Configuration and Performance]
                [Version IBD VER 2.d ]
ping: unknown host dm01db01-priv
[FAILURE] Host dm01db01-priv is Unreachable and is excluded from testing
ping: unknown host dm01db02-priv
[FAILURE] Host dm01db02-priv is Unreachable and is excluded from testing

 Please supply Infiniband IP addresses only in cell_ib_group

这里很清晰的告诉我们,ping不通,O(∩_∩)O哈哈~,这个就好办了。
接下来,我们手工ping看看:

[root@dm01db01 ~]# ping  dm01db02-priv
ping: unknown host 
[root@dm01db01 ~]# 

那么ping第2个节点的主机名试试看,证实一下是不是解析的问题:

[root@dm01db01 ~]# ping 192.168.10.3
PING 192.168.10.3 (192.168.10.3) 56(84) bytes of data.
64 bytes from 192.168.10.3: icmp_seq=1 ttl=64 time=0.026 ms
64 bytes from 192.168.10.3: icmp_seq=2 ttl=64 time=0.025 ms
^C
--- 192.168.10.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1902ms
rtt min/avg/max/mdev = 0.025/0.025/0.026/0.005 ms
[root@dm01db01 ~]# 

这里我们看到,果然是解析的问题。
由于IB网络是Exadata内部互联用的,因此没有在DNS解析,只在/etc/hosts中解析。
而/etc/hosts文件是由onecommand配置的(除非手工安装,否则使用了onecommand后,所有配置文件都由onecommand根据配置xml文件自动生成)

[root@dm01db01 ~]# cat /etc/hosts
#### BEGIN Generated by Exadata. DO NOT MODIFY ####
127.0.0.1 localhost.localdomain localhost

# 192.168.10.1 dm01db01-priv1.lunar.com dm01db01-priv1
# 192.168.10.2 dm01db01-priv2.lunar.com dm01db01-priv2
此处略去了vip和scan ip
#### END Generated by Exadata ####
#### BEGIN Added by Configuration Utility #### 
192.168.10.1 dm01db01-priv1 dm01db01-priv1.800best.com
192.168.10.10 dm01cel03-priv2 dm01cel03-priv2.800best.com
192.168.10.2 dm01db01-priv2 dm01db01-priv2.800best.com
192.168.10.3 dm01db02-priv1 dm01db02-priv1.800best.com
192.168.10.4 dm01db02-priv2 dm01db02-priv2.800best.com
192.168.10.5 dm01cel01-priv1 dm01cel01-priv1.800best.com
192.168.10.6 dm01cel01-priv2 dm01cel01-priv2.800best.com
192.168.10.7 dm01cel02-priv1 dm01cel02-priv1.800best.com
192.168.10.8 dm01cel02-priv2 dm01cel02-priv2.800best.com
192.168.10.9 dm01cel03-priv1 dm01cel03-priv1.800best.com
#### END Added by Configuration Utility ####
[root@dm01db01 ~]# 

从这里我们看到,IB网络的IP配置格式是错误的,正确的是:
127.0.0.1 localhost.localdomain localhost
错误的是:
192.168.10.1 dm01db01-priv1.lunar.com dm01db01-priv1
修改了上述hosts文件后,
纠正hosts文件后,发现ping主机名的问题解决了:

[root@dm01db01 ~]# ping dm01db01-priv1
PING dm01db01-priv1.800best.com (192.168.10.1) 56(84) bytes of data.
64 bytes from dm01db01-priv1.800best.com (192.168.10.1): icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from dm01db01-priv1.800best.com (192.168.10.1): icmp_seq=2 ttl=64 time=0.010 ms
^C
--- dm01db01-priv1.800best.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1586ms
rtt min/avg/max/mdev = 0.010/0.010/0.010/0.000 ms
[root@dm01db01 ~]# ping dm01db01-priv1.800best.com
PING dm01db01-priv1.800best.com (192.168.10.1) 56(84) bytes of data.
64 bytes from dm01db01-priv1.800best.com (192.168.10.1): icmp_seq=1 ttl=64 time=0.011 ms
64 bytes from dm01db01-priv1.800best.com (192.168.10.1): icmp_seq=2 ttl=64 time=0.008 ms
^C
--- dm01db01-priv1.800best.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1525ms
rtt min/avg/max/mdev = 0.008/0.009/0.011/0.003 ms
[root@dm01db01 ~]# 

这里还有个问题很奇怪,cell节点的hosts文件也是错误的,但是却可以ping通,怀疑跟DNS缓存有关系:

[root@dm01cel02 ~]# cat /etc/hosts
#### BEGIN Generated by Exadata. DO NOT MODIFY ####
127.0.0.1 localhost.localdomain localhost

# 192.168.10.7 dm01cel02-priv1.800best.com dm01cel02-priv1
# 192.168.10.8 dm01cel02-priv2.800best.com dm01cel02-priv2
10.45.1.194 dm01cel02.800best.com dm01cel02
#### END Generated by Exadata ####
#### BEGIN Added by Configuration Utility #### 
192.168.10.1 dm01db01-priv1 dm01db01-priv1.800best.com
192.168.10.10 dm01cel03-priv2 dm01cel03-priv2.800best.com
192.168.10.2 dm01db01-priv2 dm01db01-priv2.800best.com
192.168.10.3 dm01db02-priv1 dm01db02-priv1.800best.com
192.168.10.4 dm01db02-priv2 dm01db02-priv2.800best.com
192.168.10.5 dm01cel01-priv1 dm01cel01-priv1.800best.com
192.168.10.6 dm01cel01-priv2 dm01cel01-priv2.800best.com
192.168.10.7 dm01cel02-priv1 dm01cel02-priv1.800best.com
192.168.10.8 dm01cel02-priv2 dm01cel02-priv2.800best.com
192.168.10.9 dm01cel03-priv1 dm01cel03-priv1.800best.com
#### END Added by Configuration Utility ####
[root@dm01cel02 ~]# ping dm01db01-priv1
PING dm01db01-priv1 (192.168.10.1) 56(84) bytes of data.
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=1 ttl=64 time=0.056 ms
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=2 ttl=64 time=0.064 ms
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=3 ttl=64 time=0.051 ms
^C
--- dm01db01-priv1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2637ms
rtt min/avg/max/mdev = 0.051/0.057/0.064/0.005 ms
[root@dm01cel02 ~]# ping dm01db01-priv1.800best.com
PING dm01db01-priv1 (192.168.10.1) 56(84) bytes of data.
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=2 ttl=64 time=0.027 ms
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=3 ttl=64 time=0.029 ms
^C
--- dm01db01-priv1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2833ms
rtt min/avg/max/mdev = 0.027/0.033/0.043/0.007 ms
[root@dm01cel02 ~]# 

现在,再次使用infinicheck 带参数-b -g 来检查一下DB节点的IB的SSH连通性,这次没问题了(过程很长,而且无需用户输入和交互,因此不列举在这里了)。
配置完成后,测试一下:

[root@dm01db01 oracle.SupportTools]# dcli -g all_ibip_group -l root 'date'
192.168.10.1: Sun Apr  5 08:14:58 CST 2015
192.168.10.2: Sun Apr  5 08:14:58 CST 2015
192.168.10.3: Sun Apr  5 08:14:59 CST 2015
192.168.10.4: Sun Apr  5 08:14:58 CST 2015
192.168.10.5: Sun Apr  5 08:14:58 CST 2015
192.168.10.6: Sun Apr  5 08:14:59 CST 2015
192.168.10.7: Sun Apr  5 08:14:58 CST 2015
192.168.10.8: Sun Apr  5 08:14:58 CST 2015
192.168.10.9: Sun Apr  5 08:14:58 CST 2015
192.168.10.10: Sun Apr  5 08:14:58 CST 2015
[root@dm01db01 oracle.SupportTools]# 

这里我们看到,每个节点有2个IP,是因为从11.2.3.3.0以后,Exadata为了增加带宽,不再绑定IB了,所有的IB私有网络类似一个大的VLAN。

下面我们再次执行一下infinicheck ,因为网络不好,infinicheck执行时间大概要5分钟以上,因此使用screen后台执行:

[root@dm01db01 ~]# screen -S lunar

[root@dm01db01 ~]# date;/opt/oracle.SupportTools/ibdiagtools/infinicheck;date
Sun Apr  5 08:33:13 CST 2015

                        INFINICHECK
                [Network Connectivity, Configuration and Performance]
                [Version IBD VER 2.d ]

 Verifying User Equivalance of user=root to all hosts.
(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)

 Verifying User Equivalance of user=root to all cells.
(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)


                    ####  CONNECTIVITY TESTS  ####
                    [COMPUTE NODES -> STORAGE CELLS]
                           (30 seconds approx.)
[SUCCESS]..............Results OK

[SUCCESS]....... All  can talk to all storage cells

        Verifying Subnet Masks on all nodes
[SUCCESS] ......... Subnet Masks is same across the network

         Prechecking for uniformity of rds-tools on all nodes
[SUCCESS].... rds-tools version is the same across the cluster

        Checking for bad links in the fabric
[SUCCESS].......... No bad fabric links found

                    [COMPUTE NODES -> COMPUTE NODES]
                           (30 seconds approx.)
[SUCCESS]..............Results OK

[SUCCESS]....... All hosts can talk to all other nodes


                    ####  PERFORMANCE TESTS  ####

                    [(1) Storage Cell to Compute Node]
                                 (375 seconds approx)
[  INFO  ].............Performance test between 192.168.10.5 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.6 and 192.168.10.4 has been started.
[  INFO  ].............Performance test between 192.168.10.7 and 192.168.10.2 has been started.
[  INFO  ].............Performance test between 192.168.10.8 and 192.168.10.1 has been started.
[  INFO  ].............Performance test between 192.168.10.9 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.10 and 192.168.10.4 has been started.
[SUCCESS]..............Results OK

                    [(2) Every COMPUTE NODE to another COMPUTE NODE]
                                 (195 seconds approx)
[  INFO  ].............Performance test between 192.168.10.2 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.1 and 192.168.10.4 has been started.
[SUCCESS]..............Results OK

                    [(3) Every COMPUTE NODE to ALL STORAGE CELLS]
                        (looking for SymbolErrors)
                                 (195 seconds approx)
[  INFO  ].............Performance test between 192.168.10.5 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.6 and 192.168.10.4 has been started.
[  INFO  ].............Performance test between 192.168.10.7 and 192.168.10.2 has been started.
[  INFO  ].............Performance test between 192.168.10.8 and 192.168.10.1 has been started.
[  INFO  ].............Performance test between 192.168.10.9 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.10 and 192.168.10.4 has been started.
[SUCCESS]..............Results OK

[SUCCESS]....... No port errors found
INFINICHECK REPORTS SUCCESS FOR NETWORK CONNECTIVITY and PERFORMANCE


----------DIAGNOSTICS -----------

6 Cell ips found: ..
192.168.10.5 | 192.168.10.6 | 192.168.10.7 | 192.168.10.8 | 192.168.10.9 | 192.168.10.10  
4 Host ips found: ..
192.168.10.3 | 192.168.10.4 | 192.168.10.2 | 192.168.10.1  

##########   Host to Cell Connectivity   ##########

Analyzing cells_conntest.log...


[SUCCESS]..... All nodes can talk to all other nodes


Now Analyzing Compute Node-Compute Node connectivity 

##########    Inter-Host Connectivity     ##########

Analyzing hosts_conntest.log...


[SUCCESS]..... All hosts can talk to all its peers

##########     Performance Diagnostics     ##########

###   [(1) STORAGE CELL to COMPUTE NODE    ######

Analyzing perf_cells.log.* logfile(s)....

        --------Throughput results using rds-stress --------
        2300 MB/s and above is expected for runs on quiet machines
dm01db02( 192.168.10.3 ) to dm01cel01( 192.168.10.5 ) : 3987 MB/s...OK
dm01db02( 192.168.10.4 ) to dm01cel01( 192.168.10.6 ) : 4204 MB/s...OK
dm01db01( 192.168.10.2 ) to dm01cel02( 192.168.10.7 ) : 3848 MB/s...OK
dm01db01( 192.168.10.1 ) to dm01cel02( 192.168.10.8 ) : 3876 MB/s...OK
dm01db02( 192.168.10.3 ) to dm01cel03( 192.168.10.9 ) : 3868 MB/s...OK
dm01db02( 192.168.10.4 ) to dm01cel03( 192.168.10.10 ) : 3971 MB/s...OK

##########     Performance Diagnostics     ##########

####      [(2) Every DBNODE to its PEER        ######

Analyzing perf_hosts.log.* logfile(s)....

        --------Throughput results using rds-stress --------
        2300 MB/s and above is expected for runs on quiet machines
dm01db02( 192.168.10.3 ) to dm01db01( 192.168.10.2 ) : 3958 MB/s...OK
dm01db02( 192.168.10.4 ) to dm01db01( 192.168.10.1 ) : 3865 MB/s...OK

-------------------------
Results are available in the file diagnostics.output
Sun Apr  5 08:39:30 CST 2015
[root@dm01db01 ~]#

这里我们看到,每秒的吞吐量在3.8GB/s~4GB左右(注意是大B),很强悍,O(∩_∩)O哈哈~

此条目发表在 故障诊断 分类目录,贴了 , 标签。将固定链接加入收藏夹。

发表评论

电子邮件地址不会被公开。 必填项已用 * 标注

您可以使用这些 HTML 标签和属性: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>