在Double-E之间配置基于Infiniband的SDP连接(比如,两个Exadata,或者1个Exadata和1个Exaligic之间)

联系:QQ(5163721)

标题:在Double-E之间配置基于Infiniband的SDP连接(比如,两个Exadata,或者1个Exadata和1个Exaligic之间)

作者:Lunar©版权所有[文章允许转载,但必须以链接方式注明源地址,否则追究法律责任.]

在Double-E之间(比如,两个Exadata,或者1个Exadata和1个Exaligic之间),我们可以使用IB来连接
比如两个Exadata之间互相做灾备(ADG)
再比如,Exalogic上有大量的空间,即使是1/4配置的Exalogic上也有几十T的容量空闲(没有部署应用的空间)。
有时候我们会见可以购买了Double-E的客户用这个空间来做Exadata上的数据库的ADG。
为了充分利用IB的传输速度块,低延迟和CPU消耗少等优点,Double-E之间的互联,一般建议通过SDP协议访问Exadata上的数据库
文档建议(Exadata X2和X3),Exadata和Exalogic使用8个IB线进行连接(4跟的话也可以,但是没有冗余,一般不建议)
Exadata 1/4 rack和Exalogic 1/4 rack上分别都有2个叶IB交换机(半配的会有第三个主IB交换机,从Exadata X3开始,出厂时缺省没有最下的主交换机了)
这4个交换机如下:
Exadata-IB1 Exadata–IB2
Exalogic-IB1 Exalogic–IB2

连接方式:
Exadata-IB1——-Exalogic–IB2
Exadata-IB1——-Exalogic–IB1
Exadata-IB2——-Exalogic–IB1
Exadata-IB2——-Exalogic–IB2

配置方式具体如下:
1、修改db node上的/etc/infiniband/openib.conf:

SDP_LOAD=yes

2、在/etc/ofed/libsdp.conf中添加:

use tcp server * *:*
use tcp client * *:*

3、在/etc/modprobe.conf都添加并且reboot节点:

options ib_sdp sdp_zcopy_thresh=0 recv_poll=0 

配置Double-E(比如Exadata和ExalogicIB,或者Exadata和Exadata之间的)网络上配置监听:

192.168.10.21 dm01db01-ibvip.lunar.com dm01db01-ibvip
192.168.10.22 dm01db02-ibvip.lunar.com dm01db02-ibvip

添加network2(for IB):

[root@dm01db01 ~]# srvctl add network -k 2 -S 192.168.8.0/255.255.252.0/bondib0 
[root@dm01db01 ~]# 

验证添加结果:

[root@dm01db01 ~]# crsctl stat res -t | grep net
ora.net1.network
ora.net2.network 					-- Output indicating new Network resource
[root@dm01db01 ~]# 
 
[root@dm01db01 ~]# srvctl config network -k 2
Network exists: 2/192.168.8.0/255.255.252.0/bondib0, type static
[root@dm01db01 ~]# 

添加vip:

[root@dm01db01 ~]# srvctl add vip -n dm01db01 -A dm01db01-ibvip/255.255.252.0/bondib0 -k 2
[root@dm01db01 ~]# srvctl add vip -n dm01db02 -A dm01db02-ibvip/255.255.252.0/bondib0 -k 2
[root@dm01db01 ~]# 

检查vip是否可以正常启动:

[root@dm01db01 ~]# srvctl start vip -i dm01db01-ibvip
[root@dm01db01 ~]#

[root@dm01db02 ~]# srvctl start vip -i dm01db02-ibvip
[root@dm01db02 ~]# 

添加这个IB网络的监听(LISTENER_IB):

[root@dm01db01 ~]# srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522,/SDP:1522
[root@dm01db01 ~]# srvctl start listener -l LISTENER_IB
[root@dm01db01 ~]#

[grid@dm01db01 ~]$ lsnrctl status listener_ib

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 18-DEC-2012 12:43:15

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_IB)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_IB
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                18-DEC-2012 12:41:03
Uptime                    0 days 0 hr. 2 min. 11 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0.3/grid/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/dm01db01/listener_ib/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_IB)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.10.21)(PORT=1522)))
The listener supports no services
The command completed successfully
[grid@dm01db01 ~]$

配置tnsnames.ora

DBFS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = dbfs)
    )
  )

LUNARDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = LUNARDB)
    )
  )


#the following are for IB
LISTENER_IBREMOTE=
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip.lunar.com)(PORT = 1522))
  )

LISTENER_IBLOCAL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip.lunar.com)(PORT = 1522))
    (ADDRESS = (PROTOCOL = SDP)(HOST = dm01db01-ibvip.lunar.com)(PORT = 1522))
  )


LISTENER_IPLOCAL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm0101-vip.lunar.com)(PORT = 1521))
  )


LISTENER_IPREMOTE=
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan.lunar.com)(PORT = 1521))
  )


LUNARDB_IB =
  (DESCRIPTION =
  (LOAD_BALANCE=on)
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = LUNARDB)
    )
  )

修改数据库参数listener_networks:

SQL> alter system set listener_networks='((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))', '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;

System altered.

SQL> 

测试连接,使用oracle用户:

conn sys/oracle@LUNARDB
conn sys/oracle@LUNARDB_IB

在客户端(Exalogic或者其他需要连接db的主机上),需要配置 如下连接串:

LUNARDB_IB =
  (DESCRIPTION =
  (LOAD_BALANCE=on)
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = LUNARDB)
    )
  )

好了,这样Exalogic就可以使用IB连接Exadata了。

此条目发表在 Exadata, 日常运维 分类目录,贴了 , 标签。将固定链接加入收藏夹。

发表评论

电子邮件地址不会被公开。 必填项已用 * 标注

您可以使用这些 HTML 标签和属性: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>