Keepalived+Ceph RBD配置NFS的高可用

admin

原理比较简单，利用ceph的rbd导出nfs，然后配置keepalived，在不同主机映射的rbd块间进行vip漂移。

1. 创建并映射rbd块设备

试验机器为ceph中的osd1和osd2主机，ip分别为192.168.32.3、192.168.32.4，vip为192.168.32.5。

先创建一个rbd块设备，然后在两台机器上导出此相同的块：

• 
  ' ^  _$ _. K) a
  ) w4 l) R. S# Y$ g, d[root@osd1 keepalived]# rbd ls test
  ( `5 Y6 T0 b5 A) @5 R7 C1 {' ^
  $ b( l: f. \5 I/ Y
• 
  % b% E. `+ h" P' f
  * [! |, f8 Y$ J0 S* qtest.img
  8 U2 w; q% t# n
  5 d: S5 Y' ]7 [% w
• 
  7 ?1 M$ h# U  j  G$ h7 f8 A
  + ^4 i. ?4 T' z7 e* j2 X[root@osd1 keepalived]# rbd showmapped
  
  $ P; f! }7 w, I' a6 @8 E
• 
  
  id pool image    snap device
  6 i$ K9 g8 H9 ?6 I( m8 r; h% W
  
• 
  
  6 H! W( b% A, B0  test test.img -    /dev/rbd0
    H) I' D+ \) c  I
  : Y8 i% t/ w: ~, v( e* }
• 
  ' H$ h+ X; Z, I# f: r
  1 @0 w3 p* q: y! f[root@osd2 keepalived]# rbd showmapped
  ! ^& b9 A  Y1 Z9 }3 s" ?
  
• 
  
  , [  Y" k9 w; c" |& S$ R! \. t: n5 did pool image    snap device
  # R: {2 b& s6 Q+ Q1 S
  
• 
  . ]; u1 z" z8 ]. `7 x
  8 ]# u2 X* G+ j/ l+ Z0  test test.img -    /dev/rbd0
  # I# @3 j$ a8 F) h8 b2 ]; b
  
  : w- v2 N$ ~6 K! Q7 g

然后将/dev/rbd0格式化后进行挂载，如/mnt目录。

2. 配置keepalived
# j4 [7 S0 m- p. s' Y' ?! m% ]# \

从http://www.keepalived.org/download.html下载最新的keepalived 1.2.15版本，安装比较简单，直接按照INSTALL的说明进行默认安装即可。为了方便管理keepalived服务，需进行以下操作：

cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/mkdir /etc/keepalivedcp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/cp /usr/local/sbin/keepalived /usr/sbin/chkconfig --add keepalivedchkconfig keepalived onchkconfig --list keepalived

配置/etc/keepalived.conf。osd1的配置如下：

• 
  9 z- G2 e, z% D( D# o, Y1 F. u& ?
  - ]. ?" J; _/ i' \% y1 ][root@osd1 keepalived]# cat keepalived.conf
  
  
• 
  $ h3 u: p$ R7 Z6 g
  global_defs
  
  3 {/ m/ p- L7 {. J
• 
  
  {
  , p* G/ ~& a/ w+ U7 c
  
• 
  ; S" U+ g3 x. @6 Z: N9 \
  2 u2 T- ?* K0 E5 y# n    notification_email
  
  ( o# U' e- ^. I+ g2 \, m
• 
  
  3 Q% r- n5 {9 O* k2 {. Q3 D    {
  
  $ s: _: a% F  c" _/ g/ ^. n& a& P
• 
  
  9 A( k; S# D" a+ X" Y" N/ ?    }
  8 J, Y# O( ]+ [6 Z% v! K
  
• 
  , J1 M( q. d( Z. q! `  p4 k
  % `" ^' F2 X; j; _2 v% T5 C/ C    router_id osd1
  2 j5 q! k& F) H5 z
  ) J8 a. u6 P# k% p/ ~, P
• 
  ) v7 @4 O* a/ V* R) I# v4 @7 k8 b
  , H6 I2 C) \4 d" _) Q9 L}
  1 W8 l3 ]$ W% U; G3 d6 {% Q
  0 j; Z) p# G3 V* `8 y
• 
  - M- E- C' H, T/ e
  ' N7 k8 A; l( X/ O1 X
  
  4 f0 e! Z" E$ C" C& \
• 
  0 q5 X( S) l/ n: M& u" ^
  - Z( ?4 U9 Q* T0 W8 J* wvrrp_instance VI_1 {
  0 @$ ^) t. W1 F  |) M  [2 w
  
• 
  
      state MASTER
  
  
• 
  : X+ ~; {8 Y+ {% E2 O2 K4 ~/ t
  7 J; ^; N9 `' M  ?    interface em1
  5 L' q) k, Q( H0 X7 h. E
  
• 
  0 s; h- }3 p# G$ W; a
      virtual_router_id 100
  
  
• 
  
  4 ~% M/ T% R  R    priority 100
  
  
• 
  ! D$ Q8 _: w6 E( E5 f: n0 f
      advert_int 1
  ( v; R* W# J1 G& j' K
  ) w- h0 R* }( F. d% o& J. `
• 
  0 I3 {+ w9 H# c9 @
  4 [4 i3 c; ~  f! M    authentication {
  8 X4 R. F% ]" D
  
• 
  & E# b+ D' Z. A& m) C
  ' G" h  N1 B! [        auth_type PASS
  
  
• 
  
          auth_pass 1111
  
  
• 
  
      }
  
  
• 
  
      virtual_ipaddress {
  " G7 q( M: O- J, C( [
  7 B% u( B& U( I
• 
  
          192.168.32.5/24
  
  5 s# ]% A& |0 P  ~
• 
  
  & j5 q' n# p& Q- y. D% I) M    }
  
  5 z( z* \3 s/ ~) _& \+ c6 {
• 
  
  * Y1 M6 |7 a  K% J}
  7 H- G0 ]2 e2 C7 P* t
  0 X8 x3 `: u2 `3 p
  

osd2的配置如下：

• 
  ! g  c# S( A) d7 ?; ?7 U2 b/ t) A
  6 Q$ C$ A1 H, A" F  g( q& U0 z[root@osd2 keepalived]# cat keepalived.conf
  
  
• 
  
  global_defs
  
  
• 
  1 C' s( m8 [5 f/ S( ?9 o  c+ U
  # I4 Q& n  E9 f0 X7 B{
  7 a, W" ]) A6 g) ~- g5 ]
  
• 
  
  & U) X' a3 \# C. Y/ N    notification_email
  - z& K; W) Y. k( B0 W
  , I0 \) t* V: q
• 
  
      {
  
  
• 
  * F& i( f6 X$ N. s  J+ H( Y# W
  7 d0 G, w4 s! [- ~* V. ~    #    admin@example.com
  
  
• 
  & U: S) @3 {0 [1 k0 m) i2 P
      }
  
  
• 
  : l; N- _/ o6 F& V: E- w
  ! \( a% j" J' ?5 M1 B1 f    #notification_email_from admin@example.com
  % y$ W2 c) Q9 R8 y
    s% `/ y& h% l. O
• 
  
  ) ]. c8 p" h2 O6 E' ~    #smtp_server 127.0.0.1
  
  
• 
  ' U! W" m4 q: w& i8 v# y4 C" @( d' D
      #stmp_connect_timeout 30
  
  
• 
  $ `& V4 X5 }% T" j; }
  : H. _' d1 F# B/ j    router_id osd2
  
  1 r: v) g! k1 l8 }2 ~9 G
• 
  9 g0 D- x4 Z9 ^* ~1 I* @: {' y$ i
  8 k3 X0 q5 _  `% ~3 c) h6 x0 ^}
  
  
• 
  
  
  1 [4 o! \4 H% O
  $ F4 x8 y( Z- v! y
• 
  & p8 W4 s) g, v% B" P
  vrrp_instance VI_1 {
  
  
• 
  ' ?6 g8 R& h" K/ [7 `
  % Z7 I" u- A* j    state BACKUP
  ) A4 `2 w% a- S( x( e; V9 C3 C$ u
  * V- q. S, S" @! n5 ]+ d* P2 N
• 
    ?/ P$ `( n) A
      interface em1
  
  3 p$ N; `$ E8 W
• 
  
      virtual_router_id 100
  - T* B& }" H2 V. r! T6 Y$ D
  
• 
  * V# O  C9 s0 Y9 Y2 V
  $ }) s  B" \5 C; j    priority 99
  " v' Y# ]! k: Q+ E2 u
  
• 
  
      advert_int 1
  
  * ]: G4 z' a: U" f3 k1 h" E5 V
• 
  4 S3 Q1 s( Q& Y0 }3 M7 P  T0 Z
  " y4 H! @- m4 ]4 S7 d& V    authentication {
  * J: |/ I  _; f7 }, T1 b
  
• 
  - d! t( N7 N8 F+ T5 q1 Q3 [/ _
  8 k  U! A& Q, B' E# t) p$ z        auth_type PASS
  
  / v% T3 T! L( e
• 
  
  3 Q" B* A, Q: z3 x* n        auth_pass 1111
  1 @( D  _+ C3 w
  
• 
  
  + S- c& g# c6 w' D( X% @. V$ c    }
  4 m* Z* b$ \7 M8 g& K4 x. [! `- B
  
• 
  
      notify_master "/etc/keepalived/ChangeToMaster.sh"
  " [7 _+ \- W2 F
  
• 
  # H/ ]0 E( G2 M
  / z# V( V3 b. M( `, _/ J% K5 n    notify_backup "/etc/keepalived/ChangeToBackup.sh"
  ( S$ m. ~5 Y$ ]$ q& L! b
  . e2 }8 p# I8 `. W# Q% O
• 
  
  ( t4 N( c: C) J2 ~# m0 N    virtual_ipaddress {
  
  
• 
  
  ' L" K% M  j+ M3 ^, ~' r        192.168.32.5/24
  
  , x4 e! M( D/ U9 h
• 
  
      }
  ; c/ f0 L. T6 b& S& H
  5 }6 N; H. `& q$ f6 G/ P! U
• 
  ; T6 D2 s5 j0 r: C
  8 i. c) |3 N% V}
  
  ) c! c6 d# D# p
  

在osd2写了两个控制脚本，用于osd2的keepalived状态改变时执行。ChangeToMaster.sh：

[root@osd2 keepalived]# cat ChangeToMaster.sh#!/bin/bashservice nfs startssh lm "umount -f /mnt"ssh lm "mount -t nfs 192.168.32.5:/mnt /mnt"

ChangeToBackup.sh:

[root@osd2 keepalived]# cat ChangeToBackup.sh#!/bin/bashssh lm "umount -f /mnt"ssh osd1 "service nfs stop"ssh osd1 "umount /mnt"ssh osd1 "rbd unmap /dev/rbd0"ssh osd1 "rbd map test/test.img"ssh osd1 "mount /dev/rbd0 /mnt"ssh osd1 "service nfs start"ssh lm "mount -t nfs 192.168.32.5:/mnt /mnt"service nfs stopumount /mntrbd unmap /dev/rbd0rbd map test/test.imgmount /dev/rbd03. 配置nfs
, q% b9 c  ?- t0 k

在ceph的一个节点利用rbd map一个块设备，然后格式化并挂载在一个目录，如/mnt。在此节点上安装nfs的rpm包：

yum -y install nfs-utils

设置挂载目录：

cat /etc/exports/mnt 192.168.101.157(rw,async,no_subtree_check,no_root_squash)/mnt 192.168.108.4(rw,async,no_subtree_check,no_root_squash)

启动并导出：

service nfs startchkconfig nfs onexportfs -r

客户端查看一下：

showmount -e mon0Export list for mon0:/mnt 192.168.108.4,192.168.101.157

然后挂载：

mount -t nfs mon0:/mnt /mnt

需要注意的是，NFS默认是用UDP协议，如果网络不稳定，换成TCP协议即可：

mount -t nfs mon0:/mnt /mnt -o proto=tcp -o nolock

4. 测试

关闭osd1的网卡em1查看结果：

[root@osd1 keepalived]# ifdown em1[root@osd1 keepalived]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: em1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000    link/ether c8:1f:66:de:5e:65 brd ff:ff:ff:ff:ff:ff

查看osd2的网卡：

[root@osd2 keepalived]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether c8:1f:66:f7:61:5d brd ff:ff:ff:ff:ff:ff    inet 192.168.32.4/24 brd 192.168.32.255 scope global em1       valid_lft forever preferred_lft forever    inet 192.168.32.5/24 scope global secondary em1       valid_lft forever preferred_lft forever    inet6 fe80::ca1f:66ff:fef7:615d/64 scope link       valid_lft forever preferred_lft forever

vip已经漂移到osd2，查看客户端挂载情况：

[root@lm /]# df -hTFilesystem        Type   Size  Used Avail Use% Mounted on/dev/sda1         ext4   454G   79G  353G  19% /tmpfs             tmpfs  1.7G  4.6M  1.7G   1% /dev/shm192.168.32.5:/mnt nfs    100G   21G   80G  21% /mnt

打开osd1的网卡em1：

[root@osd1 keepalived]# ifup em1Determining if ip address 192.168.32.3 is already in use for device em1...[root@osd1 keepalived]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether c8:1f:66:de:5e:65 brd ff:ff:ff:ff:ff:ff    inet 192.168.32.3/24 brd 192.168.32.255 scope global em1       valid_lft forever preferred_lft forever    inet 192.168.32.5/24 scope global secondary em1       valid_lft forever preferred_lft forever    inet6 fe80::ca1f:66ff:fede:5e65/64 scope link       valid_lft forever preferred_lft forever

osd2的em1：

[root@osd2 keepalived]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether c8:1f:66:f7:61:5d brd ff:ff:ff:ff:ff:ff    inet 192.168.32.4/24 brd 192.168.32.255 scope global em1       valid_lft forever preferred_lft forever    inet6 fe80::ca1f:66ff:fef7:615d/64 scope link       valid_lft forever preferred_lft forever

现在的客户端：

[root@lm /]# df -hTFilesystem        Type   Size  Used Avail Use% Mounted on/dev/sda1         ext4   454G   79G  353G  19% /tmpfs             tmpfs  1.7G  4.6M  1.7G   1% /dev/shm192.168.32.5:/mnt nfs    100G   21G   80G  21% /mnt[root@lm /]# ls /mnt31.txt  a.txt  b.txt  c.txt  etc  linux-3.17.4  linux-3.17.4.tar  m2.txt  test.img  test.img2

也可以为ceph rbd的iscsi配置vip，后面会进行记录。