Keepalived+Ceph RBD配置NFS的高可用

admin

原理比较简单，利用ceph的rbd导出nfs，然后配置keepalived，在不同主机映射的rbd块间进行vip漂移。

1. 创建并映射rbd块设备

试验机器为ceph中的osd1和osd2主机，ip分别为192.168.32.3、192.168.32.4，vip为192.168.32.5。

先创建一个rbd块设备，然后在两台机器上导出此相同的块：

• 
  
  [root@osd1 keepalived]# rbd ls test
  " X' K. ?/ ?! o9 U# Q
  
• 
  
  9 |4 Z4 \1 e9 t- ^7 {test.img
  
  - B- v' ?9 Y/ R& E2 J. Z
• 
  0 y+ ~/ C; _" [( w% c
  : ~( ^* U. x% i# u) ~$ t[root@osd1 keepalived]# rbd showmapped
  0 u( H5 B; d5 A9 `! \
  ' q/ Q% t; t- N; k% o- }! b4 b) q2 q/ T: k
• 
  
  # ?5 `& o' M+ B% e/ vid pool image    snap device
  / j* r& p  p/ ^$ |
  
• 
  3 |1 b# P2 f! z. Z7 j
  3 ~& Z2 ?, c9 O+ B% `0  test test.img -    /dev/rbd0
  " i  y4 V6 ]. {8 n$ J& \
  7 W5 S# S; i1 E! T  f$ A+ i  m
• 
  
  0 N, N0 u1 |9 ][root@osd2 keepalived]# rbd showmapped
  
  
• 
  2 K, ^* ^" E0 u4 \0 P+ C9 S4 e9 Q
  " B, ]" N6 X4 y. l8 A+ |. R! ^( A# ?' y6 qid pool image    snap device
  
  % _" V# n  g4 V: F6 x0 K2 n( U  c
• 
  8 c) q/ a0 L# @; v6 z
  0  test test.img -    /dev/rbd0
  6 z1 |3 q- S# M2 e! y+ c
  7 \, B7 Z- a  I3 A; {3 z% T
  

然后将/dev/rbd0格式化后进行挂载，如/mnt目录。

2. 配置keepalived

从http://www.keepalived.org/download.html下载最新的keepalived 1.2.15版本，安装比较简单，直接按照INSTALL的说明进行默认安装即可。为了方便管理keepalived服务，需进行以下操作：

cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/mkdir /etc/keepalivedcp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/cp /usr/local/sbin/keepalived /usr/sbin/chkconfig --add keepalivedchkconfig keepalived onchkconfig --list keepalived

配置/etc/keepalived.conf。osd1的配置如下：

• 
  8 {& V! e, t1 c# [; ~3 t
  [root@osd1 keepalived]# cat keepalived.conf
  ' B* p# L) i0 V
  
• 
  
  ) W# I; S* s% \global_defs
  $ ^9 ]2 F0 G) k1 g% d2 y6 r
  - H& j: n+ ~; c& \5 i
• 
  ' b" g% P+ Z. {4 P/ o9 }- Z% V1 ^; k, u
  + ~( Z$ n, a# }" _: F{
    ], g% C% b) u5 `# U9 [; F' B8 z8 }8 b% w
  % c, E' q0 Q/ H- r1 o- V: v
• 
  
      notification_email
  . Q* e* d9 b3 e! e; E
  0 Z+ Y: q4 Q7 A8 h) y6 t& |* u' w
• 
  
      {
  : \) o- A7 Z1 s* s3 P
  8 k. e- K2 \. v; }
• 
  
  ( m. k; c& e6 X0 i; n    }
  
  
• 
  
      router_id osd1
  * F' k( x* F  v. X
  
• 
  2 \3 `" I, N- x# m
  }
  
  / K1 Y" q4 q: o5 h3 C) G6 A* c
• 
  
  5 U; G  v3 W6 \' Y5 e7 W3 c
    G* E0 H. @5 ~5 w+ B8 k
  5 f) \4 `+ j% @3 S* k
• 
  
  vrrp_instance VI_1 {
  & V+ U& e) ]6 \' `# `- u) B. @
  - x6 k0 r' Z$ a* K" s+ y$ f+ l( _
• 
  ' E2 V. d/ a" l; T
      state MASTER
  - H, x+ V: R( f# v
  
• 
  7 Y3 W+ H! Z  i3 e  a
      interface em1
  
  
• 
  
      virtual_router_id 100
  6 Q# Y# o$ f3 R# z
  % s+ B6 b6 ~1 S6 l
• 
  
      priority 100
  
  6 \0 M/ A- q& R$ i, Q% X. l
• 
  ; [# O( l/ N9 ?2 q1 R: k) I  b2 ~
      advert_int 1
  
  
• 
  
      authentication {
  
    j# s' X' k5 w8 C4 q6 V
• 
  , O: d, S6 G" Y7 r" v9 H
  3 {- X3 E/ S! L7 W9 {        auth_type PASS
  
  
• 
  
  2 d4 C+ @2 B' O" _; S* B        auth_pass 1111
  
  , g0 H, J: g& f0 F+ u* _( P, ?$ k
• 
  
      }
  " a1 c7 W) h7 W5 d8 e/ h; {
  0 g, v2 A7 x- ?, S6 {
• 
  6 `# w3 J1 q% r) }& ^
  3 }! p# d$ h2 N- J7 X2 j1 u% s1 J    virtual_ipaddress {
  
  5 w) ?! X3 q8 [; X( w
• 
  # m1 G7 \5 U; Q: i+ ~0 G6 S. S' T
          192.168.32.5/24
  5 l* B  i( Q$ [" }$ T
  
• 
  - S0 i) a. G- |. \
  4 O' t6 J1 `) ^5 K+ u2 W6 a    }
  & `9 a& L2 _$ N  }2 X
  
• 
  $ c+ m& o7 z* L2 D7 \
  ) C  X! ]; f  Y& i# f}
  
  
  

osd2的配置如下：

• 
  
  [root@osd2 keepalived]# cat keepalived.conf
  
  
• 
  9 c5 w$ v$ i; a
    S& D$ [5 x4 x- `global_defs
  # U. I! l) Y: T+ O
  - G/ v6 `6 Q7 T: a- z$ w% b9 u
• 
  " R/ M0 h5 |" u& Y3 U! J, Z) K9 ]% H( E
  {
  : s' L( B9 Q: R4 b- F$ j, t
  
• 
  : }$ a. p" A1 b+ {
  0 u6 ~3 e; F# n    notification_email
  4 g$ M4 b9 Q) Q  c. m1 @' S. r
  
• 
  
      {
  
  , W  @/ c- `9 \+ E- H& I
• 
  4 r* B9 w% O9 c6 }) `
      #    admin@example.com
  
  
• 
  ( K7 ~2 X) N1 F% m+ t- K
  ' m' \& B6 D2 Y  E5 X" Q& G    }
  0 w# T# A7 I& D$ e) a, W2 P# s
  
• 
  4 c% T0 ]: E2 T0 p4 C' J; T
      #notification_email_from admin@example.com
  
  7 N- r' X, P+ I! g' j1 Z; P
• 
  9 |, }* f% a: \: {  J
      #smtp_server 127.0.0.1
  3 y' \0 b! i! m/ l
  * ^: g  P$ A- V. V" Z* L6 ^
• 
  
      #stmp_connect_timeout 30
  
  
• 
  
  5 ^. F3 m! B4 t0 y4 ^/ g) f1 O" s% B  A    router_id osd2
  ' ~' o' T, p# l% z
  ( ^- R5 i4 T+ P. G
• 
  
  ' i$ m- a0 g& F0 _# b4 Y: ?}
  
  ' t( F/ u* Q; I  M4 I, T
• 
  
  ! i7 W# o% F+ K
  
  
• 
  
  vrrp_instance VI_1 {
  
  7 Y8 D( v  S; V. f
• 
  
  & V2 r7 Q1 N9 j  g, O  _* y" w    state BACKUP
  # G" ?5 C# V* N' l9 Z1 o/ e% @
  $ U9 B8 r' j; V6 |' s7 `
• 
  4 Y9 Y  R( T/ Q/ i0 V7 ]: m
  ; V7 T! J" Q+ B: \    interface em1
  ; D, ~  L" p+ Y/ p) ]0 \
  
• 
  
      virtual_router_id 100
  
  0 y8 |3 n, I9 _( F# n3 H
• 
  
  / v0 R2 [6 b- j7 J* o( |) U    priority 99
  1 k5 U% M1 T. p) l/ d( U
  4 U7 {# X+ k6 G; u. G$ F# i: S
• 
  
  ! {! H* ~4 `+ Q! t# x3 I& n    advert_int 1
  $ h! Y. n4 z: a! W- W  ]0 ^
  
• 
  ) ?. c  q, O+ u
  3 F/ y' n% S2 s$ I! o2 S" t( G8 _    authentication {
  4 N& z: g: i) b* j
  7 K1 Z4 g" q( _9 p0 T% _: c0 D
• 
  
          auth_type PASS
  
  2 B6 @! Y3 k+ h! z0 F
• 
  
          auth_pass 1111
  ) R- f* r) M6 T- U/ x1 [
    V  W2 b% d  N% s
• 
  8 m6 W* z+ z- i1 |' |, {8 t: |0 Q
      }
  
  
• 
  $ b) ?2 u6 ?  W( S7 I
  8 M) L$ |. S2 P" O' y    notify_master "/etc/keepalived/ChangeToMaster.sh"
  
  
• 
  
  & k- I3 W& q6 b; |# S    notify_backup "/etc/keepalived/ChangeToBackup.sh"
  
  ' J$ G2 h7 l) f  b6 h$ Z6 e# Q2 D
• 
  
      virtual_ipaddress {
  & W  }+ T$ t- `( j8 B7 i( {5 }  }/ M0 ^
  & O6 s5 r+ S) t- ^+ u' z1 |# n
• 
  ( L! m5 T1 r7 X- o6 V7 x$ c3 V
          192.168.32.5/24
  
  & E" S# e2 `9 m& B. G( ?
• 
  . a. j# {# u( Z8 I" N
  ; E8 i0 \$ Z- t1 g* B    }
  , Q8 V7 y2 K9 v; g. ?
  
• 
  
  }
  
  8 Q  H+ h! k+ _
  

在osd2写了两个控制脚本，用于osd2的keepalived状态改变时执行。ChangeToMaster.sh：

[root@osd2 keepalived]# cat ChangeToMaster.sh#!/bin/bashservice nfs startssh lm "umount -f /mnt"ssh lm "mount -t nfs 192.168.32.5:/mnt /mnt"

ChangeToBackup.sh:

[root@osd2 keepalived]# cat ChangeToBackup.sh#!/bin/bashssh lm "umount -f /mnt"ssh osd1 "service nfs stop"ssh osd1 "umount /mnt"ssh osd1 "rbd unmap /dev/rbd0"ssh osd1 "rbd map test/test.img"ssh osd1 "mount /dev/rbd0 /mnt"ssh osd1 "service nfs start"ssh lm "mount -t nfs 192.168.32.5:/mnt /mnt"service nfs stopumount /mntrbd unmap /dev/rbd0rbd map test/test.imgmount /dev/rbd03. 配置nfs
, V( \7 e* ^- A

在ceph的一个节点利用rbd map一个块设备，然后格式化并挂载在一个目录，如/mnt。在此节点上安装nfs的rpm包：

yum -y install nfs-utils

设置挂载目录：

cat /etc/exports/mnt 192.168.101.157(rw,async,no_subtree_check,no_root_squash)/mnt 192.168.108.4(rw,async,no_subtree_check,no_root_squash)

启动并导出：

service nfs startchkconfig nfs onexportfs -r

客户端查看一下：

showmount -e mon0Export list for mon0:/mnt 192.168.108.4,192.168.101.157

然后挂载：

mount -t nfs mon0:/mnt /mnt

需要注意的是，NFS默认是用UDP协议，如果网络不稳定，换成TCP协议即可：

mount -t nfs mon0:/mnt /mnt -o proto=tcp -o nolock

4. 测试
/ }- ?* [. C9 R) L4 R' w; [+ z

关闭osd1的网卡em1查看结果：

[root@osd1 keepalived]# ifdown em1[root@osd1 keepalived]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: em1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN qlen 1000    link/ether c8:1f:66:de:5e:65 brd ff:ff:ff:ff:ff:ff

查看osd2的网卡：

[root@osd2 keepalived]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether c8:1f:66:f7:61:5d brd ff:ff:ff:ff:ff:ff    inet 192.168.32.4/24 brd 192.168.32.255 scope global em1       valid_lft forever preferred_lft forever    inet 192.168.32.5/24 scope global secondary em1       valid_lft forever preferred_lft forever    inet6 fe80::ca1f:66ff:fef7:615d/64 scope link       valid_lft forever preferred_lft forever

vip已经漂移到osd2，查看客户端挂载情况：

[root@lm /]# df -hTFilesystem        Type   Size  Used Avail Use% Mounted on/dev/sda1         ext4   454G   79G  353G  19% /tmpfs             tmpfs  1.7G  4.6M  1.7G   1% /dev/shm192.168.32.5:/mnt nfs    100G   21G   80G  21% /mnt

打开osd1的网卡em1：

[root@osd1 keepalived]# ifup em1Determining if ip address 192.168.32.3 is already in use for device em1...[root@osd1 keepalived]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether c8:1f:66:de:5e:65 brd ff:ff:ff:ff:ff:ff    inet 192.168.32.3/24 brd 192.168.32.255 scope global em1       valid_lft forever preferred_lft forever    inet 192.168.32.5/24 scope global secondary em1       valid_lft forever preferred_lft forever    inet6 fe80::ca1f:66ff:fede:5e65/64 scope link       valid_lft forever preferred_lft forever

osd2的em1：

[root@osd2 keepalived]# ip a1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo       valid_lft forever preferred_lft forever    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000    link/ether c8:1f:66:f7:61:5d brd ff:ff:ff:ff:ff:ff    inet 192.168.32.4/24 brd 192.168.32.255 scope global em1       valid_lft forever preferred_lft forever    inet6 fe80::ca1f:66ff:fef7:615d/64 scope link       valid_lft forever preferred_lft forever

现在的客户端：

[root@lm /]# df -hTFilesystem        Type   Size  Used Avail Use% Mounted on/dev/sda1         ext4   454G   79G  353G  19% /tmpfs             tmpfs  1.7G  4.6M  1.7G   1% /dev/shm192.168.32.5:/mnt nfs    100G   21G   80G  21% /mnt[root@lm /]# ls /mnt31.txt  a.txt  b.txt  c.txt  etc  linux-3.17.4  linux-3.17.4.tar  m2.txt  test.img  test.img2

也可以为ceph rbd的iscsi配置vip，后面会进行记录。