Multipath相关软件介绍

admin

1. device-mapper-multipath

　　即multipath-tools，主要提供multipathd和multipath等工具和 multipath.conf等配置文件。这些工具通过device mapper的ioctr的接口创建和配置multipath，设备创建的多路径设备映射会在/dev /mapper中。

2. device-mapper

　　主要包括两大部分：内核部分和用户部分。内核部分主要由device mapper核心（dm.ko）和一些target driver（md-multipath.ko）。核心完成设备的映射，而target根据映射关系和自身特点具体处理从mappered device 下来的i/o。同时，在核心部分，提供了一个接口，用户通过ioctr可和内核部分通信，以指导内核驱动的行为，比如如何创建mappered device，这些divece的属性等。linux device mapper的用户空间部分主要包括device-mapper这个包。其中包括dmsetup工具和一些帮助创建和配置mappered device的库。这些库主要抽象，封装了与ioctr通信的接口，以便方便创建和配置mappered device。multipath-tool的程序中就需要调用这些库

3. dm-multipath.ko和dm.ko：dm.ko是device mapper驱动。它是实现multipath的基础。dm-multipath其实是dm的一个target驱动。

4 y: }' K- R$ m6 M& b3 n

https://www.cnblogs.com/jianqian ... /09/01/2162574.html

AIX

http://blog.163.com/herod_xiao/blog/static/871883992011819112227689/

1. 检查multipath模块，如果没有相关模块就说明没有安装相关软件包

lsmod | grep dm_multipath

yum -y install device-mapper device-mapper-multipath

$ o" W" k. F- o0 i. i

2. 加载相关模块

modprobe dm-multipath

modprobe dm-round-robin

# V3 }+ q) ?# h5 E+ m

3. 生成multipath配置文件，并配置

/sbin/mpathconf --enable

配置multipath.conf文件

4. 启动multipath服务

service multipathd start

chkcofig multipathd on

[backcolor=rgb(245, 245, 245) !important][url=]

9 T/ X' S! y8 h/ W) X4 ]                               
登录/注册后可看大图

[/url]
[root@tyhr ~]# multipath -llmpathc (36005076801820709e000000000000087) dm-3 IBM,2145size=300G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active  |- 4:0:0:1 sdc 8:32  active ready running  |- 4:0:2:1 sde 8:64  active ready running  |- 5:0:2:1 sdg 8:96  active ready running  `- 5:0:4:1 sdi 8:128 active ready runningmpathb (36005076801820709e000000000000086) dm-2 IBM,2145size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active  |- 4:0:0:0 sdb 8:16  active ready running  |- 4:0:2:0 sdd 8:48  active ready running  |- 5:0:2:0 sdf 8:80  active ready running  `- 5:0:4:0 sdh 8:112 active ready running[backcolor=rgb(245, 245, 245) !important][url=]

- n/ m& ]) o. [2 u% X* K                               
登录/注册后可看大图

[/url]
$ ?7 b9 b1 {3 _" q

; w  \+ W# _, c4 Z( q  b: ~
4 k$ b3 r8 d, ?, P7 ?

如果完成配置后，不能使用multipath -ll查看到任何多路径信息，就需要手工重新扫描HBA信息

一般新增LUN映射给主机后，系统无法直接更新挂载的存储盘，需要重新扫描。

# cd /sys/class/scsi_host/host4/# echo '- - - ' > scan或echo "1" > /sys/class/fc_host/host/issue_lip  //某些存储或系统没有scan文件，可以通issue_lip文件识别
; p' u5 _5 Q5 S0 U7 _/ l0 p3 S
5 a, I3 l  f" d1 x

RHEL5.x 操作系统，新增multipath后，能够识别磁盘，但是通过fdisk分区后，不能显示分区信息（/dev/mapper/dm-app 磁盘不显示/dev/mapper/dm-appp1）。partprob无报错但是没有效果。fdisk w保存后有报错：

WARNING: Re-reading the partition table failed with error 22: Invalid argument.

[backcolor=rgb(255, 255, 255) !important][size=1em]

1   `8 j+ j, O5 O* R- m4 N, W  A 2 3 & L  W+ O- c3 I 4 5 6 7 6 W; \7 h8 P& ? 8 2 D& a# P% D4 H+ U9 ~$ V+ P0 M& f 9 & A' b& V" a( ^" g8 ~( n 10 11 12 13 1 p4 G0 @5 R% |/ v8 p 14 . U  U3 [: S& `  [% E! o7 q 15 7 v" k+ R5 D5 k- c( p 16 17 18 1 j! @, V( C" J

Run "kpartx -a" after FDISK is completed to add all partition mappings on the ( c: j# V' M0 q6 Q& |7 c newly-created multipath device 5 R( H/ O/ [. @' I8 k [root@TYEDI1 host4]# ls -l /dev/mapper/dm-* 4 K9 i" J5 u/ Y% I, f) q  o brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app & s$ ^5 H6 t6 r0 I7 z brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db 8 k7 l! U+ [4 [. K : w; S: b6 `% l  `2 |0 ?" g [root@TYEDI1 host4]# kpartx -a /dev/mapper/dm-app ( Q5 x( W+ i; w3 d: G% P. `4 u, W [root@TYEDI1 host4]# ls -l /dev/mapper/dm-* brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app brw-rw---- 1 root disk 253, 3 Jan 21 23:01 /dev/mapper/dm-app1 7 k; G* Z( M. s, W% t. `& h brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch ' J9 J1 ?- r& i) d) c. C brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db " M" {  R+ o# r . Y7 t5 D! G" T* t& q

- J: ]" U4 ^* K6 f
4 h7 o) r/ S1 Z$ t1 B7 [3 t

9 G1 o4 P  P- \3 d/ o( K

其他配置：

1. udev绑定权限

RHEL6.5，对于multipath的权限，手工去修改几秒后会变回root。所以需要使用udev去绑定好权限。

[backcolor=rgb(245, 245, 245) !important][url=]

                               
登录/注册后可看大图

[/url]
1)搜索对应的配置文件模板：[root@jyrac1 ~]# find / -name 12-*/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules\
" h! Z; h# t5 W5 S) `. r2)根据模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：vi /etc/udev/rules.d/12-dm-permissions.rules# MULTIPATH DEVICES## Set permissions for all multipath devicesENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"# Set permissions for first two partitions created on a multipath device (and detected by kpartx)# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"3)查看多路径对应的底层dm设备：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-34)启动start_udev[root@jyrac2 rules.d]# start_udev Starting udev: [  OK  ]5)查看权限：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk     253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk     253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-36)等30s后再查，权限固定：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk     253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk     253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3[backcolor=rgb(245, 245, 245) !important][url=]

! ?) J4 t7 n2 C, K! y: J8 Z' z                               
登录/注册后可看大图

[/url]


6 Q; T' r1 A) T- _2 w7 q; Z
5 u8 d, N  y$ m0 ?6 X$ P+ i- M

% T" l+ N3 I7 @) L" z
9 Q) r) C' k& C( ~0 {' U

可以使用lsscsi查看识别的磁盘信息：

[backcolor=rgb(245, 245, 245) !important][url=]

                               
登录/注册后可看大图

[/url]
  {' X( d& w! R$ P& s* A2 ^[root@tyhr ~]#  lsscsi [0:0:0:0]    cd/dvd  TSSTcorp DVD-ROM TS-L333H ID03  /dev/sr0 [2:0:0:0]    cd/dvd  IMM      Virtual CD/DVD   0316  /dev/sr1 [3:0:0:0]    disk    IBM-ESXS MBF2300RC        SB27  -       [3:0:1:0]    disk    IBM-ESXS MBF2300RC        SB27  -       [3:1:1:0]    disk    LSILOGIC Logical Volume   3000  /dev/sda [4:0:0:0]    disk    IBM      2145             0000  /dev/sdb [4:0:0:1]    disk    IBM      2145             0000  /dev/sdc [4:0:2:0]    disk    IBM      2145             0000  /dev/sdd [4:0:2:1]    disk    IBM      2145             0000  /dev/sde [5:0:2:0]    disk    IBM      2145             0000  /dev/sdf [5:0:2:1]    disk    IBM      2145             0000  /dev/sdg [5:0:4:0]    disk    IBM      2145             0000  /dev/sdh [5:0:4:1]    disk    IBM      2145             0000  /dev/sdi [backcolor=rgb(245, 245, 245) !important][url=]

                               
登录/注册后可看大图

[/url]
  x) {" }& f* t
+ ]" |* I/ |5 e/ i( Y4 P
3 L# m/ H4 j0 m
; A, y: M. g+ H
) V* n5 O/ }; ?" I5 P  z( a) P' C

1. multipath常用操作：

multipath -r （修改multipath.conf配置文件之后重新加载）
multipath -ll （查看多路径状态）
- [; n! k8 T5 s& I  h9 Q- Fmultipath -v2 （格式化路径，检测路径，合并路径）
multipath -v3 （查看多路径详情blacklist、whitelist和设备wwid）
6 s, r, M% B. P( ?2 j/ s% zmultipath -F （删除现有链路）

2. multipath文件详细介绍

接下来的工作就是要编辑/etc/multipath.conf的配置文件

multipath.conf主要包括blacklist、multipaths、devices三部份的配置
# K: L1 i' N' ~5 j: rblacklist配置
- D; G& ~5 Y% i* f5 [& e* q

blacklist {

; N1 ~7 w" z1 g

　　devnode "^sda"

7 h0 r3 |- G* r7 C* `+ s

}

5 U8 E% R- u) _# IMultipaths部分配置multipaths和devices两部份的配置。
multipaths {
9 Q) V( N2 H) R' Q/ _/ e! j0 K5 s/ q　　multipath {
3 ?% M) R" d6 K　　　　wwid **************** 　　　　　　#此值multipath -v3可以看到
" F3 V3 E  h$ h% k　　　　alias iscsi-dm0 　　　　　　　　#映射后的别名,可以随便取
　　　　path_grouping_policy multibus 　　#路径组策略
　　　　path_checker tur 　　　　　　　　#决定路径状态的方法
6 a4 u$ P1 q' E$ `　　　　path_selector "round-robin 0" 　　#选择那条路径进行下一个IO操作的方法
, L7 [9 g- W/ `; n# L& R; X　　}
* H  G; N0 u2 [& D5 t3 ]% r}
3 V& @9 }9 @! z+ X1 V2 y' c8 rDevices部分配置
devices {
' J% |8 @) Q4 L4 o　　device {
　　　　vendor "iSCSI-Enterprise" 　　　　　　　　　　#厂商名称
　　　　product "Virtual disk" 　　　　　　　　　　　　#产品型号
　　　　path_grouping_policy multibus 　　　　　　　　#默认的路径组策略
　　　　getuid_callout "/sbin/scsi_id -g -u -s /block/%n"　#获得唯一设备号使用的默认程序
- B3 y+ S# t7 l& l! _, {6 U　　　　prio_callout      "/sbin/acs_prio_alua %d" 　　　  #获取有限级数值使用的默认程序
, j1 n. [3 V) }　　　　path_checker readsector0 　　　　　　　　　　#决定路径状态的方法
( Z4 O  t& ^% _' t　　　　path_selector "round-robin 0" 　　#选择那条路径进行下一个IO操作的方法
7 ]4 s4 G& i' e% L& x; U　　　　failback        immediate 　　　　#故障恢复的模式
4 k( G6 l8 a5 W8 K& V- O   　　　no_path_retry      queue 　　　　#在disable queue之前系统尝试使用失效路径的次数的数值
' C* V' u& e% m" b3 P  　　　　rr_min_io       100 　　　　　　#在当前的用户组中，在切换到另外一条路径之前的IO请求的数目
　　}
. s& c8 B( Q/ _. T}
+ Y/ b0 k9 J+ O" [2 Y3 S  s: V8 k

相关参数的标准文档的介绍：

Attribute

Description

1 b; @6 b8 J5 q7 w8 K2 c

wwid

Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this section of themultipath.conf file.

; S, d8 W( ~# B  }5 C- g9 J4 I

alias

Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are using user_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names.

path_grouping_policy

Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include:

failover = 1 path per priority group

' i% }2 }# `. B! i7 E, v3 _

multibus = all valid paths in 1 priority group

/ A3 H3 c' x( q' d. J

group_by_serial = 1 priority group per detected serial number

  }. V+ V0 V6 l- U7 }

group_by_prio = 1 priority group per path priority value

group_by_node_name = 1 priority group per target node name

path_selector

Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include:

round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each.

queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests.

service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput.

- ^8 Z! h, c- o

failback

Manages path group failback.

A value of immediate specifies immediate failback to the highest priority path group that contains active paths.

A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention.

A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover.

A numeric value greater than zero specifies deferred failback, expressed in seconds.

! t- _9 a) v* R5 P

prio

Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include:

const: Set a priority of 1 to all paths.

emc: Generate the path priority for EMC arrays.

alua: Generate the path priority based on the SCSI-3 ALUA settings.

tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit.

ontap: Generate the path priority for NetApp arrays.

rdac: Generate the path priority for LSI/Engenio RDAC controller.

hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode.

hds: Generate the path priority for Hitachi HDS Modular storage arrays.

7 O0 ?7 w; N+ c; r# L7 H- `4 ~2 V' W! V

no_path_retry

A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing.

A value of fail indicates immediate failure, without queueing.

A value of queue indicates that queueing should not stop until the path is fixed.

$ c. h. b; V6 \/ R' S8 G

rr_min_io

Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000.

; c4 Z# d& V6 S$ R8 u

rr_min_io_rq

Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1.

rr_weight

If set to priorities, then instead of sending rr_min_io requests to a path before callingpath_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by the prio function. If set touniform, all path weights are equal.

6 F( ?7 \2 P% y+ ]7 L

flush_on_last_del

If set to yes, then multipath will disable queueing when the last path to a device has been deleted.

+ ~! C8 H% `0 h

multipath与oracle ASM结合

https://www.cnblogs.com/jyzhao/p/7208620.html

已有配置文件：

[backcolor=rgb(245, 245, 245) !important][url=]

                               
登录/注册后可看大图

[/url]
defaults {    polling_interval    30    failback            immediate    no_path_retry       5    rr_min_io           100    path_checker        tur    user_friendly_names yes }devnode_blacklist {        devnode "sda"}devices {    device {        vendor                   "IBM"        product                  "2145"        path_grouping_policy     group_by_prio        prio_callout             "/sbin/mpath_prio_alua /dev/%n"    }    device {        vendor                   "IBM"        product                  "1750500"        path_grouping_policy     group_by_prio        prio_callout             "/sbin/mpath_prio_alua /dev/%n"    }    device {        vendor                   "IBM"        product                  "2107900"        path_grouping_policy     group_by_serial    }    device {        vendor                   "IBM"        product                  "2105800"        path_grouping_policy     group_by_serial    }}multipaths {        multipath {                wwid    36005076801820709e000000000000043                alias   dm-vote01        }}[backcolor=rgb(245, 245, 245) !important][url=]

                               
登录/注册后可看大图

[/url]

, }$ d6 K6 b  g  Q