|
1. device-mapper-multipath 即multipath-tools，主要提供multipathd和multipath等工具和 multipath.conf等配置文件。这些工具通过device mapper的ioctr的接口创建和配置multipath，设备创建的多路径设备映射会在/dev /mapper中。 2. device-mapper 主要包括两大部分：内核部分和用户部分。内核部分主要由device mapper核心（dm.ko）和一些target driver（md-multipath.ko）。核心完成设备的映射，而target根据映射关系和自身特点具体处理从mappered device 下来的i/o。同时，在核心部分，提供了一个接口，用户通过ioctr可和内核部分通信，以指导内核驱动的行为，比如如何创建mappered device，这些divece的属性等。linux device mapper的用户空间部分主要包括device-mapper这个包。其中包括dmsetup工具和一些帮助创建和配置mappered device的库。这些库主要抽象，封装了与ioctr通信的接口，以便方便创建和配置mappered device。multipath-tool的程序中就需要调用这些库 3. dm-multipath.ko和dm.ko：dm.ko是device mapper驱动。它是实现multipath的基础。dm-multipath其实是dm的一个target驱动。
, @% F# f: [( s. k: hAIX http://blog.163.com/herod_xiao/blog/static/871883992011819112227689/
# v, _- N, J; m8 u$ |! A9 m# K8 s
9 W2 E9 S" o9 p4 }1 R1. 检查multipath模块，如果没有相关模块就说明没有安装相关软件包 lsmod | grep dm_multipath yum -y install device-mapper device-mapper-multipath / Z4 B- ]5 N+ l. N0 l
2. 加载相关模块 modprobe dm-multipath modprobe dm-round-robin ; I4 C8 {6 m+ b P H' @8 k4 f5 a
3. 生成multipath配置文件，并配置 /sbin/mpathconf --enable 配置multipath.conf文件
6 ~- l; b6 x/ E& j5 B) P4. 启动multipath服务 service multipathd start chkcofig multipathd on [backcolor=rgb(245, 245, 245) !important][url=] [/url]
5 x; R2 t6 N& L- ?3 ?[root@tyhr ~]# multipath -llmpathc (36005076801820709e000000000000087) dm-3 IBM,2145size=300G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:1 sdc 8:32 active ready running |- 4:0:2:1 sde 8:64 active ready running |- 5:0:2:1 sdg 8:96 active ready running `- 5:0:4:1 sdi 8:128 active ready runningmpathb (36005076801820709e000000000000086) dm-2 IBM,2145size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:0 sdb 8:16 active ready running |- 4:0:2:0 sdd 8:48 active ready running |- 5:0:2:0 sdf 8:80 active ready running `- 5:0:4:0 sdh 8:112 active ready running[backcolor=rgb(245, 245, 245) !important][url=][/url]
! q4 r! n( ~3 H
9 k: U- ]" F0 B2 E+ q. n
& x8 E% a# C: l# E/ P1 H7 O' u1 D- i3 H F6 O v" g$ o# i C
如果完成配置后，不能使用multipath -ll查看到任何多路径信息，就需要手工重新扫描HBA信息 一般新增LUN映射给主机后，系统无法直接更新挂载的存储盘，需要重新扫描。 # cd /sys/class/scsi_host/host4/# echo '- - - ' > scan或echo "1" > /sys/class/fc_host/host/issue_lip //某些存储或系统没有scan文件，可以通issue_lip文件识别
8 V9 b3 { u E, d) v9 u
7 }1 ]9 i5 a2 K$ l* D2 z; e* TRHEL5.x 操作系统，新增multipath后，能够识别磁盘，但是通过fdisk分区后，不能显示分区信息（/dev/mapper/dm-app 磁盘不显示/dev/mapper/dm-appp1）。partprob无报错但是没有效果。fdisk w保存后有报错： WARNING: Re-reading the partition table failed with error 22: Invalid argument. [backcolor=rgb(255, 255, 255) !important][size=1em]1
0 d3 d. D. Z" R& M( M2 2 \; Z1 m& `- [6 y3 G
3
* }8 R2 j) [, V$ ]8 N' i4
o z% O" A) _' N! ^5
- q1 ~* |* x6 [# c7 c5 T- G6 ) i8 n% G. f9 ^4 |3 A: x% x; s3 K
7 : t% B% g8 S. X/ \! l! z2 P9 ?
8 4 E" ~7 @& r0 C' [
9
+ Y: x0 a/ Z: v3 M3 D5 B10 / D: X" } D: B; t C, n, m
11
( N' N' U& ?: K- K3 e12 ! ]) I, l% X( j8 H+ d( L
13 & c% [- B7 ]) W4 L" Y. F. G; ?
14
# r$ O8 K4 Q" {+ _0 j9 G* g0 r15 % ^( H' g) I1 ~9 n+ I
16 0 i$ p* r$ ?# M" G
17 E9 W5 K0 h7 {0 ?$ N/ R! z# u
18
8 y3 R, H/ t. \' G7 m% o | Run "kpartx -a" after FDISK is completed to add all partition mappings on the
! g! |. [7 C/ tnewly-created multipath device
! U6 v9 I' ]4 W$ i8 U# e* |' j8 g4 X! Z3 y! w+ H; k( W
: A2 J% E- a! N x! F
$ a5 r6 N- `* s) C[root@TYEDI1 host4]# ls -l /dev/mapper/dm-* + l Y5 N& n3 {
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app
t5 N) }1 a8 L0 M4 v3 f9 vbrw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch & e9 j; j( y' x3 P
brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db
9 X4 z6 l7 O9 k7 w2 o; P
: R% i- V) }7 E[root@TYEDI1 host4]# kpartx -a /dev/mapper/dm-app 6 S8 W! U8 o/ A
) n: H% M( y3 w! T* L4 Q; i
$ L( r3 j, y3 u* g[root@TYEDI1 host4]# ls -l /dev/mapper/dm-* * j. Y' L0 V2 o9 z1 P
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app - M" k% H" N- `- d3 n
brw-rw---- 1 root disk 253, 3 Jan 21 23:01 /dev/mapper/dm-app1
8 ]3 G! ~& ^; ~( ?: Qbrw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
: @( `2 M! f% B0 Rbrw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db ! { j6 N1 h: z! Z; ]3 ^2 B
|
5 M+ z/ B' K- R8 D6 T }9 i2 M
$ s4 k1 ?7 G: y7 B
; g# j. V' k U
5 V% c0 t& I7 x$ N5 T
) O- s# Y* q' i" R6 X7 J$ f4 J
7 X# `+ b# ?1 z0 j, ~; _' m 其他配置： 1. udev绑定权限 RHEL6.5，对于multipath的权限，手工去修改几秒后会变回root。所以需要使用udev去绑定好权限。 [backcolor=rgb(245, 245, 245) !important][url=][/url]* i/ s3 E. Z0 H
1)搜索对应的配置文件模板：[root@jyrac1 ~]# find / -name 12-*/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules\% S Q4 q9 Z0 F# e
2)根据模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：vi /etc/udev/rules.d/12-dm-permissions.rules# MULTIPATH DEVICES## Set permissions for all multipath devicesENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"# Set permissions for first two partitions created on a multipath device (and detected by kpartx)# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"3)查看多路径对应的底层dm设备：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-34)启动start_udev[root@jyrac2 rules.d]# start_udev Starting udev: [ OK ]5)查看权限：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-36)等30s后再查，权限固定：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3[backcolor=rgb(245, 245, 245) !important][url=][/url]* k4 Q0 s% u! ^9 j; j, H; V
5 u5 b1 o( _" \6 ~4 y
5 h7 v7 e0 b e4 v; P8 d5 Y& V2 U1 \6 T3 b
: t. ~4 W t: I; C" S4 J
; H; ~8 W* v4 ]" W% J8 m; H k
. {1 E8 N6 C. V8 \' K# w! N
可以使用lsscsi查看识别的磁盘信息： [backcolor=rgb(245, 245, 245) !important][url=][/url]+ h. S h6 @/ [% U. t2 c
[root@tyhr ~]# lsscsi [0:0:0:0] cd/dvd TSSTcorp DVD-ROM TS-L333H ID03 /dev/sr0 [2:0:0:0] cd/dvd IMM Virtual CD/DVD 0316 /dev/sr1 [3:0:0:0] disk IBM-ESXS MBF2300RC SB27 - [3:0:1:0] disk IBM-ESXS MBF2300RC SB27 - [3:1:1:0] disk LSILOGIC Logical Volume 3000 /dev/sda [4:0:0:0] disk IBM 2145 0000 /dev/sdb [4:0:0:1] disk IBM 2145 0000 /dev/sdc [4:0:2:0] disk IBM 2145 0000 /dev/sdd [4:0:2:1] disk IBM 2145 0000 /dev/sde [5:0:2:0] disk IBM 2145 0000 /dev/sdf [5:0:2:1] disk IBM 2145 0000 /dev/sdg [5:0:4:0] disk IBM 2145 0000 /dev/sdh [5:0:4:1] disk IBM 2145 0000 /dev/sdi [backcolor=rgb(245, 245, 245) !important][url=][/url]
9 w4 K& E; e3 H% [; v
3 _ D- O4 {; y, O) C8 [, s. z' G6 ?/ E; D
8 @# M6 x3 ?. i$ i$ o( Z/ Q
$ u- F, }% c' ?% }' L. V1 A3 g7 _
4 s/ n! ? O. P* {1. multipath常用操作： multipath -r （修改multipath.conf配置文件之后重新加载）
5 m- ~& H+ @ Ymultipath -ll （查看多路径状态）
/ R# I+ ^, o omultipath -v2 （格式化路径，检测路径，合并路径）
: G. i, L- y2 O+ t) t5 amultipath -v3 （查看多路径详情blacklist、whitelist和设备wwid）: F# @7 u; m' i+ U$ M3 R- Y
multipath -F （删除现有链路） 2. multipath文件详细介绍 接下来的工作就是要编辑/etc/multipath.conf的配置文件 multipath.conf主要包括blacklist、multipaths、devices三部份的配置7 w6 Q, T0 ?9 S3 n2 ?
blacklist配置5 t: _7 K0 ^. V# z2 k- Y6 N4 z/ }
blacklist { 9 ]. j* X9 r" c
devnode "^sda" ! X5 n" k2 t1 U% D! X& m
} % H' l, p5 V" q& G
Multipaths部分配置multipaths和devices两部份的配置。
7 l9 W( P, S; I* g+ b, Qmultipaths {
% f. j. {6 L. Y# V2 F4 y multipath {: _- c8 j. X! S" a% U+ x
wwid **************** #此值multipath -v3可以看到6 j i" @6 F3 u- y4 A
alias iscsi-dm0 #映射后的别名,可以随便取: m; r+ v+ y( g* Y, O2 o# S; ^
path_grouping_policy multibus #路径组策略
# J% c7 ^- e7 l# X path_checker tur #决定路径状态的方法
. ]% {& @* b( t4 D( ^% W path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
! {8 Z" x/ D& h. G }
1 d. q: a9 u6 ?2 b$ T* C5 f}
8 l% I3 y7 G4 x& L0 i9 XDevices部分配置
0 {5 j+ o1 I9 X6 S2 W5 ldevices {: x7 o* A" f7 r$ J' i/ M2 t; t
device {) K1 A# \+ k3 Q- i' f
vendor "iSCSI-Enterprise" #厂商名称3 R( z- U/ x& k8 q# E
product "Virtual disk" #产品型号
2 a# b; ~* D& g# O* Y. |4 f path_grouping_policy multibus #默认的路径组策略
) J6 p$ h) _: v( D, P, Y1 I getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #获得唯一设备号使用的默认程序
1 C+ ]( _1 d3 L4 K( v: r. A& _ prio_callout "/sbin/acs_prio_alua %d" #获取有限级数值使用的默认程序
( H/ H! T$ Q/ X. m! j path_checker readsector0 #决定路径状态的方法* c: e9 h( G, A, J# Z @) Z
path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
' t+ W1 \" l5 m1 C r failback immediate #故障恢复的模式
# k8 b/ W- Q8 G/ j/ X% g no_path_retry queue #在disable queue之前系统尝试使用失效路径的次数的数值
! u7 i% k7 {4 Q0 W! M2 ? P l rr_min_io 100 #在当前的用户组中，在切换到另外一条路径之前的IO请求的数目
- Q3 B) W" X. [2 D: t* P) D; F. w }% d8 r1 c# n% M+ d8 Y
}
, ?2 n# b' H% Q, B相关参数的标准文档的介绍： Attribute Description
9 j' {2 s4 z9 Z n m7 S" ?3 Ewwid Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this section of themultipath.conf file.
; ]! ]: R: c; D/ N' w! x* Qalias Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are using user_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names. " ^( j! p5 ]" R1 ^! J# s
path_grouping_policy Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include:
1 g( t5 C( j" `! J$ Efailover = 1 path per priority group
5 G/ V0 q% p! S; K' Pmultibus = all valid paths in 1 priority group
) S7 ` m- e, V7 Ugroup_by_serial = 1 priority group per detected serial number
- i0 n& u5 c! ?: o6 a4 Agroup_by_prio = 1 priority group per path priority value * F( u1 b. r. J" l9 q
group_by_node_name = 1 priority group per target node name
8 f7 S) @/ n9 G7 @path_selector Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include: | round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each. | queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests. | service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput. |
+ W+ H7 l8 l- w/ \failback Manages path group failback. | A value of immediate specifies immediate failback to the highest priority path group that contains active paths. | A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention. | A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover. | A numeric value greater than zero specifies deferred failback, expressed in seconds. |
7 l ^, ~: I" r$ s* x# ?. v+ fprio Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include: | const: Set a priority of 1 to all paths. | emc: Generate the path priority for EMC arrays. | alua: Generate the path priority based on the SCSI-3 ALUA settings. | tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit. | ontap: Generate the path priority for NetApp arrays. | rdac: Generate the path priority for LSI/Engenio RDAC controller. | hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode. | hds: Generate the path priority for Hitachi HDS Modular storage arrays. |
3 `! A. i3 C& r; B" O0 k0 l
no_path_retry A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing. | A value of fail indicates immediate failure, without queueing. | A value of queue indicates that queueing should not stop until the path is fixed. |
' R5 U, ~, x1 ~
rr_min_io Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000. ; ~* T, L- [+ [9 q" @" y
rr_min_io_rq Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1. 4 V& x! A( L7 i+ o& Q- f9 E
rr_weight If set to priorities, then instead of sending rr_min_io requests to a path before callingpath_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by the prio function. If set touniform, all path weights are equal.
7 X0 G! |. f9 E" G; w4 Q4 S* J2 Zflush_on_last_del If set to yes, then multipath will disable queueing when the last path to a device has been deleted. " K$ K8 X" _# z
$ l( | n/ K; ] Y4 \2 f9 n/ B" G; smultipath与oracle ASM结合 https://www.cnblogs.com/jyzhao/p/7208620.html , d9 F9 T( v0 n% }2 A; a1 }; J
已有配置文件： [backcolor=rgb(245, 245, 245) !important][url=][/url]
9 F( l+ M. j1 d* H Ddefaults { polling_interval 30 failback immediate no_path_retry 5 rr_min_io 100 path_checker tur user_friendly_names yes }devnode_blacklist { devnode "sda"}devices { device { vendor "IBM" product "2145" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "1750500" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "2107900" path_grouping_policy group_by_serial } device { vendor "IBM" product "2105800" path_grouping_policy group_by_serial }}multipaths { multipath { wwid 36005076801820709e000000000000043 alias dm-vote01 }}[backcolor=rgb(245, 245, 245) !important][url=][/url]& U2 _ @, R. S% N
! b1 b; [/ e2 A2 ~5 A# N S |
发表于 2019-5-7 12:21:16