|
1. device-mapper-multipath 即multipath-tools，主要提供multipathd和multipath等工具和 multipath.conf等配置文件。这些工具通过device mapper的ioctr的接口创建和配置multipath，设备创建的多路径设备映射会在/dev /mapper中。 2. device-mapper 主要包括两大部分：内核部分和用户部分。内核部分主要由device mapper核心（dm.ko）和一些target driver（md-multipath.ko）。核心完成设备的映射，而target根据映射关系和自身特点具体处理从mappered device 下来的i/o。同时，在核心部分，提供了一个接口，用户通过ioctr可和内核部分通信，以指导内核驱动的行为，比如如何创建mappered device，这些divece的属性等。linux device mapper的用户空间部分主要包括device-mapper这个包。其中包括dmsetup工具和一些帮助创建和配置mappered device的库。这些库主要抽象，封装了与ioctr通信的接口，以便方便创建和配置mappered device。multipath-tool的程序中就需要调用这些库 3. dm-multipath.ko和dm.ko：dm.ko是device mapper驱动。它是实现multipath的基础。dm-multipath其实是dm的一个target驱动。
* g) u/ J4 b z. KAIX http://blog.163.com/herod_xiao/blog/static/871883992011819112227689/ $ A+ e- s9 D# \7 E2 e1 g+ x! N8 _
4 P$ ~7 ?; N" ~1 `, D; w1. 检查multipath模块，如果没有相关模块就说明没有安装相关软件包 lsmod | grep dm_multipath yum -y install device-mapper device-mapper-multipath
( C5 ?1 ^+ w0 h# _' |2 X2. 加载相关模块 modprobe dm-multipath modprobe dm-round-robin
1 |/ B$ r3 Y% |% `3. 生成multipath配置文件，并配置 /sbin/mpathconf --enable 配置multipath.conf文件
+ T( t8 W4 O C4 v6 s/ }- h" s4. 启动multipath服务 service multipathd start chkcofig multipathd on [backcolor=rgb(245, 245, 245) !important][url=] [/url]
& W- C$ t1 A/ w2 w5 w" f+ ?[root@tyhr ~]# multipath -llmpathc (36005076801820709e000000000000087) dm-3 IBM,2145size=300G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:1 sdc 8:32 active ready running |- 4:0:2:1 sde 8:64 active ready running |- 5:0:2:1 sdg 8:96 active ready running `- 5:0:4:1 sdi 8:128 active ready runningmpathb (36005076801820709e000000000000086) dm-2 IBM,2145size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:0 sdb 8:16 active ready running |- 4:0:2:0 sdd 8:48 active ready running |- 5:0:2:0 sdf 8:80 active ready running `- 5:0:4:0 sdh 8:112 active ready running[backcolor=rgb(245, 245, 245) !important][url=][/url] n1 a% ]7 a3 g( X5 E) ]: {2 i7 s1 b
# l, e0 p# y8 b9 ~' h$ Z
, T4 L) O# i% h v8 _3 O4 c! ~3 `- E& v- L% X( E! f5 N
如果完成配置后，不能使用multipath -ll查看到任何多路径信息，就需要手工重新扫描HBA信息 一般新增LUN映射给主机后，系统无法直接更新挂载的存储盘，需要重新扫描。 # cd /sys/class/scsi_host/host4/# echo '- - - ' > scan或echo "1" > /sys/class/fc_host/host/issue_lip //某些存储或系统没有scan文件，可以通issue_lip文件识别5 a# D: A; b4 q5 z" t$ L' S8 Y
E/ k5 ?) h; z9 g$ [, gRHEL5.x 操作系统，新增multipath后，能够识别磁盘，但是通过fdisk分区后，不能显示分区信息（/dev/mapper/dm-app 磁盘不显示/dev/mapper/dm-appp1）。partprob无报错但是没有效果。fdisk w保存后有报错： WARNING: Re-reading the partition table failed with error 22: Invalid argument. [backcolor=rgb(255, 255, 255) !important][size=1em]1 - w5 A% d" z: }" i% F/ F
2
0 @% G8 O3 j6 I3 0 X3 B. H9 \6 J7 F2 M) c
4
5 d! p2 |; d! d$ S5
( f) c6 A- Q+ |# C1 X6
C7 m) M, r s8 T' [' m7 9 n+ i! e* R* {# T+ G
8 * A) H$ o) A; S) a% y- l
9 : t2 V3 c. s9 T% m
10
4 W, [3 P/ Y0 Y1 T11
' ]2 C; }/ e% E+ x12 ( }5 n S6 v$ i& a0 Q* f
13
& R* u/ B/ L4 l9 G) O2 G14
( z5 C' M! T8 n/ E& `15 G5 @# S! m, {( A2 ?
16
" ]5 Q8 ~- V4 C3 s3 y5 M* r17
" g* v1 s# r h6 Z; Q8 L18 0 v8 t2 H; ]( c2 S9 l3 L
| Run "kpartx -a" after FDISK is completed to add all partition mappings on the ( r1 }0 }" Z$ l) }. _, F: e! a. j* H
newly-created multipath device
: \* ^0 q+ `6 U4 H: r) r# x2 N
7 t ~8 o A, l% k
* b, b5 [7 P* m- X
2 W+ ?$ D2 u/ h% d+ Y6 a[root@TYEDI1 host4]# ls -l /dev/mapper/dm-* * [0 H. p6 ?8 f' M0 z
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app 8 d4 {$ e7 @$ z; Q$ w7 }! M7 Q
brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
u, J/ I# I4 _! f0 }4 h0 i# U8 ubrw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db
+ \, ~! u M. O9 u, A) z
) k; ?, ]' x7 e5 }3 `. z2 g |[root@TYEDI1 host4]# kpartx -a /dev/mapper/dm-app ( N# D8 T% V6 ?* [- I- ]+ F
8 _$ |1 |* k/ C ]6 n
# k; n0 X2 s N
[root@TYEDI1 host4]# ls -l /dev/mapper/dm-*
) w2 `+ j, I0 t7 zbrw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app 2 O C5 p0 e& U
brw-rw---- 1 root disk 253, 3 Jan 21 23:01 /dev/mapper/dm-app1
& F0 F5 B0 U# R$ \' G8 ebrw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch * T* L/ n! z4 U/ p# N+ T
brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db
. i& {; @, ?' {7 Q+ |; \ |
) ?: t1 Q$ c& J5 e* G$ ^- D
8 T& h, A ^$ S' O1 n8 k$ }9 s
- T2 e0 y/ Q$ ^; w- [3 l% o+ R+ J' n( E: `% c( {( w
, j$ r& N: X( ?1 Y. d. e! X0 q( R% ?9 k
其他配置： 1. udev绑定权限 RHEL6.5，对于multipath的权限，手工去修改几秒后会变回root。所以需要使用udev去绑定好权限。 [backcolor=rgb(245, 245, 245) !important][url=][/url]/ p P4 w( O$ _5 Y$ L3 ?
1)搜索对应的配置文件模板：[root@jyrac1 ~]# find / -name 12-*/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules\1 j; Y, G4 ]/ o
2)根据模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：vi /etc/udev/rules.d/12-dm-permissions.rules# MULTIPATH DEVICES## Set permissions for all multipath devicesENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"# Set permissions for first two partitions created on a multipath device (and detected by kpartx)# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"3)查看多路径对应的底层dm设备：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-34)启动start_udev[root@jyrac2 rules.d]# start_udev Starting udev: [ OK ]5)查看权限：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-36)等30s后再查，权限固定：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3[backcolor=rgb(245, 245, 245) !important][url=][/url]
) f$ r" v: H$ L4 g ]$ q0 F
* D7 I2 }* F2 E( o4 H# V# Y4 S, Y6 D: f7 }6 j3 A) M* }7 ~# r: Z
- |! \ }, E' Y1 f
4 L* F/ Q8 l- p1 \2 k G* l- Z: _9 @5 W
" ^" `7 I% ?& T* ~
可以使用lsscsi查看识别的磁盘信息： [backcolor=rgb(245, 245, 245) !important][url=][/url]& G, R _: ]1 j) p, ^) [+ h2 S
[root@tyhr ~]# lsscsi [0:0:0:0] cd/dvd TSSTcorp DVD-ROM TS-L333H ID03 /dev/sr0 [2:0:0:0] cd/dvd IMM Virtual CD/DVD 0316 /dev/sr1 [3:0:0:0] disk IBM-ESXS MBF2300RC SB27 - [3:0:1:0] disk IBM-ESXS MBF2300RC SB27 - [3:1:1:0] disk LSILOGIC Logical Volume 3000 /dev/sda [4:0:0:0] disk IBM 2145 0000 /dev/sdb [4:0:0:1] disk IBM 2145 0000 /dev/sdc [4:0:2:0] disk IBM 2145 0000 /dev/sdd [4:0:2:1] disk IBM 2145 0000 /dev/sde [5:0:2:0] disk IBM 2145 0000 /dev/sdf [5:0:2:1] disk IBM 2145 0000 /dev/sdg [5:0:4:0] disk IBM 2145 0000 /dev/sdh [5:0:4:1] disk IBM 2145 0000 /dev/sdi [backcolor=rgb(245, 245, 245) !important][url=][/url]
3 f, R5 q8 T( i* t( i3 r$ |# Q4 |7 ?+ R+ |3 R) j+ }( S/ ?% S
3 Z; ^0 ?1 s" _' b% d- q8 y
; ?, ^; P, E* F5 k C, K9 x- G6 M- x+ c& M5 w G% ~" Z4 h9 V& g E
; s0 F0 e" ~: P( q0 E4 Z
1. multipath常用操作： multipath -r （修改multipath.conf配置文件之后重新加载）
6 L( g/ T2 c% f* lmultipath -ll （查看多路径状态）! g3 ~" h9 P, K! c U
multipath -v2 （格式化路径，检测路径，合并路径）1 F5 Y6 o# G4 ^1 B( l; W" ~) G
multipath -v3 （查看多路径详情blacklist、whitelist和设备wwid）
1 R, c' {5 j7 z' ~multipath -F （删除现有链路） 2. multipath文件详细介绍 接下来的工作就是要编辑/etc/multipath.conf的配置文件 multipath.conf主要包括blacklist、multipaths、devices三部份的配置0 F2 y+ e0 d8 G8 J3 o9 }
blacklist配置- y" X. ?. U8 T& Q5 ~4 g6 @
blacklist {
, v8 a6 }4 B. ~ a1 E8 U( L$ _( g devnode "^sda"
0 d* ]. w$ b; y}
) j* f1 p3 @& @, H4 P& P0 a! ^Multipaths部分配置multipaths和devices两部份的配置。
6 N1 H$ }4 o# b# }multipaths {
$ @: m3 F5 Q; z% g2 i& t0 U- U8 Z' o multipath {3 E6 m* w( r2 h; z+ h8 \4 f
wwid **************** #此值multipath -v3可以看到
5 @8 I6 g& k4 a3 O alias iscsi-dm0 #映射后的别名,可以随便取
. G, c/ Y! X# d) L path_grouping_policy multibus #路径组策略4 _* }% I5 x1 r: f, G2 `: {
path_checker tur #决定路径状态的方法
2 b/ ^. l- I9 x path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法" k3 c- y& v! K- D5 i
}4 Z5 C' V' i& G2 j7 w& U9 H
}$ }& I" f5 D* s) V7 L
Devices部分配置/ r# e* j- n% D- H }3 Y
devices {
) V( K. `. R: m, r+ {& V' a( t P# U device {0 F8 \- Q4 L0 u
vendor "iSCSI-Enterprise" #厂商名称0 [, i. Z f3 G9 j' F
product "Virtual disk" #产品型号& T, Q. m/ t% t/ I# ~5 }, |
path_grouping_policy multibus #默认的路径组策略! [. ~5 Z7 Y+ k }. U7 j
getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #获得唯一设备号使用的默认程序! n5 N1 Z2 y3 O! `) E/ W z& H
prio_callout "/sbin/acs_prio_alua %d" #获取有限级数值使用的默认程序
1 B* ]7 I6 h: z+ p, u path_checker readsector0 #决定路径状态的方法* O5 M$ \2 t& G8 r& }
path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法, K( X# `. J) h7 S& f
failback immediate #故障恢复的模式" _* a, q8 ?9 P# h/ @
no_path_retry queue #在disable queue之前系统尝试使用失效路径的次数的数值- v$ x3 V; B4 l
rr_min_io 100 #在当前的用户组中，在切换到另外一条路径之前的IO请求的数目
; D h8 B; c0 \& @+ G6 w }
+ B: L/ U* d: a3 T4 y. B}; f6 R' R5 y( ], Z" C" K
相关参数的标准文档的介绍： Attribute Description
) |! |' _% C% Z' z( P7 zwwid Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this section of themultipath.conf file. 7 Y3 ~( H& l* d- q
alias Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are using user_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names. ) I7 a! H9 C+ E' S, L
path_grouping_policy Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include:
8 h# r3 @+ ?* e" ~) c+ r5 h: Lfailover = 1 path per priority group 1 m2 t! |5 ^# i! y3 u; q/ T: J
multibus = all valid paths in 1 priority group
8 @7 Z: P1 B( E' x7 k" l" C3 @group_by_serial = 1 priority group per detected serial number % d3 v' X- v8 i$ L! t
group_by_prio = 1 priority group per path priority value ! S6 M2 n' h0 `1 M
group_by_node_name = 1 priority group per target node name
: F. h5 Q1 \; a; O+ [path_selector Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include: | round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each. | queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests. | service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput. |
6 } Q* e4 l6 H: O- rfailback Manages path group failback. | A value of immediate specifies immediate failback to the highest priority path group that contains active paths. | A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention. | A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover. | A numeric value greater than zero specifies deferred failback, expressed in seconds. |
/ L7 k. y5 h/ T7 d% T# P; nprio Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include: | const: Set a priority of 1 to all paths. | emc: Generate the path priority for EMC arrays. | alua: Generate the path priority based on the SCSI-3 ALUA settings. | tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit. | ontap: Generate the path priority for NetApp arrays. | rdac: Generate the path priority for LSI/Engenio RDAC controller. | hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode. | hds: Generate the path priority for Hitachi HDS Modular storage arrays. |
8 A. B2 m* `4 n7 ~0 @' `; q' d
no_path_retry A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing. | A value of fail indicates immediate failure, without queueing. | A value of queue indicates that queueing should not stop until the path is fixed. |
4 D4 g( y% M" ^3 J7 U5 krr_min_io Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000. # l4 N; ^, t: c* j
rr_min_io_rq Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1. `/ j1 }( L9 k( h0 C' @ i# a
rr_weight If set to priorities, then instead of sending rr_min_io requests to a path before callingpath_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by the prio function. If set touniform, all path weights are equal. 0 y8 b O' f1 i: s( v( q7 O, y
flush_on_last_del If set to yes, then multipath will disable queueing when the last path to a device has been deleted. 4 K: ~% B3 o" Q! V2 u8 v1 |
6 D/ P, |' a ^$ Z; Q- h4 E
multipath与oracle ASM结合 https://www.cnblogs.com/jyzhao/p/7208620.html
- U; C, X+ p T& b已有配置文件： [backcolor=rgb(245, 245, 245) !important][url=][/url]: f) O) c |- E2 v1 f
defaults { polling_interval 30 failback immediate no_path_retry 5 rr_min_io 100 path_checker tur user_friendly_names yes }devnode_blacklist { devnode "sda"}devices { device { vendor "IBM" product "2145" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "1750500" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "2107900" path_grouping_policy group_by_serial } device { vendor "IBM" product "2105800" path_grouping_policy group_by_serial }}multipaths { multipath { wwid 36005076801820709e000000000000043 alias dm-vote01 }}[backcolor=rgb(245, 245, 245) !important][url=][/url]) x; v1 ]* q6 b& K* W0 _8 L
/ u: C9 v7 O" D |
发表于 2019-5-7 12:21:17