|
1. device-mapper-multipath 即multipath-tools，主要提供multipathd和multipath等工具和 multipath.conf等配置文件。这些工具通过device mapper的ioctr的接口创建和配置multipath，设备创建的多路径设备映射会在/dev /mapper中。 2. device-mapper 主要包括两大部分：内核部分和用户部分。内核部分主要由device mapper核心（dm.ko）和一些target driver（md-multipath.ko）。核心完成设备的映射，而target根据映射关系和自身特点具体处理从mappered device 下来的i/o。同时，在核心部分，提供了一个接口，用户通过ioctr可和内核部分通信，以指导内核驱动的行为，比如如何创建mappered device，这些divece的属性等。linux device mapper的用户空间部分主要包括device-mapper这个包。其中包括dmsetup工具和一些帮助创建和配置mappered device的库。这些库主要抽象，封装了与ioctr通信的接口，以便方便创建和配置mappered device。multipath-tool的程序中就需要调用这些库 3. dm-multipath.ko和dm.ko：dm.ko是device mapper驱动。它是实现multipath的基础。dm-multipath其实是dm的一个target驱动。
/ \- T _, |, DAIX http://blog.163.com/herod_xiao/blog/static/871883992011819112227689/ 0 L7 L; |2 F" f6 Q' C3 f. t
) m! J; f, g7 h T* P
1. 检查multipath模块，如果没有相关模块就说明没有安装相关软件包 lsmod | grep dm_multipath yum -y install device-mapper device-mapper-multipath
9 O; t8 l9 N0 @ M2. 加载相关模块 modprobe dm-multipath modprobe dm-round-robin
. J! n4 [. v5 S$ s' r2 i R3. 生成multipath配置文件，并配置 /sbin/mpathconf --enable 配置multipath.conf文件
' D+ H/ K+ ]" c& _/ L4. 启动multipath服务 service multipathd start chkcofig multipathd on [backcolor=rgb(245, 245, 245) !important][url=] [/url]; E K4 V5 t2 z' `9 `$ h& X
[root@tyhr ~]# multipath -llmpathc (36005076801820709e000000000000087) dm-3 IBM,2145size=300G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:1 sdc 8:32 active ready running |- 4:0:2:1 sde 8:64 active ready running |- 5:0:2:1 sdg 8:96 active ready running `- 5:0:4:1 sdi 8:128 active ready runningmpathb (36005076801820709e000000000000086) dm-2 IBM,2145size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:0 sdb 8:16 active ready running |- 4:0:2:0 sdd 8:48 active ready running |- 5:0:2:0 sdf 8:80 active ready running `- 5:0:4:0 sdh 8:112 active ready running[backcolor=rgb(245, 245, 245) !important][url=][/url]; K; C# |2 Y! Q" j) d0 n; j
/ N5 C( ^( |7 a1 m1 ~ V& [2 F4 E. j) [' K) G
1 z) V3 i- Y6 c# C$ c如果完成配置后，不能使用multipath -ll查看到任何多路径信息，就需要手工重新扫描HBA信息 一般新增LUN映射给主机后，系统无法直接更新挂载的存储盘，需要重新扫描。 # cd /sys/class/scsi_host/host4/# echo '- - - ' > scan或echo "1" > /sys/class/fc_host/host/issue_lip //某些存储或系统没有scan文件，可以通issue_lip文件识别
$ d' Z) m! s( C" a% X
4 t8 {3 f0 a+ J( a2 nRHEL5.x 操作系统，新增multipath后，能够识别磁盘，但是通过fdisk分区后，不能显示分区信息（/dev/mapper/dm-app 磁盘不显示/dev/mapper/dm-appp1）。partprob无报错但是没有效果。fdisk w保存后有报错： WARNING: Re-reading the partition table failed with error 22: Invalid argument. [backcolor=rgb(255, 255, 255) !important][size=1em]1
. ?+ S- R8 H* j8 n, g8 Z2
# K9 s7 R7 Q5 X# }6 g3 * G& ?& E9 s! ^6 H: D* j8 ~
4 + z8 U& p( N$ t% ]
5 ( n: K" u3 C4 H8 v* Z9 _
6 1 I( u1 y, H* T! Y; O
7
! S1 Q; S; `3 b9 O. T0 b8
% Z. G! F! u; P0 H/ A" d4 K9
3 h/ b: g: m g5 m* i! v2 C9 ]( e z10
3 y" q. s$ i2 h# V( D( }+ }11 9 i: i$ A1 E2 s# q
12
/ b2 j9 Q0 f& z! |0 A. X* B1 V3 C* k13
& S- F w! n! G% R' m8 f, P14
) M& }8 s2 p4 | o$ m1 f15 " P9 s- ^' y) \; H
16
, Y4 t' w- z& |17
! z$ N' E3 O: w2 Z) o: C18
6 W$ r- ~6 P; |0 @' D1 I | Run "kpartx -a" after FDISK is completed to add all partition mappings on the . E. g- n! F8 x) Z
newly-created multipath device + U4 L) u* n1 D4 F/ J+ w( G
+ s; |' z' g* V& b7 x P5 e$ {
2 Q3 D4 _8 _# U5 D& f; u
# e$ P; }4 ]2 I" t q4 v[root@TYEDI1 host4]# ls -l /dev/mapper/dm-*
+ m1 J9 S7 B& s* Ubrw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app % ]2 _/ J4 f2 n% u/ K, G
brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch & l1 \1 }: x0 r( D& S/ I! G) `* j
brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db , O; t$ u# i" o' X/ n
& K: i. x& X2 \[root@TYEDI1 host4]# kpartx -a /dev/mapper/dm-app $ T( R$ n7 G+ d
m& M9 v' x0 d0 I0 R
3 B! m3 i; }6 ~' y9 r' r6 b9 a[root@TYEDI1 host4]# ls -l /dev/mapper/dm-* 7 {5 E1 B: j: Q, n' G. U
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app
6 q& u% E- H! B$ Vbrw-rw---- 1 root disk 253, 3 Jan 21 23:01 /dev/mapper/dm-app1
* a7 R0 H+ I' t! j1 Zbrw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
+ Q( l4 p% s F% b7 e6 Y# _* nbrw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db % e$ L b, K7 J+ r
|
% B7 E. g( F4 ~) w6 X- i
9 K( D/ b& V4 C2 Q$ B1 T: A1 C' a$ b0 p: e3 r
, K9 \, u% ^$ w' Y7 ~4 p. V8 g0 Z5 H9 U
; X( u4 t1 R, H, E
其他配置： 1. udev绑定权限 RHEL6.5，对于multipath的权限，手工去修改几秒后会变回root。所以需要使用udev去绑定好权限。 [backcolor=rgb(245, 245, 245) !important][url=][/url]3 X3 h" _2 y8 ?5 ?+ m( z
1)搜索对应的配置文件模板：[root@jyrac1 ~]# find / -name 12-*/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules\8 a+ M3 S1 |$ _7 c, L7 h2 t
2)根据模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：vi /etc/udev/rules.d/12-dm-permissions.rules# MULTIPATH DEVICES## Set permissions for all multipath devicesENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"# Set permissions for first two partitions created on a multipath device (and detected by kpartx)# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"3)查看多路径对应的底层dm设备：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-34)启动start_udev[root@jyrac2 rules.d]# start_udev Starting udev: [ OK ]5)查看权限：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-36)等30s后再查，权限固定：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3[backcolor=rgb(245, 245, 245) !important][url=][/url]- g4 D5 @7 \0 ^" T9 ?3 c7 I/ ]
! ^ h, |8 K1 Y3 |, _7 D$ b8 b7 J/ ]- \( `# c1 F
" B8 A5 `9 G6 u: R
2 s) p. M- ~) w# t0 o5 f3 O+ w; G! H& s; ~9 }
* P! L6 l p( o) Q5 x3 p I; Y 可以使用lsscsi查看识别的磁盘信息： [backcolor=rgb(245, 245, 245) !important][url=][/url]: A" N+ S. ?% v
[root@tyhr ~]# lsscsi [0:0:0:0] cd/dvd TSSTcorp DVD-ROM TS-L333H ID03 /dev/sr0 [2:0:0:0] cd/dvd IMM Virtual CD/DVD 0316 /dev/sr1 [3:0:0:0] disk IBM-ESXS MBF2300RC SB27 - [3:0:1:0] disk IBM-ESXS MBF2300RC SB27 - [3:1:1:0] disk LSILOGIC Logical Volume 3000 /dev/sda [4:0:0:0] disk IBM 2145 0000 /dev/sdb [4:0:0:1] disk IBM 2145 0000 /dev/sdc [4:0:2:0] disk IBM 2145 0000 /dev/sdd [4:0:2:1] disk IBM 2145 0000 /dev/sde [5:0:2:0] disk IBM 2145 0000 /dev/sdf [5:0:2:1] disk IBM 2145 0000 /dev/sdg [5:0:4:0] disk IBM 2145 0000 /dev/sdh [5:0:4:1] disk IBM 2145 0000 /dev/sdi [backcolor=rgb(245, 245, 245) !important][url=][/url]% z0 R# N" |3 P1 `+ f; o
! J. r: U" z. {" p' R
4 }" ?9 g3 K! b+ M6 g* ]! Z7 R4 p. K/ Y6 J, ^0 \
: A8 J; a' ?# L
1 O4 ] A$ X/ U$ \1 Z1. multipath常用操作： multipath -r （修改multipath.conf配置文件之后重新加载）
o+ F9 F* P. v' J( L3 K4 ]multipath -ll （查看多路径状态）" d6 e$ o9 ~3 S8 k5 @
multipath -v2 （格式化路径，检测路径，合并路径）
) f7 h& G% t8 G* O) Kmultipath -v3 （查看多路径详情blacklist、whitelist和设备wwid）
# Z z( Z. j3 B% k D. Smultipath -F （删除现有链路） 2. multipath文件详细介绍 接下来的工作就是要编辑/etc/multipath.conf的配置文件 multipath.conf主要包括blacklist、multipaths、devices三部份的配置
2 P/ l, Z+ h2 ?; r: n6 c$ |2 Oblacklist配置
E* K1 M+ s, Y# K, ~blacklist {
" K+ Y4 k* |* y3 I, b6 `- e8 o devnode "^sda"
" g; N1 @2 |. }9 L}
4 K7 n4 q9 c, I" g) C- `) oMultipaths部分配置multipaths和devices两部份的配置。
1 E0 j+ r8 a: g: b& Q; wmultipaths {1 s/ C; a4 C* w: h% Y g' I
multipath {, [! r- S+ V; ]' H9 n
wwid **************** #此值multipath -v3可以看到
, E; ~' }5 S( W; A# Y5 c9 O. C alias iscsi-dm0 #映射后的别名,可以随便取
, A* U) N0 E' W3 {; G/ F/ E1 e path_grouping_policy multibus #路径组策略' \) c& w2 j1 v" c& B, F8 H
path_checker tur #决定路径状态的方法' d$ o$ h. Q$ b1 ]* n' U& w- m! |
path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
4 h2 d# l3 X( Z6 q0 _; u9 U }
, m% y" L$ ]* Z h% O% ?8 B}
+ H+ H) `/ b2 A4 g# JDevices部分配置2 Y; b; p: y) r$ | H4 C6 D- ^
devices {
. a1 G! O8 G. U& [9 N) d- b device {" q* Z7 i0 X( F' ?3 u. S
vendor "iSCSI-Enterprise" #厂商名称
; ^5 v* _, Q3 @* `5 F; u/ A, ] product "Virtual disk" #产品型号6 I4 N" N. p. X# J1 {; `
path_grouping_policy multibus #默认的路径组策略
( C$ w! T* R& O, W0 M" J getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #获得唯一设备号使用的默认程序
! r0 B3 F4 g% p; u" w7 D& { prio_callout "/sbin/acs_prio_alua %d" #获取有限级数值使用的默认程序' ^; y3 I8 w/ S& _$ F1 u4 X7 t
path_checker readsector0 #决定路径状态的方法
* h' e9 f% b' A! o) Q% {# W path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
X3 h; f5 {, ~' Q( g failback immediate #故障恢复的模式1 P& G4 f) C4 L% y; ~5 @* s! {
no_path_retry queue #在disable queue之前系统尝试使用失效路径的次数的数值1 I8 _, j/ W9 {% u
rr_min_io 100 #在当前的用户组中，在切换到另外一条路径之前的IO请求的数目2 j6 f+ v' }9 |! ?" v9 a# l
}9 [- t0 l6 [; k5 O
}
) D; l+ C9 ?; {. s* F& H. @; P/ F相关参数的标准文档的介绍： Attribute Description
, N& P' f9 E1 twwid Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this section of themultipath.conf file. * x% M0 }8 `4 Y) z1 b5 `
alias Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are using user_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names. ) V/ u# W3 I) t+ }5 t! W. |
path_grouping_policy Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include: 1 i- r0 j3 S# ]$ u
failover = 1 path per priority group
8 j8 C( S& r4 M% \* ~) |5 Lmultibus = all valid paths in 1 priority group
V% v4 j4 V9 q& l! mgroup_by_serial = 1 priority group per detected serial number - G) o; W) r( n' ?( f7 s/ W
group_by_prio = 1 priority group per path priority value
2 I) Q. c" n" Sgroup_by_node_name = 1 priority group per target node name " x1 k4 n' J4 k1 l$ s' O# E
path_selector Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include: | round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each. | queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests. | service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput. |
: y" r3 ], V1 Y4 b( a4 B& L
failback Manages path group failback. | A value of immediate specifies immediate failback to the highest priority path group that contains active paths. | A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention. | A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover. | A numeric value greater than zero specifies deferred failback, expressed in seconds. |
; r" O; ^2 ~% Z7 U0 P
prio Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include: | const: Set a priority of 1 to all paths. | emc: Generate the path priority for EMC arrays. | alua: Generate the path priority based on the SCSI-3 ALUA settings. | tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit. | ontap: Generate the path priority for NetApp arrays. | rdac: Generate the path priority for LSI/Engenio RDAC controller. | hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode. | hds: Generate the path priority for Hitachi HDS Modular storage arrays. |
- k( D3 b' D& ono_path_retry A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing. | A value of fail indicates immediate failure, without queueing. | A value of queue indicates that queueing should not stop until the path is fixed. |
& r, F& n2 _0 O4 x0 s3 Z/ P
rr_min_io Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000.
& T. J+ D8 U$ F/ W/ l0 ^; Z$ v$ Q! Lrr_min_io_rq Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1. ( H9 { k, Y ^6 L* ?+ Q0 i7 Q7 d
rr_weight If set to priorities, then instead of sending rr_min_io requests to a path before callingpath_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by the prio function. If set touniform, all path weights are equal.
( s8 ]/ Z E* ?5 e+ Lflush_on_last_del If set to yes, then multipath will disable queueing when the last path to a device has been deleted. : h/ b) d) f# }/ T" b
2 B# u+ @3 U& c, bmultipath与oracle ASM结合 https://www.cnblogs.com/jyzhao/p/7208620.html
% x& Q9 x1 Y" V# F! g9 t已有配置文件： [backcolor=rgb(245, 245, 245) !important][url=][/url]
8 J2 |1 {+ `% Q7 N/ Ddefaults { polling_interval 30 failback immediate no_path_retry 5 rr_min_io 100 path_checker tur user_friendly_names yes }devnode_blacklist { devnode "sda"}devices { device { vendor "IBM" product "2145" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "1750500" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "2107900" path_grouping_policy group_by_serial } device { vendor "IBM" product "2105800" path_grouping_policy group_by_serial }}multipaths { multipath { wwid 36005076801820709e000000000000043 alias dm-vote01 }}[backcolor=rgb(245, 245, 245) !important][url=][/url]2 {5 z* W0 N1 x! i3 {. C( H5 I" }8 Q
( l5 \# }9 ^& I- | |
发表于 2019-5-7 12:21:16