马上注册，结交更多好友，享用更多功能，让你轻松玩转社区。
您需要 登录 才可以下载或查看，没有账号？开始注册
x
1. device-mapper-multipath 即multipath-tools，主要提供multipathd和multipath等工具和 multipath.conf等配置文件。这些工具通过device mapper的ioctr的接口创建和配置multipath，设备创建的多路径设备映射会在/dev /mapper中。 2. device-mapper 主要包括两大部分：内核部分和用户部分。内核部分主要由device mapper核心（dm.ko）和一些target driver（md-multipath.ko）。核心完成设备的映射，而target根据映射关系和自身特点具体处理从mappered device 下来的i/o。同时，在核心部分，提供了一个接口，用户通过ioctr可和内核部分通信，以指导内核驱动的行为，比如如何创建mappered device，这些divece的属性等。linux device mapper的用户空间部分主要包括device-mapper这个包。其中包括dmsetup工具和一些帮助创建和配置mappered device的库。这些库主要抽象，封装了与ioctr通信的接口，以便方便创建和配置mappered device。multipath-tool的程序中就需要调用这些库 3. dm-multipath.ko和dm.ko：dm.ko是device mapper驱动。它是实现multipath的基础。dm-multipath其实是dm的一个target驱动。
, G" w( g D2 m) f- ?# eAIX http://blog.163.com/herod_xiao/blog/static/871883992011819112227689/
. y" U; U) X, | I4 e9 q! p2 J: Y; s$ `0 Q6 N, Z7 w; z: O9 E
1. 检查multipath模块，如果没有相关模块就说明没有安装相关软件包 lsmod | grep dm_multipath yum -y install device-mapper device-mapper-multipath
# B1 L# B- u! P! n' s3 e% {2. 加载相关模块 modprobe dm-multipath modprobe dm-round-robin ' {' G# e; V- J3 i
3. 生成multipath配置文件，并配置 /sbin/mpathconf --enable 配置multipath.conf文件
: Y6 x) _- x* l5 ~0 d. z! w7 D4. 启动multipath服务 service multipathd start chkcofig multipathd on [backcolor=rgb(245, 245, 245) !important][url=][/url]9 z: S0 F' g0 i8 A7 `) W
[root@tyhr ~]# multipath -llmpathc (36005076801820709e000000000000087) dm-3 IBM,2145size=300G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:1 sdc 8:32 active ready running |- 4:0:2:1 sde 8:64 active ready running |- 5:0:2:1 sdg 8:96 active ready running `- 5:0:4:1 sdi 8:128 active ready runningmpathb (36005076801820709e000000000000086) dm-2 IBM,2145size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:0 sdb 8:16 active ready running |- 4:0:2:0 sdd 8:48 active ready running |- 5:0:2:0 sdf 8:80 active ready running `- 5:0:4:0 sdh 8:112 active ready running[backcolor=rgb(245, 245, 245) !important][url=][/url], P" K Q0 [3 }! M7 o; I
5 f3 Y f8 L) d
, _0 B! H2 I2 F7 S) [) }5 T. i# Q5 M- e
如果完成配置后，不能使用multipath -ll查看到任何多路径信息，就需要手工重新扫描HBA信息 一般新增LUN映射给主机后，系统无法直接更新挂载的存储盘，需要重新扫描。 # cd /sys/class/scsi_host/host4/# echo '- - - ' > scan或echo "1" > /sys/class/fc_host/host/issue_lip //某些存储或系统没有scan文件，可以通issue_lip文件识别- F: A. P5 q X; D( z
3 X, }# a* w- Z+ {/ e' g
RHEL5.x 操作系统，新增multipath后，能够识别磁盘，但是通过fdisk分区后，不能显示分区信息（/dev/mapper/dm-app 磁盘不显示/dev/mapper/dm-appp1）。partprob无报错但是没有效果。fdisk w保存后有报错： WARNING: Re-reading the partition table failed with error 22: Invalid argument. [backcolor=rgb(255, 255, 255) !important][size=1em]1
) ~* o' c- D- x2 _8 S5 w2 ; x) o) E- I2 l, z
3
/ P' o' }( ?2 J9 |- t6 |7 N s4
( O8 B5 L. \0 I- i+ W5
+ I5 _6 ?' R' B4 F6 5 S) b! x) o3 Q& t
7 ! V0 u% [; w# q) X8 }( [( [
8 / _; d' a3 P: s5 W/ I
9 ! P2 o- ~, o* D4 T* H
10
% m5 y, F7 {$ R6 f3 ~11 # j* @5 X* s( A
12 , e$ P/ ~& g; c" _, |! q
13
8 F. y( D& P- Y) P& F4 w6 `14 & v `1 g m& p0 |
15
. h- ]( a7 X. b& L16 ( t8 g# F+ ]' J5 j4 Q% ]
17 " L0 G( Q1 t! ]; I: |1 M& G7 z
18 0 n6 z) f' n3 J: s- R, B7 l
| Run "kpartx -a" after FDISK is completed to add all partition mappings on the
& q7 H# P( Y9 t& Z7 T# X) [newly-created multipath device + t' C6 j/ Y8 o; h3 v0 V. E$ E
$ f& ^ k& K" _5 d# N/ Y, m. F% B+ u8 {. O" Q
% P8 \0 ]6 x" u1 Y4 }
[root@TYEDI1 host4]# ls -l /dev/mapper/dm-*
% C/ F& n7 q% o8 U# q/ abrw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app
# F ^! k" A/ _6 X- g) u9 lbrw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch ! n* r' E, [: V) n# D6 S
brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db
/ F! X+ }, x7 b2 r3 i6 J' T6 o' k' P2 k, q( Z" V
[root@TYEDI1 host4]# kpartx -a /dev/mapper/dm-app
% p5 S' @- P5 ^! x' n2 M+ f' K5 N- c0 O
; v2 [* b* R# Q% Q# T) a# O$ g[root@TYEDI1 host4]# ls -l /dev/mapper/dm-* ( s9 @$ s: T4 s4 D
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app 2 V% X" t) \4 }( l5 Z! H- J
brw-rw---- 1 root disk 253, 3 Jan 21 23:01 /dev/mapper/dm-app1 # K% }+ B$ G; ~2 _5 t' g' ]
brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
$ b9 m, {- s6 e! F2 m1 i) @: jbrw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db ; J( l0 x( c- p8 p
|
* C0 K0 F6 {$ a% M
4 c' }9 ]% `. R+ _# {
; @0 W" j: N: y) o& }
" s: D6 _; Q' Z( N4 g% j8 H7 v. v0 P# s5 _
0 j* ~2 R3 A- O# G/ r
其他配置： 1. udev绑定权限 RHEL6.5，对于multipath的权限，手工去修改几秒后会变回root。所以需要使用udev去绑定好权限。 [backcolor=rgb(245, 245, 245) !important][url=][/url]
/ x7 \" B' Q& l( f1)搜索对应的配置文件模板：[root@jyrac1 ~]# find / -name 12-*/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules\
; s) p* y( ]4 Z0 j' h& E: H5 U2)根据模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：vi /etc/udev/rules.d/12-dm-permissions.rules# MULTIPATH DEVICES## Set permissions for all multipath devicesENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"# Set permissions for first two partitions created on a multipath device (and detected by kpartx)# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"3)查看多路径对应的底层dm设备：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-34)启动start_udev[root@jyrac2 rules.d]# start_udev Starting udev: [ OK ]5)查看权限：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-36)等30s后再查，权限固定：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3[backcolor=rgb(245, 245, 245) !important][url=][/url]
* } X3 P! C* g% P/ s2 |% \5 ]. Y) v! O4 X8 H# y
" \2 F; m: ]5 w1 F8 d9 r9 a
3 w% u& N4 m% f. t
" C7 u3 p/ N4 s% h& E
) I; q7 V8 A7 x6 b3 f: v* Q$ f; {8 R9 u- e8 f* w
可以使用lsscsi查看识别的磁盘信息： [backcolor=rgb(245, 245, 245) !important][url=][/url]
% U8 a# S7 R" L+ R( W6 J[root@tyhr ~]# lsscsi [0:0:0:0] cd/dvd TSSTcorp DVD-ROM TS-L333H ID03 /dev/sr0 [2:0:0:0] cd/dvd IMM Virtual CD/DVD 0316 /dev/sr1 [3:0:0:0] disk IBM-ESXS MBF2300RC SB27 - [3:0:1:0] disk IBM-ESXS MBF2300RC SB27 - [3:1:1:0] disk LSILOGIC Logical Volume 3000 /dev/sda [4:0:0:0] disk IBM 2145 0000 /dev/sdb [4:0:0:1] disk IBM 2145 0000 /dev/sdc [4:0:2:0] disk IBM 2145 0000 /dev/sdd [4:0:2:1] disk IBM 2145 0000 /dev/sde [5:0:2:0] disk IBM 2145 0000 /dev/sdf [5:0:2:1] disk IBM 2145 0000 /dev/sdg [5:0:4:0] disk IBM 2145 0000 /dev/sdh [5:0:4:1] disk IBM 2145 0000 /dev/sdi [backcolor=rgb(245, 245, 245) !important][url=][/url]# q4 R% }, b6 B; E4 R
9 g7 e5 T. Y& }3 {# n8 [( W/ D' o& _, o4 x
: k! W8 p t( T8 e
, N( A9 j8 I/ h' @! }4 `
( H7 Z4 S" u# \' Y
1. multipath常用操作： multipath -r （修改multipath.conf配置文件之后重新加载）
2 d; O: d( Q- K3 P) a1 |multipath -ll （查看多路径状态）
; i0 u9 Q) q" n9 I" v* o6 y. V; hmultipath -v2 （格式化路径，检测路径，合并路径）0 |% |4 \) Z* q6 g: x
multipath -v3 （查看多路径详情blacklist、whitelist和设备wwid）" Z1 t6 K T5 Q3 | L
multipath -F （删除现有链路） 2. multipath文件详细介绍 接下来的工作就是要编辑/etc/multipath.conf的配置文件 multipath.conf主要包括blacklist、multipaths、devices三部份的配置0 u4 }1 t! q. O2 T1 ~2 ?
blacklist配置" [5 _# j1 A) F
blacklist { ( S% O; g0 k6 E8 B* Q
devnode "^sda"
2 g3 N8 [7 _7 Z# l. e} 2 l3 l0 x# ?8 p. g s3 G
Multipaths部分配置multipaths和devices两部份的配置。
2 o; M1 T j- r8 e+ `5 T( S, j- _6 y! Smultipaths {# f+ j4 a8 _7 y1 E9 T/ m! \
multipath {, P9 z/ w" g: j- Z
wwid **************** #此值multipath -v3可以看到; g) V9 H0 c8 r' L! U
alias iscsi-dm0 #映射后的别名,可以随便取
. j7 g. B6 [, @& Q6 b/ E6 ?6 K path_grouping_policy multibus #路径组策略: ?5 m" A# D# U9 L9 _: W
path_checker tur #决定路径状态的方法
; j( v1 R& O% Q& A- ` path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法* k+ S& n6 g6 X4 C
}! L! M( y8 V! d3 @
}9 \# L) F- F: b0 \ o4 b
Devices部分配置
, C+ @, F5 I, m! T( r, Xdevices {
* D, G3 P1 `+ W3 d device {& w$ k) e. O# D
vendor "iSCSI-Enterprise" #厂商名称
0 H- _ ]/ J( C6 R product "Virtual disk" #产品型号
( a) @/ ?( R! [ path_grouping_policy multibus #默认的路径组策略+ L( Z8 `9 R$ l/ E. O# @' n. P' @# U
getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #获得唯一设备号使用的默认程序8 q1 s- l7 A1 B& e6 H3 n( _
prio_callout "/sbin/acs_prio_alua %d" #获取有限级数值使用的默认程序
$ F& G# |0 U; h+ u. U$ {5 N1 p% ~ path_checker readsector0 #决定路径状态的方法
7 d' P, C& a+ T0 s$ I path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法# q' x4 R, S$ v- g2 `$ G9 ~
failback immediate #故障恢复的模式
5 X" i2 L" ^: e no_path_retry queue #在disable queue之前系统尝试使用失效路径的次数的数值 g f0 S! Y% l3 Y! u4 L& L0 H: t5 ]
rr_min_io 100 #在当前的用户组中，在切换到另外一条路径之前的IO请求的数目
/ P$ V$ q) [1 w8 b }
+ \- e4 A$ D" Q- }" @}
) K' o$ U6 A5 \$ C5 f! m% Q' \相关参数的标准文档的介绍： Attribute Description
3 a# }! l, a9 k K5 iwwid Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this section of themultipath.conf file.
) Z: F+ h, O" M d7 H( ]alias Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are using user_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names.
, J" T3 B0 D1 ]' E0 Vpath_grouping_policy Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include:
9 ^' p4 d4 q: E) j- I: Pfailover = 1 path per priority group 3 G Q; _$ j2 L1 @9 K7 ^
multibus = all valid paths in 1 priority group $ b7 I6 o" r7 \9 @" D! @4 y; j% I
group_by_serial = 1 priority group per detected serial number
( v9 K8 F( U' n! M+ x+ qgroup_by_prio = 1 priority group per path priority value + M! t' F+ D, x& t \3 b+ j
group_by_node_name = 1 priority group per target node name
/ X4 k. \; R5 G6 R( fpath_selector Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include: | round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each. | queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests. | service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput. |
7 C1 ^7 G7 H p) @* h7 V3 ]! v! ~failback Manages path group failback. | A value of immediate specifies immediate failback to the highest priority path group that contains active paths. | A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention. | A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover. | A numeric value greater than zero specifies deferred failback, expressed in seconds. |
) i4 a' P3 X& Q8 m9 j7 p, Q
prio Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include: | const: Set a priority of 1 to all paths. | emc: Generate the path priority for EMC arrays. | alua: Generate the path priority based on the SCSI-3 ALUA settings. | tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit. | ontap: Generate the path priority for NetApp arrays. | rdac: Generate the path priority for LSI/Engenio RDAC controller. | hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode. | hds: Generate the path priority for Hitachi HDS Modular storage arrays. |
' w- }. k: T: `7 j: k9 w8 x
no_path_retry A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing. | A value of fail indicates immediate failure, without queueing. | A value of queue indicates that queueing should not stop until the path is fixed. |
+ D7 }$ M/ k" H' ~5 @/ Irr_min_io Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000. & f' E9 s7 v- p; f% }6 x
rr_min_io_rq Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1. V- I1 k. J6 b! {) o" |
rr_weight If set to priorities, then instead of sending rr_min_io requests to a path before callingpath_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by the prio function. If set touniform, all path weights are equal. $ K3 s7 g3 y, B+ K8 M J5 _
flush_on_last_del If set to yes, then multipath will disable queueing when the last path to a device has been deleted.
/ u3 C' G; S5 @4 w, }3 }
7 r+ |5 [! o, X" D0 X/ B* \# xmultipath与oracle ASM结合 https://www.cnblogs.com/jyzhao/p/7208620.html
; s1 ~" _! ?1 B3 g已有配置文件： [backcolor=rgb(245, 245, 245) !important][url=][/url]/ ?" t( x1 ?" V
defaults { polling_interval 30 failback immediate no_path_retry 5 rr_min_io 100 path_checker tur user_friendly_names yes }devnode_blacklist { devnode "sda"}devices { device { vendor "IBM" product "2145" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "1750500" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "2107900" path_grouping_policy group_by_serial } device { vendor "IBM" product "2105800" path_grouping_policy group_by_serial }}multipaths { multipath { wwid 36005076801820709e000000000000043 alias dm-vote01 }}[backcolor=rgb(245, 245, 245) !important][url=][/url]6 |9 Y5 U y. T$ y3 p. H! h! J
" U: n6 |+ A2 P X1 { |
/4 
|返回首页|Archiver|小黑屋|易陆发现技术论坛
( 蜀ICP备2026014127号-1 )
窥视卡
雷达卡
发表于 2019-5-7 12:21:17
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
照妖镜