|
1. device-mapper-multipath 即multipath-tools，主要提供multipathd和multipath等工具和 multipath.conf等配置文件。这些工具通过device mapper的ioctr的接口创建和配置multipath，设备创建的多路径设备映射会在/dev /mapper中。 2. device-mapper 主要包括两大部分：内核部分和用户部分。内核部分主要由device mapper核心（dm.ko）和一些target driver（md-multipath.ko）。核心完成设备的映射，而target根据映射关系和自身特点具体处理从mappered device 下来的i/o。同时，在核心部分，提供了一个接口，用户通过ioctr可和内核部分通信，以指导内核驱动的行为，比如如何创建mappered device，这些divece的属性等。linux device mapper的用户空间部分主要包括device-mapper这个包。其中包括dmsetup工具和一些帮助创建和配置mappered device的库。这些库主要抽象，封装了与ioctr通信的接口，以便方便创建和配置mappered device。multipath-tool的程序中就需要调用这些库 3. dm-multipath.ko和dm.ko：dm.ko是device mapper驱动。它是实现multipath的基础。dm-multipath其实是dm的一个target驱动。 ; {- G& `* ~; ]! d) M" X L# n$ |. @/ _
AIX http://blog.163.com/herod_xiao/blog/static/871883992011819112227689/
; j3 r; J6 z) `" Y U" I# }8 Q& F" K$ y9 b0 n
1. 检查multipath模块，如果没有相关模块就说明没有安装相关软件包 lsmod | grep dm_multipath yum -y install device-mapper device-mapper-multipath
6 e* v) c) S" B1 s" _2. 加载相关模块 modprobe dm-multipath modprobe dm-round-robin ' | X& K: }0 S. F7 _
3. 生成multipath配置文件，并配置 /sbin/mpathconf --enable 配置multipath.conf文件 1 ^3 ]* |' c3 }- i3 ?$ ]: l) g7 P
4. 启动multipath服务 service multipathd start chkcofig multipathd on [backcolor=rgb(245, 245, 245) !important][url=] [/url]6 G8 L5 C' F- Z) a' @% O" |
[root@tyhr ~]# multipath -llmpathc (36005076801820709e000000000000087) dm-3 IBM,2145size=300G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:1 sdc 8:32 active ready running |- 4:0:2:1 sde 8:64 active ready running |- 5:0:2:1 sdg 8:96 active ready running `- 5:0:4:1 sdi 8:128 active ready runningmpathb (36005076801820709e000000000000086) dm-2 IBM,2145size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 4:0:0:0 sdb 8:16 active ready running |- 4:0:2:0 sdd 8:48 active ready running |- 5:0:2:0 sdf 8:80 active ready running `- 5:0:4:0 sdh 8:112 active ready running[backcolor=rgb(245, 245, 245) !important][url=][/url]" n- [9 u2 [0 _" q9 v- M9 `, M6 f
: p8 |2 x4 U. r3 F6 C$ V. }- m" N& s$ D+ Z
9 T. `5 T* ]' y: S' Q: ?
如果完成配置后，不能使用multipath -ll查看到任何多路径信息，就需要手工重新扫描HBA信息 一般新增LUN映射给主机后，系统无法直接更新挂载的存储盘，需要重新扫描。 # cd /sys/class/scsi_host/host4/# echo '- - - ' > scan或echo "1" > /sys/class/fc_host/host/issue_lip //某些存储或系统没有scan文件，可以通issue_lip文件识别 n; Q# O; c) D2 G" r
: r$ @# ]# y( v6 m% f+ I8 n5 ~" n
RHEL5.x 操作系统，新增multipath后，能够识别磁盘，但是通过fdisk分区后，不能显示分区信息（/dev/mapper/dm-app 磁盘不显示/dev/mapper/dm-appp1）。partprob无报错但是没有效果。fdisk w保存后有报错： WARNING: Re-reading the partition table failed with error 22: Invalid argument. [backcolor=rgb(255, 255, 255) !important][size=1em]1 1 H9 [ a0 a0 ]
2 7 @$ w) |( t/ w3 m2 t. [
3
X3 j% a% w5 d% f4
# ^9 e/ [" Q$ w2 V5 . E' B+ S" J1 k# j- v
6
: `+ d/ f0 F C7 8 P9 v6 J- z1 [7 ^
8
* s; P3 T- [3 e, P& E' N9
4 t: M1 N- z6 Y( C' X8 Y" H10
: N: l9 W/ m9 \$ i* X" a1 [' N; D11
9 ~$ p+ I+ x5 ?: u4 H12 0 r! M; Y3 p% J" P- B. d. I
13
" F7 r5 j- M& ^14
* [2 N) Q- H+ u# B; V15
! I8 q$ o1 r) S' x16
- g t/ ]" m1 y% X+ [17
8 M& G% [1 E9 J9 l5 S6 \$ [& S18
- `. j6 l% _6 i9 R4 m1 ~ | Run "kpartx -a" after FDISK is completed to add all partition mappings on the
4 q0 T+ W, X% ]3 q3 O6 q& Nnewly-created multipath device . Y3 q" [& _ e, ]5 k
% f- Q; F! t7 p( z a' T
4 I+ y: L, ]$ D/ c* k0 S; P7 L8 _+ t( U6 _' ^% M7 {2 g! l
[root@TYEDI1 host4]# ls -l /dev/mapper/dm-* 4 j/ [) K7 S9 p6 R; i9 X
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app
7 P6 w$ h. R8 k9 Z# R7 ybrw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
$ b/ _0 E+ |% g# G2 U& i3 L1 Gbrw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db
8 v6 f0 q: ^9 _5 D
|" } B2 v8 R) ^: l% {3 E A( V) T[root@TYEDI1 host4]# kpartx -a /dev/mapper/dm-app d% {3 {- R/ ]" n" y
% {; q8 ?2 t3 \# d9 W* Y, P0 |
6 y, @% d9 } A0 T[root@TYEDI1 host4]# ls -l /dev/mapper/dm-*
' s3 ]3 K% s5 R1 V/ v w6 rbrw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app 9 t7 Q/ h; S t* y7 s* L
brw-rw---- 1 root disk 253, 3 Jan 21 23:01 /dev/mapper/dm-app1
+ [' B" M9 @5 R- c. S" n* U: s) ^brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
% E" S$ p4 E t4 x0 mbrw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db
2 S6 v4 c& e& r" j |
: m; q' _2 n% [. w1 a+ T% C$ [5 o5 f
5 ~. e5 U1 h* F Y3 W/ B3 [# t+ y7 }) Y1 N/ Y* i; v- f6 n
! z% \* H* M6 p% z& K' N. J6 E: Y' z
! U& \% z/ Y9 O" X \4 s! Y+ G4 p$ v* U: [
其他配置： 1. udev绑定权限 RHEL6.5，对于multipath的权限，手工去修改几秒后会变回root。所以需要使用udev去绑定好权限。 [backcolor=rgb(245, 245, 245) !important][url=][/url]/ D( E7 M- q; M* N# n
1)搜索对应的配置文件模板：[root@jyrac1 ~]# find / -name 12-*/usr/share/doc/device-mapper-1.02.79/12-dm-permissions.rules\, W" g9 L. D( C
2)根据模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：vi /etc/udev/rules.d/12-dm-permissions.rules# MULTIPATH DEVICES## Set permissions for all multipath devicesENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"# Set permissions for first two partitions created on a multipath device (and detected by kpartx)# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:="660"3)查看多路径对应的底层dm设备：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 16:40 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 16:40 /dev/dm-1brw-rw---- 1 root disk 253, 2 Jul 19 21:20 /dev/dm-2brw-rw---- 1 root disk 253, 3 Jul 19 21:20 /dev/dm-34)启动start_udev[root@jyrac2 rules.d]# start_udev Starting udev: [ OK ]5)查看权限：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-36)等30s后再查，权限固定：[root@jyrac2 rules.d]# ls -lh /dev/dm*brw-rw---- 1 root disk 253, 0 Jul 19 21:35 /dev/dm-0brw-rw---- 1 root disk 253, 1 Jul 19 21:35 /dev/dm-1brw-rw---- 1 grid asmadmin 253, 2 Jul 19 21:35 /dev/dm-2brw-rw---- 1 grid asmadmin 253, 3 Jul 19 21:35 /dev/dm-3[backcolor=rgb(245, 245, 245) !important][url=][/url]; A$ Z1 `8 C9 l( R4 t1 d' `/ l
- @! n ~- j! i( e- i- p/ d. H, N" p9 l; h' S6 ^) r
% ^) o7 o2 q! V D" V- ^2 p( I
7 [1 i; \: p/ p- @7 Y5 s
$ D! n+ w2 ^& _. V) O
. b' J) C: z; g- N& e 可以使用lsscsi查看识别的磁盘信息： [backcolor=rgb(245, 245, 245) !important][url=][/url]2 p6 e0 E6 W" D1 a$ _- h9 j
[root@tyhr ~]# lsscsi [0:0:0:0] cd/dvd TSSTcorp DVD-ROM TS-L333H ID03 /dev/sr0 [2:0:0:0] cd/dvd IMM Virtual CD/DVD 0316 /dev/sr1 [3:0:0:0] disk IBM-ESXS MBF2300RC SB27 - [3:0:1:0] disk IBM-ESXS MBF2300RC SB27 - [3:1:1:0] disk LSILOGIC Logical Volume 3000 /dev/sda [4:0:0:0] disk IBM 2145 0000 /dev/sdb [4:0:0:1] disk IBM 2145 0000 /dev/sdc [4:0:2:0] disk IBM 2145 0000 /dev/sdd [4:0:2:1] disk IBM 2145 0000 /dev/sde [5:0:2:0] disk IBM 2145 0000 /dev/sdf [5:0:2:1] disk IBM 2145 0000 /dev/sdg [5:0:4:0] disk IBM 2145 0000 /dev/sdh [5:0:4:1] disk IBM 2145 0000 /dev/sdi [backcolor=rgb(245, 245, 245) !important][url=][/url]
+ F+ ~! n$ D, b h0 h
/ {1 _$ {$ ~% C& d2 _. s* G' z, m- R" @/ {( \& }
# j4 Y$ u0 N- w% K) z
4 B* x1 G5 K% y! p; a- S3 ^- A, x
1. multipath常用操作： multipath -r （修改multipath.conf配置文件之后重新加载）" k6 p1 u% d! G
multipath -ll （查看多路径状态）
8 \' R( l R; { o- Z# ~% A5 K9 dmultipath -v2 （格式化路径，检测路径，合并路径）
2 b" M- q8 R# a2 k1 rmultipath -v3 （查看多路径详情blacklist、whitelist和设备wwid）: ]5 z1 V- N9 J" W9 R0 ]
multipath -F （删除现有链路） 2. multipath文件详细介绍 接下来的工作就是要编辑/etc/multipath.conf的配置文件 multipath.conf主要包括blacklist、multipaths、devices三部份的配置1 z; z1 E3 U! Y/ \; d& `2 _7 {: r% b
blacklist配置
# U6 s1 C, w- k |* {5 a) }blacklist {
9 ^) A1 j8 U: S# `. S9 ~8 n devnode "^sda"
+ W, O' M3 w! L7 ?" t6 v}
0 v1 q' ~. E! z, N3 n- h/ TMultipaths部分配置multipaths和devices两部份的配置。
+ t4 S4 k! B; g9 _multipaths {
. l. S! }" M& V1 |- K5 y# P! C/ A multipath {
8 M8 n. m; j4 }$ J2 w* s* N7 }3 _ wwid **************** #此值multipath -v3可以看到8 S1 J* J" S, }
alias iscsi-dm0 #映射后的别名,可以随便取( N5 y; } ~! F: W- C/ d ^
path_grouping_policy multibus #路径组策略$ K7 D" { g4 j [% i/ D( R
path_checker tur #决定路径状态的方法4 J: `6 _; S5 u5 v1 C8 S( k
path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法
! y& L6 d3 `9 ~; A }$ B2 \6 l& e. [! P4 a5 e5 O
}
& i8 I" a- z# f/ i. qDevices部分配置: c3 k* S- Z' z2 G* [8 V- W
devices {1 |. T5 x v6 W! m/ b' w q
device {
1 {1 S! v8 B a3 ], F ?" h( h vendor "iSCSI-Enterprise" #厂商名称9 S2 q4 p' _' H) }+ q: R
product "Virtual disk" #产品型号
2 V5 B% w$ L. Y2 r6 @ z6 J path_grouping_policy multibus #默认的路径组策略4 f$ @4 P+ R8 w
getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #获得唯一设备号使用的默认程序
; r$ m! j" t% W- v prio_callout "/sbin/acs_prio_alua %d" #获取有限级数值使用的默认程序; b0 b4 N3 n! u- ~9 S% q6 Z$ k
path_checker readsector0 #决定路径状态的方法
1 w1 D& }# A( W. H( L' Q path_selector "round-robin 0" #选择那条路径进行下一个IO操作的方法7 I: }% O4 I$ o+ W% g9 H4 S2 ]) d
failback immediate #故障恢复的模式8 X$ U6 r& Y0 ~% z. [+ ?
no_path_retry queue #在disable queue之前系统尝试使用失效路径的次数的数值" g8 l2 r- k, A& J: [: M& v
rr_min_io 100 #在当前的用户组中，在切换到另外一条路径之前的IO请求的数目
9 N c* Q9 m0 k% o) i- U }
v3 Z9 O( l- z9 w& \. v}
6 e7 W( q' `7 n2 I4 \相关参数的标准文档的介绍： Attribute Description 7 C% S! k2 P6 Z( F7 ]5 g
wwid Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this section of themultipath.conf file.
# S) E( i7 A0 Q' r. L G6 v* @alias Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are using user_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names.
; Q8 n# @) j' Bpath_grouping_policy Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include:
) i& \6 Q& v& A4 m8 `failover = 1 path per priority group
" e% Z0 m `* A2 W- z% T5 ^( m3 P: Kmultibus = all valid paths in 1 priority group ' o- ~6 ^3 e) c+ k0 r2 E' v% C
group_by_serial = 1 priority group per detected serial number 5 `2 W' m2 R, a$ w; z; Z; P* E
group_by_prio = 1 priority group per path priority value 1 }$ V2 @" U- R
group_by_node_name = 1 priority group per target node name ! Z; e8 _' \% b" ?3 c2 Y; D
path_selector Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include: | round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each. | queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests. | service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput. |
! E! X+ c6 r1 x5 lfailback Manages path group failback. | A value of immediate specifies immediate failback to the highest priority path group that contains active paths. | A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention. | A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover. | A numeric value greater than zero specifies deferred failback, expressed in seconds. |
3 i0 \+ K) c3 x0 Z2 H; p
prio Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include: | const: Set a priority of 1 to all paths. | emc: Generate the path priority for EMC arrays. | alua: Generate the path priority based on the SCSI-3 ALUA settings. | tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit. | ontap: Generate the path priority for NetApp arrays. | rdac: Generate the path priority for LSI/Engenio RDAC controller. | hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode. | hds: Generate the path priority for Hitachi HDS Modular storage arrays. |
% D# [5 @& [ ]4 @4 o9 g2 @
no_path_retry A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing. | A value of fail indicates immediate failure, without queueing. | A value of queue indicates that queueing should not stop until the path is fixed. |
2 }9 P) [; v) Z+ x) ]9 crr_min_io Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000.
* z1 R: H. {+ \: J* y- p& H4 prr_min_io_rq Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1. : @* L2 l6 z. ?
rr_weight If set to priorities, then instead of sending rr_min_io requests to a path before callingpath_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by the prio function. If set touniform, all path weights are equal. , Z! `/ N" d- Y6 w
flush_on_last_del If set to yes, then multipath will disable queueing when the last path to a device has been deleted. ) N1 B* d: n' Q0 ^
' J, q2 g' p2 N( w; n" B& d
multipath与oracle ASM结合 https://www.cnblogs.com/jyzhao/p/7208620.html
' c' N( ~! i% x' t已有配置文件： [backcolor=rgb(245, 245, 245) !important][url=][/url]/ O; e: y% @/ G
defaults { polling_interval 30 failback immediate no_path_retry 5 rr_min_io 100 path_checker tur user_friendly_names yes }devnode_blacklist { devnode "sda"}devices { device { vendor "IBM" product "2145" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "1750500" path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_alua /dev/%n" } device { vendor "IBM" product "2107900" path_grouping_policy group_by_serial } device { vendor "IBM" product "2105800" path_grouping_policy group_by_serial }}multipaths { multipath { wwid 36005076801820709e000000000000043 alias dm-vote01 }}[backcolor=rgb(245, 245, 245) !important][url=][/url]
( Y) P! }- U* c2 q7 ?+ @$ b! T0 O, B# z% N+ R( r
|
发表于 2019-5-7 12:21:17