ceph 测试磁盘 recovery的速度控制

admin

本次测试在无读写情况下进程的
& z; H2 h$ g9 a1 H4 X+ e+ A几个需要用到脚本和命令
) [7 y+ U$ O8 M) w! ?! O; b磁盘本身的大概速度


! M( |) p& v6 l[root@ceph-2 ~]# ceph tell osd.2 bench
) @( I6 e- ^5 v6 P0 @) i; E
{
2 F9 {3 v1 J4 u: z: u2 [6 J& [    "bytes_written": 1073741824,
: E4 ^/ R  T9 n( }6 Z& d. l    "blocksize": 4194304,
    "bytes_per_sec": 33975481
) e  u0 Z2 c/ q! R3 S+ n}
3 T" J/ f# Q9 D, @) P8 ~

得到的结果为32MB/s
获取osd上pg迁移的对象的脚本

OSD的日志需要开启到10，这里采取动态开启的方式

[root@ceph-2 ~]#
[root@ceph-2 ~]# ceph daemon osd.2 config set debug_osd 10
" t: |) q4 |/ q1 C8 D{
    "success": ""
3 g8 |2 m7 R/ M7 Q3 A}
+ m1 k: @  u. t. Y, s
日志解析的脚本

$ j3 k6 H- b" _" E: d. h[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
/ Z: r! M) \5 f7 |& L[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
; K0 @+ `3 p1 M6 l[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
8 r' r) ^8 F, N) P[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
8 I" b# Q7 {& _! M2 Y* k' L
6 U) R# ]% d3 A) J7 X* ?! i
4 N9 J$ n( G! X
[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
" E% v  x+ M! I" P7 h, h8 R  S[root@ceph-2 ~]# ceph pg dump |grep recovering
" V4 _; u1 ]% H7 R0 O4 A" D: y% tdumped all
' {! h; i. y) z5 @# a获取当前的正在迁移的PG
[root@ceph-2 ~]#  ceph pg dump|grep recovering|awk '{print $1,$2,$4,$10,$15,$16,$17,$18}'
7 M7 R8 I3 i" D( B# G-bash: a: command not found
dumped all
: P$ F7 |) j) M, L- ^# A0 e[root@ceph-2 ~]# ceph pg dump |grep recovering |awk '{print $1,$2,$4,$10,$15,$16,$17,$18}'
dumped all
# I' p+ @- p' l
! a" B( f" X5 R  `- j过滤下输出结果

; r; K& U" E2 _3 w, ]4 J
- z7 v+ U/ W: B4 S9 {9 l, d: B
5 Q1 W& W! R! E) ?7 V动态监控PG的迁移
$ z7 B3 K4 i) `[root@ceph-2 ~]# watch -n 1 -d "ceph pg dump|grep recovering |awk '{print ,}'"
+ w. m5 W2 I8 s. }3 N8 X[root@ceph-2 ~]# watch -n 1 -d "ceph pg dump|grep recovering |awk '{print }'"
+ A6 \! S% e8 w/ |[root@ceph-2 ~]#


) c1 s$ M5 X( X( `3 D: F# v( S. X

重启OSD进程

systemctl restart ceph-osd.target

磁盘的读写速度

dstat -td -D /dev/sdb -o disk.csv

sdb为需要监控的盘
测试的步骤与流程

整个测试需要保证每一次获取数据的过程都近似，这样才能最大程度减少环境对数据的影响

开始需要写入一些测试数据，这个可以用

rados -p rbd bench 3600 --no-cleanup

这个让每个PG上面大概有600-700个object，写入这个数据后就不再写入数据了