ceph 测试磁盘 recovery的速度控制

admin

本次测试在无读写情况下进程的
几个需要用到脚本和命令
磁盘本身的大概速度
- X" [( f* i8 G# O1 y/ y

& W9 w* L2 y1 G[root@ceph-2 ~]# ceph tell osd.2 bench

0 S. |6 u, j2 X) w8 v- @6 ~" I, |{
    "bytes_written": 1073741824,
+ H, }3 E+ b6 Z( ?' K" ~- i% D. d    "blocksize": 4194304,
    "bytes_per_sec": 33975481
}
; @& Q- I6 ^# L; p1 i/ ]! C' b

得到的结果为32MB/s
9 F9 h+ [" R, L4 V获取osd上pg迁移的对象的脚本

OSD的日志需要开启到10，这里采取动态开启的方式

[root@ceph-2 ~]#
[root@ceph-2 ~]# ceph daemon osd.2 config set debug_osd 10
{
# z9 \  I" s# |; x- G1 d0 X' s: @    "success": ""
}
7 C( r# j! B: S: t# G
* o/ x! w7 {. Q日志解析的脚本

[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
6 o$ u7 K; s7 J8 [[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
) ?, h$ m( s/ K" }) k" a
4 F2 m, K9 h. }, G+ ?1 {

[root@ceph-2 ~]# cat /var/log/ceph/ceph-osd.2.log |awk '$7=="finish_recovery_op" && $8=="pg[0.15("{sub(/.*/,substr($2,1,8),$2);print $0}' |awk '{a[$1,"",$2]++}END{for (j in a )print j,a[j]|"sort -k 1"}'
[root@ceph-2 ~]# ceph pg dump |grep recovering
dumped all
% I1 Z/ L6 X) o' q# J获取当前的正在迁移的PG
6 A/ f0 b8 I7 Y& y" I1 ][root@ceph-2 ~]#  ceph pg dump|grep recovering|awk '{print $1,$2,$4,$10,$15,$16,$17,$18}'
* Q! R8 m0 h9 s! `2 ~8 Y-bash: a: command not found
dumped all
8 n. v1 r- q% r4 _8 x6 c$ T[root@ceph-2 ~]# ceph pg dump |grep recovering |awk '{print $1,$2,$4,$10,$15,$16,$17,$18}'
dumped all
+ W; w* a0 y0 r% K8 p; ^& e
( }2 K6 R% \2 V' g% d' b6 j过滤下输出结果
; ?& V$ B! o; V

3 g1 j- I4 M0 l) X: t7 L
动态监控PG的迁移
$ M% a4 Q5 M; T/ f[root@ceph-2 ~]# watch -n 1 -d "ceph pg dump|grep recovering |awk '{print ,}'"
& F, c" g8 o- @* k  m) e[root@ceph-2 ~]# watch -n 1 -d "ceph pg dump|grep recovering |awk '{print }'"
[root@ceph-2 ~]#


% K! p' |2 A# e6 }4 r

重启OSD进程

systemctl restart ceph-osd.target

磁盘的读写速度

dstat -td -D /dev/sdb -o disk.csv

sdb为需要监控的盘
测试的步骤与流程

整个测试需要保证每一次获取数据的过程都近似，这样才能最大程度减少环境对数据的影响

开始需要写入一些测试数据，这个可以用

rados -p rbd bench 3600 --no-cleanup

这个让每个PG上面大概有600-700个object，写入这个数据后就不再写入数据了