找回密码
 注册
查看: 589|回复: 3

ceph存储backfills值和recover速度设置

[复制链接]

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
发表于 2022-10-19 17:00:09 | 显示全部楼层 |阅读模式
ceph daemon osd.2 config show|grep osd_max_backfills5 r/ Y, `( c# n9 ?& T+ k  W3 N# n
ceph daemon osd.2 config show|grep osd_recovery_max_active0 p8 m( c; d9 |4 A7 I

% A, e3 r5 J6 ]9 s% dceph daemon osd.2 config show|grep osd_recovery_sleep
" t8 Y) m1 _* u: g8 x, q: D" K% y  o# |' k0 Q- j, J5 f+ G

7 u; L" ^8 V% R, m) `ceph tell osd.* injectargs "--osd_recovery_sleep_hdd=0"2 N0 O& G5 A4 m4 b* {- K
ceph tell osd.* injectargs "--osd_max_backfills=10"! ^7 R( p4 x8 }2 I: m  S
ceph tell osd.* injectargs "--osd_recovery_max_active=15"
" A/ g1 H: T- T( p, q$ D9 f+ U& Z, w8 i+ @2 l1 @

4 Z% N: u. Y6 @4 L% n$ {# k同步速度就会上去' m& R9 C6 G6 W. e
, R9 l4 o+ @+ \, ^! a; W1 V( U# }0 B" L

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
 楼主| 发表于 2022-10-20 08:33:25 | 显示全部楼层
官方建议的PG使用数量:% Z3 _* s; b% _* F8 K/ z

7 ~: {% q* A  _( _/ z* J: ~( F! V集群中小于5个OSD,则设置PG的数量为128。" t8 X. `" X" R; ~
集群有5-10个OSD时,设置PG的数量为512。% `  e2 Z; N  h: \  a
集群中有10-50个OSD时,设置PG的数量为1024.) D. K& o$ a$ B- b5 W. a
当集群中超过50个OSD时,需要权衡PG的数量,有一种公式:
- [6 L2 f+ Z  u- Z0 d1 D3 Q% }4 I. K' d4 K
​ (osd_number * 100)
8 M) ~2 n$ f; \/ y
8 d) r5 Y8 ?# T- Kpg_num = ------------------------------8 a5 M  n" Y  N* b6 z. k2 c1 a+ d
) ^8 D. {% h" I5 N# `: \3 {
​ pool size3 \8 A9 P, D+ Y# j- s6 t; h

2 J' l  K, D; J# j* h% V例如集群有90个OSD,Pool设置的副本数为3:(90*100)÷3=3000
7 Q0 Y2 t/ k) e- Z
# ]3 s$ c) ]; V% Z' JPGP的数量要和PG的数量保持一致。
5 q/ D$ }' k2 S' K
. \4 i) ?3 }# }7 z% X注意这个PG数量是集群所能承担的总PG数量,每个资源池分配多个PG,还需要通过总的PG数乘以资源池占整个集群数据比例,最终拿到一个资源池应该分配多少个PG的数量。

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
 楼主| 发表于 2022-11-8 10:35:12 | 显示全部楼层
控制数据恢复及回填速度
1 F+ X- {) z3 @8 H! N$ b! f- E3 j+ l# 将用于数据恢复操作的优先级提到最高级别* K9 i9 x8 D0 c4 M
ceph tell osd.* injectargs "--osd_recovery_op_priority=63"/ a: n+ r" P. O  E

/ h$ s" B4 d/ q2 Z3 u# 将用于client I/O操作的优先级降到35 X( b1 u7 F3 s* x+ M
ceph tell osd.* injectargs "--osd_client_op_priority=3", X- k4 ?% a1 Y; d% r

6 ^% x, A4 P! g. n7 Z; a# 将每个osd上用于回填并发操作数由默认的1调整到50
" E4 [9 d- D* }ceph tell osd.* injectargs "--osd_max_backfills=50"
# g8 m& X. s' \  U/ q5 t, |6 Y5 g% @2 W6 C  k4 ]. x
# 将每个osd上用于恢复的并发操作数由默认的3调整到50
! {7 v" S& M+ ?& hceph tell osd.* injectargs "--osd_recovery_max_active=50"
* P/ b! t/ {; J3 T5 y6 z* y6 C! W3 y; ^6 z
# 将每个osd上用于执行恢复的线程数由默认的1调整到10
7 S, j/ q: T& W) i$ ~+ x6 Lceph tell osd.* injectargs "--osd_recovery_threads=10"

0

主题

0

回帖

9

积分

管理员

积分
9
QQ
 楼主| 发表于 2022-11-8 10:36:32 | 显示全部楼层
[root@hostceph1 ~]# ceph daemon osd.2 config show|grep osd_max_backfills' X+ k' T2 ^- E8 Q# c
    "osd_max_backfills": "1",
, r- i0 M1 w+ @8 W) K3 b[root@hostceph1 ~]# ceph daemon osd.2 config show|grep osd_recovery_max_active
$ M/ o# |- |# V    "osd_recovery_max_active": "3",
! h" O6 E5 g3 v" f& |$ O[root@hostceph1 ~]# ceph daemon osd.2 config show|grep osd_recovery_sleep
1 t' k+ [) T/ [" F! w9 b0 s& I0 k    "osd_recovery_sleep": "0.000000",% V) U4 `& ]; {. ]) w
    "osd_recovery_sleep_hdd": "0.100000",1 o# S3 Q6 E+ s( s. i9 ?
    "osd_recovery_sleep_hybrid": "0.025000",
2 T# N$ v) ?) J    "osd_recovery_sleep_ssd": "0.000000",7 z' t/ X& w0 F0 F& Z- d# l1 Q* M
[root@hostceph1 ~]# ceph daemon osd.2 config show|grep max_backfills: T# e$ @2 o, j  t6 j3 b5 \
    "osd_max_backfills": "1",
3 q! {7 a3 ~5 e2 w6 Z[root@hostceph1 ~]# ceph daemon osd.2 config show|grep recovery_max_active
" _* n9 G6 j3 j% i6 P- @5 \    "osd_recovery_max_active": "3",
& @* E. V3 p7 ?$ E1 u) a$ y[root@hostceph1 ~]# ceph daemon osd.2 config show|grep recovery_threads& }* i* G4 ]0 I
[root@hostceph1 ~]# ceph tell osd.* injectargs "--osd_recovery_sleep_hdd=0"2 ?  V+ x) s8 k5 M2 p
osd.0: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
/ o4 o6 c8 Y8 y3 B0 h8 |osd.1: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) ! q! X% l) z2 O  U2 w# d1 U" K
osd.2: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
  x! q7 h9 W- b5 ?- J, g: fosd.3: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
/ F. u9 [. o6 L& {: K$ Y* dosd.4: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) 8 @; g/ a1 {: H- ?) n
osd.5: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
7 k) ~8 L, i# o' U! L6 `osd.6: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) * t  c9 V, ]; n- k* p
osd.7: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) 8 L$ S" k3 g8 x4 v) P
osd.8: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) * `: `' @- d( [: d, l
osd.9: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) 7 W# D7 K9 }4 l! ?
osd.10: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) , f5 \, q& j" `( o; @6 W2 _! c, o3 O- U
osd.11: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) ' Y' S& S6 H# Q* g+ q) |! N
osd.12: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) 8 J5 q6 `) |' h
osd.13: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) $ X9 I0 @/ M# Y0 G# r  |
osd.14: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
& @7 E0 {, ^$ x9 @% M3 p* b$ j, H$ M/ d1 Qosd.15: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
3 E0 H1 M  i- Iosd.16: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
& f7 v, ]& \8 _- c& x* vosd.17: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) 3 M3 ?7 H4 X- D% u0 X0 d
osd.18: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
% I6 m- Z6 A' ^/ {, hosd.19: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
+ ~% j3 d7 M  F% U! \, _osd.20: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
0 Y$ f* ^. n5 G# v$ U4 Posd.21: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) / t. y2 l, a' M! J8 i
osd.22: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
7 w- o5 C. U& I/ v8 X% Cosd.23: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
: x/ Q6 S) v4 x7 W9 }8 T5 w$ y; aosd.24: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
4 [* e' N; G. \# _) h5 j# @$ ~osd.25: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) 5 ?: G/ {! c& L/ Y, b  Q
osd.26: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
4 c$ j( a/ J; d7 ~1 c3 Gosd.27: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) ! Y5 z. U* _/ ?6 O/ G5 E- }  a
osd.28: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
9 q; R. O3 A  G8 Nosd.29: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) ( w/ u# ]9 I+ k$ C4 O. o" N1 B  ]
osd.30: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) ) e$ @3 @0 C* o" N+ V
osd.31: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart) & j' a" R% t5 U; k* v7 N% Q7 l
osd.32: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
  _+ c3 |. b5 a/ R8 m( W8 j# Fosd.33: osd_recovery_sleep_hdd = '0.000000' (not observed, change may require restart)
, x& @6 k3 x9 a5 B/ H/ v[root@hostceph1 ~]# ceph tell osd.* injectargs "--osd_max_backfills=10"
! E# M, N- y: S; I) ^osd.0: osd_max_backfills = '10'
8 Z. v3 x8 L7 m, }2 z5 f! hosd.1: osd_max_backfills = '10'
0 o" `- h: z- x: F1 l6 r4 P" cosd.2: osd_max_backfills = '10' 3 q5 E- r6 U0 b; l
osd.3: osd_max_backfills = '10'   a( v! A0 `0 d+ u5 `
osd.4: osd_max_backfills = '10'
! Q' X# K$ C5 y* ~+ josd.5: osd_max_backfills = '10'
3 `( d6 r3 j8 O2 c0 u6 G9 nosd.6: osd_max_backfills = '10'
/ b' c* L3 S& s2 X+ u; A( Zosd.7: osd_max_backfills = '10' 5 v/ u+ k" g) C
osd.8: osd_max_backfills = '10' 7 @. [4 w0 M& @5 {+ T9 e! s2 J
osd.9: osd_max_backfills = '10' ! p  z4 J& O% [
osd.10: osd_max_backfills = '10' * `! ?8 z4 _+ x% \8 ^8 `! m
osd.11: osd_max_backfills = '10' 2 Y0 R+ S" i. D. t" ?8 r
osd.12: osd_max_backfills = '10' $ U8 `1 W. u( \
osd.13: osd_max_backfills = '10' 0 r4 D6 v; F5 N
osd.14: osd_max_backfills = '10' # x7 z1 j9 o8 j$ `
osd.15: osd_max_backfills = '10' ( \) H- V6 D% B$ R
osd.16: osd_max_backfills = '10'
* }) z& s. ~# \osd.17: osd_max_backfills = '10' 6 g3 i; }, f, K5 B, a
osd.18: osd_max_backfills = '10' % C! A+ Q  u" V5 p
osd.19: osd_max_backfills = '10' % B7 C" J- I; g$ @0 ~4 d
osd.20: osd_max_backfills = '10' % E; X1 I2 R3 V( Q4 ]
osd.21: osd_max_backfills = '10' , k8 F' S: ~. h4 O$ g
osd.22: osd_max_backfills = '10' # C, }# J9 t3 Z; w' ^' y2 x
osd.23: osd_max_backfills = '10' & P6 Y- l7 D/ g* s  g
osd.24: osd_max_backfills = '10' ! S3 ^* V, e! X8 ?/ Q, r
osd.25: osd_max_backfills = '10' $ V- {* o) s# [9 \7 C2 p  C  @
osd.26: osd_max_backfills = '10'
; i" N, y6 S2 h) m) U) g3 O! v7 `osd.27: osd_max_backfills = '10' 5 |9 v- [9 J9 H2 _% J. X. B
osd.28: osd_max_backfills = '10'
. ^: m( P) T( bosd.29: osd_max_backfills = '10'
- B- u/ [, A# w9 l0 Nosd.30: osd_max_backfills = '10' 2 z7 V+ C  G+ Q  b  N2 W( u
osd.31: osd_max_backfills = '10' $ Q2 i) F  a7 d# `" t. ?& O- P
osd.32: osd_max_backfills = '10'
8 a2 }( C1 `: p; E0 S1 {osd.33: osd_max_backfills = '10' 1 k% h0 c0 B: z* i
[root@hostceph1 ~]# ceph tell osd.* injectargs "--osd_recovery_max_active=10"
1 u. s1 _2 @0 o: c$ uosd.0: osd_recovery_max_active = '10' (not observed, change may require restart)
: N+ d& v: o( c/ [osd.1: osd_recovery_max_active = '10' (not observed, change may require restart)
2 S) N% [( b" eosd.2: osd_recovery_max_active = '10' (not observed, change may require restart)
# ]8 o4 v4 A2 Q5 Y" [' R6 iosd.3: osd_recovery_max_active = '10' (not observed, change may require restart)
4 J+ u5 `0 u2 ~. q) Xosd.4: osd_recovery_max_active = '10' (not observed, change may require restart) , e, j0 R) b/ ~+ Y
osd.5: osd_recovery_max_active = '10' (not observed, change may require restart) 3 \# w7 R7 `, u! q5 J/ s: J+ x7 a
osd.6: osd_recovery_max_active = '10' (not observed, change may require restart) 2 L& B1 v! X9 H' b9 @) k" X- G
osd.7: osd_recovery_max_active = '10' (not observed, change may require restart) 3 d2 x$ [2 D# N8 i3 u: w7 Q* g- t
osd.8: osd_recovery_max_active = '10' (not observed, change may require restart) 4 o1 D4 \" K  s+ O! a& w
osd.9: osd_recovery_max_active = '10' (not observed, change may require restart) 7 U) V! @: o# J! U1 d7 }- Q
osd.10: osd_recovery_max_active = '10' (not observed, change may require restart)
" e* {5 Q+ y- }7 ?' b8 Hosd.11: osd_recovery_max_active = '10' (not observed, change may require restart)
' Z2 X) X8 I8 M% ^$ V4 yosd.12: osd_recovery_max_active = '10' (not observed, change may require restart) 8 v3 }3 V0 f- N7 h: z* m2 y. Y
osd.13: osd_recovery_max_active = '10' (not observed, change may require restart)
" e& y5 _: S: yosd.14: osd_recovery_max_active = '10' (not observed, change may require restart)
) H% e. J5 j: F- \7 m* Mosd.15: osd_recovery_max_active = '10' (not observed, change may require restart)
1 A% l6 x) k$ i" f) Josd.16: osd_recovery_max_active = '10' (not observed, change may require restart) + @2 N) r" {* t8 z9 W
osd.17: osd_recovery_max_active = '10' (not observed, change may require restart) 5 c$ T$ y* j/ w! G: o# b1 U
osd.18: osd_recovery_max_active = '10' (not observed, change may require restart)
: X, P& q/ z( T* x; L6 qosd.19: osd_recovery_max_active = '10' (not observed, change may require restart) 8 d7 X  n$ f9 s5 N
osd.20: osd_recovery_max_active = '10' (not observed, change may require restart)
: _2 j7 b" B. y3 j+ {osd.21: osd_recovery_max_active = '10' (not observed, change may require restart) : k3 o( |! J5 _' ?
osd.22: osd_recovery_max_active = '10' (not observed, change may require restart) + |; W* y5 V: F6 }
osd.23: osd_recovery_max_active = '10' (not observed, change may require restart)
- z- e  p% V; U& S. u4 rosd.24: osd_recovery_max_active = '10' (not observed, change may require restart) 4 G/ h1 ?' T3 M
osd.25: osd_recovery_max_active = '10' (not observed, change may require restart) 8 w3 g* a# ~5 T9 b2 r& I. C
osd.26: osd_recovery_max_active = '10' (not observed, change may require restart) 7 V% w* x$ P/ d
osd.27: osd_recovery_max_active = '10' (not observed, change may require restart)
- a- j2 Z& Y! I- K6 bosd.28: osd_recovery_max_active = '10' (not observed, change may require restart)
# y+ r# S) m8 a+ |osd.29: osd_recovery_max_active = '10' (not observed, change may require restart) 6 X# P5 h- _! I& ~# M' i
osd.30: osd_recovery_max_active = '10' (not observed, change may require restart) + F& b: a& c% U2 `7 S( q1 ~
osd.31: osd_recovery_max_active = '10' (not observed, change may require restart)
; ?1 k5 `0 k2 B+ G% I" Z( Y* wosd.32: osd_recovery_max_active = '10' (not observed, change may require restart) 7 x' v8 ?7 ]4 K3 P( k
osd.33: osd_recovery_max_active = '10' (not observed, change may require restart)
, ^3 C* _9 Y; B/ }5 ^0 H[root@hostceph1 ~]# ceph -s
/ e/ w4 e% `4 S+ _7 U$ y3 z2 p) z  cluster:- L3 y  [1 k7 y2 |: Z0 G
    id:     29046cc0-0682-496b-98b1-912e59964282
& a- c  w# e; I    health: HEALTH_WARN
, E/ t& z% ]8 U' z: L8 P8 Y            Reduced data availability: 24 pgs inactive9 U. |1 R) |/ l8 o. \& c
            24 pgs not deep-scrubbed in time
) q* {7 o  l& P& h4 g% P  L            24 pgs not scrubbed in time# |* H1 D: r3 a) }2 t+ P2 i- {
7 d& t( i+ @  P$ @9 x9 t( q
  services:9 P0 V3 }* N4 S. x7 Z; T; N
    mon: 3 daemons, quorum hostceph1,hostceph2,hostceph3 (age 3M)( y( s$ o$ r2 x, O& H
    mgr: hostceph2(active, since 3M), standbys: hostceph1, hostceph34 c: V1 Z! p3 Z2 Q! }
    osd: 34 osds: 34 up (since 3M), 34 in (since 3M)
; W3 o: W$ Z! i& u# V 6 Z: U0 T! N  ]+ l7 V. p* d
  data:# m, e0 h# H  {/ F; n' c
    pools:   9 pools, 5760 pgs2 C8 d1 q1 _/ u8 s& ?1 v
    objects: 3.11M objects, 12 TiB! ?  S+ O7 a) C- J* L* [7 b
    usage:   33 TiB used, 92 TiB / 126 TiB avail! L8 M# R2 t5 j' J! }4 N
    pgs:     0.417% pgs unknown' F. t( f* h0 K0 s
             5735 active+clean  e' ^* M$ ^3 G: }8 O
             24   unknown; ?+ y* J* M2 K# P) Z$ o# Q  Z7 {8 R
             1    active+clean+scrubbing+deep+ ?9 u2 n1 q' x/ _) I

  F( t# f  I& I# I. X, l  io:
% h" r; g! @9 i5 H, e8 E    client:   212 KiB/s rd, 20 MiB/s wr, 258 op/s rd, 647 op/s wr) C! G* ?5 d* f7 J" ?; z6 z7 O4 t
; j% y! a; M1 |1 o5 D
[root@hostceph1 ~]# ceph -s. \( o5 @/ `$ \. A0 V' h6 ]
  cluster:+ B' M4 J. D( m) m5 h  M
    id:     29046cc0-0682-496b-98b1-912e59964282
3 ]0 `$ O$ x% L3 L0 g    health: HEALTH_WARN
% ]# e! g9 }  }* J            Reduced data availability: 24 pgs inactive  [0 d% V0 ?# v1 R7 `
            24 pgs not deep-scrubbed in time* N: k: |) \$ z8 M2 C
            24 pgs not scrubbed in time* Q6 |* t; e6 s  K$ U7 R
: g: d8 }' g/ S, g/ {
  services:/ S" _. y7 d: K6 m
    mon: 3 daemons, quorum hostceph1,hostceph2,hostceph3 (age 3M)5 Q# K  i3 f3 C* i6 F
    mgr: hostceph2(active, since 3M), standbys: hostceph1, hostceph3, e. `7 R$ u! N) i! }! q
    osd: 34 osds: 34 up (since 3M), 34 in (since 3M)
* a% `7 Z- t0 J   C% w7 v  A4 p( {& U
  data:
4 H0 J6 e& N% Z4 W2 P6 m    pools:   9 pools, 5760 pgs( j- Y7 J# r/ ?- ^
    objects: 3.11M objects, 12 TiB
  j; B% O2 ]6 Q  B# P    usage:   33 TiB used, 92 TiB / 126 TiB avail: R. U" X+ H; |7 u, C  I- u
    pgs:     0.417% pgs unknown6 x2 k6 D4 r4 r, _6 `2 N# y
             5735 active+clean" G6 J1 I% P5 b4 L; J4 h5 [' p% ]
             24   unknown
1 Z5 E! t- v! e5 T3 [0 q% H+ p9 x             1    active+clean+scrubbing+deep. b6 }( x; B; _* [( A$ @' I

/ H5 A9 |! Q4 E  io:
5 E$ K' `. w+ a9 X( T    client:   9.7 KiB/s rd, 20 MiB/s wr, 12 op/s rd, 625 op/s wr
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-11 22:57 , Processed in 0.020605 second(s), 22 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表