找回密码
 注册
查看: 611|回复: 1

HEALTH_ERR 1 pgs inconsistent; 1 pgs repair; 1 scrub errors pg 1.141e is active+

[复制链接]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2023-2-28 09:58:29 | 显示全部楼层 |阅读模式
HEALTH_ERR 1 pgs inconsistent; 1 pgs repair; 1 scrub errors
! w6 r( a% @9 Y% G8 c8 ~+ Kpg 1.141e is active+clean+scrubbing+deep+inconsistent+repair, acting [86,26,85]* ]/ X# z( q' m) y  i9 t: ~0 ]& e
1 scrub errors
  }6 S+ v3 J  g0 C" Z' s6 Q7 c3 d  S" C' _# N* Z) }

0 {; O" `* Y2 v4 n. c  `% p* `[root@cn09 ~]# ceph pg deep-scrub 1.141e * y8 Q9 ^8 e/ }  x+ R5 y3 p2 s
instructing pg 1.141e on osd.86 to deep-scrub
% A4 E9 H8 |+ c$ M8 {$ P[root@cn09 ~]# ceph pg repair 1.141e: F3 h& U+ S3 h' U
instructing pg 1.141e on osd.86 to repair) m# Z- X$ Q: p6 q7 @
[root@cn09 ~]# ceph pg repair 1.141e3 Q& L1 g0 [: w( q
instructing pg 1.141e on osd.86 to repair
; J( b6 _1 G: T8 o[root@cn09 ~]# ceph pg repair 1.141e
/ D2 ?7 p2 b# ?' I" A* |1 m# zinstructing pg 1.141e on osd.86 to repair" a( o5 n# [$ C9 K' P% g& t& w
[root@cn09 ~]# ceph health detail
  ^3 [/ c+ |3 [8 qHEALTH_ERR 1 pgs inconsistent; 1 scrub errors+ ^4 e! t9 C0 J% K/ s" j+ S
pg 1.141e is active+clean+inconsistent, acting [86,26,85]
3 ]1 H( e/ X! c$ I1 H5 A, |1 scrub errors
/ A) v% \; f9 q; a# X: N
: t9 @. F: d1 J. a7 \5 h7 X- ^8 \$ f. f9 [1 c, C
[root@cn09 ~]# ceph osd set noout
; Q* F8 v" b/ y1 p. A/ O" X3 |set noout
& d3 F' [) D' @* S, X; E( p: `0 ][root@cn09 ~]# ceph -s
* @5 R, j8 G$ M; @; L    cluster d385bef6-6778-43cc-8755-4e1d5ef5485e
8 v! u) w3 }* _* c1 Z' H+ y$ {     health HEALTH_ERR
+ j: B+ A5 j- P6 R% M6 d! X( J            1 pgs inconsistent! S" H0 f6 o1 q+ Q$ r
            1 scrub errors$ U8 x! S% r6 S, K$ x, B3 E) _
            noout flag(s) set; ~9 {+ ^: g3 ^9 C7 _1 y
     monmap e3: 3 mons at {}, \; L% F! @% J! V
            election epoch 10, quorum 0,1,2
" `- r$ ~! ~8 C* j& i4 w5 e! e     osdmap e219: 90 osds: 90 up, 90 in
! c1 z: D- N- H0 l: b/ n5 Q            flags noout,sortbitwise,require_jewel_osds
. y) {( J2 B. U, Z& G3 U      pgmap v40230148: 8192 pgs, 1 pools, 29512 GB data, 7460 kobjects
% Z# e4 g  l) F- H* U            88344 GB used, 241 TB / 327 TB avail- H- E, h8 o. u* H7 R- S: T
                8185 active+clean
' ~. L& S3 i" P/ d4 n; B                   6 active+clean+scrubbing+deep
9 X; O7 E) [  d9 p! E                   1 active+clean+inconsistent9 r; G" z5 [+ o9 R; f$ B9 Q8 L  T
  client io 484 MB/s rd, 40044 kB/s wr, 8192 op/s rd, 11982 op/s wr5 H" v6 r: L4 `5 O; f3 u
1 g+ t8 T5 V) ~  R4 x. Z6 x  g8 J  e
[root@cn09 ~]# systemctl stop ceph-osd@86.service 9 J: R9 ]3 p8 Z( V7 Z
[root@cn09 ~]# systemctl status ceph-osd@86.service 4 {5 A$ L2 V+ r& k
ceph-osd@86.service - Ceph object storage daemon0 H$ l- L- v6 j" D7 H' P! m; b9 E
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)1 ^7 A, E8 N; D* S0 S% q
   Active: inactive (dead) since Tue 2023-02-28 09:33:18 CST; 3s ago2 Z# _6 Q1 l+ z
Main PID: 19110 (code=exited, status=0/SUCCESS)# Y$ z, u. Z, q

3 G3 s( L) b. X$ X) z! }; D  DFeb 27 19:31:09 cn09 ceph-osd[19110]: 2023-02-27 19:31:09.098552 7fddf529b700 -1 log_channel(cluster) log [ERR] : 1.141e shard 26: soid 1:782c65a9::...read error
  G5 w' M* u& c! M9 r* ]. PFeb 27 19:31:46 cn09 ceph-osd[19110]: 2023-02-27 19:31:46.357585 7fddf1c96700 -1 log_channel(cluster) log [ERR] : 1.141e deep-scrub 0 missing, 1 inc...nt objects
9 Q0 W" b" \$ m8 Q+ E/ I. tFeb 27 19:31:46 cn09 ceph-osd[19110]: 2023-02-27 19:31:46.357597 7fddf1c96700 -1 log_channel(cluster) log [ERR] : 1.141e deep-scrub 1 errors7 u5 w$ r' r6 C3 u7 b: w" _; X+ O
Feb 28 09:33:14 cn09 ceph-osd[19110]: 2023-02-28 09:33:14.304486 7fdde4e85700 -1 osd.86 219 *** Got signal Terminated ***
1 H, o' {, c4 }1 H* ~/ L2 rFeb 28 09:33:14 cn09 systemd[1]: Stopping Ceph object storage daemon...
% {, H3 {- T$ p/ B1 {* uFeb 28 09:33:14 cn09 ceph-osd[19110]: 2023-02-28 09:33:14.397327 7fdde4e85700 -1 osd.86 219 shutdown4 g- u0 s. @5 |% A
Feb 28 09:33:18 cn09 systemd[1]: Stopped Ceph object storage daemon.
! I& ?7 U+ @, P" `Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.$ m! k& }: k& ?
Hint: Some lines were ellipsized, use -l to show in full.' b" {% p6 I! h% o
[root@cn09 ~]# systemctl start ceph-osd@86.service
% ?9 r! Z% Q$ M1 O- K, A[root@cn09 ~]# systemctl status ceph-osd@86.service
- J% P' }  }& }/ o, N/ |ceph-osd@86.service - Ceph object storage daemon0 b. v7 ]. b1 @5 ^, _, a% l7 {
   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)
: p" q8 U! \4 j/ b   Active: active (running) since Tue 2023-02-28 09:33:26 CST; 2s ago) Q/ P5 o; A7 p
  Process: 2460810 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
: G  c* Q  g0 _+ R8 t7 F* a Main PID: 2460821 (ceph-osd)
0 m: S) y1 _3 t( b  f) Y   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@86.service9 L6 p* d; S! m) {& Z
           └─2460821 /usr/bin/ceph-osd -f --cluster ceph --id 86 --setuser ceph --setgroup ceph0 U4 Z: ^+ v' i4 n4 i
& _2 D/ D" n' w) r- V$ B$ r1 y% A* {
Feb 28 09:33:26 cn09 systemd[1]: Starting Ceph object storage daemon...
: H5 E- f2 ?  f; w8 \6 d; fFeb 28 09:33:26 cn09 systemd[1]: Started Ceph object storage daemon.
0 V; n% j7 E" T5 h3 |1 T! DFeb 28 09:33:26 cn09 ceph-osd[2460821]: starting osd.86 at :/0 osd_data /var/lib/ceph/osd/ceph-86 /var/lib/ceph/osd/ceph-86/journal8 U4 s) ?  A2 [5 p
[root@cn09 ~]# systemctl status ceph-osd@86.service $ i* \6 W) Y. Q5 [: C. G
ceph-osd@86.service - Ceph object storage daemon
5 @) B2 V: Q7 B/ {: f; O   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)
. V) z6 d& N$ _& P2 y+ \   Active: active (running) since Tue 2023-02-28 09:33:26 CST; 3s ago
2 i. `0 }3 c6 x  ^  Process: 2460810 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
! }+ T0 r: G9 F0 b6 Q& a$ S Main PID: 2460821 (ceph-osd)
/ B+ G) J+ \4 X2 Z' j6 d% k3 g" R   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@86.service$ P8 m2 u! {- x" X$ ?# U& G
           └─2460821 /usr/bin/ceph-osd -f --cluster ceph --id 86 --setuser ceph --setgroup ceph$ `5 d0 Q; J' W8 o/ G# i
, O$ B8 |9 ?% k! q8 ]2 o, \
Feb 28 09:33:26 cn09 systemd[1]: Starting Ceph object storage daemon...
+ p- y/ G0 O1 vFeb 28 09:33:26 cn09 systemd[1]: Started Ceph object storage daemon.
# t1 O( V) o. T3 a. F& lFeb 28 09:33:26 cn09 ceph-osd[2460821]: starting osd.86 at :/0 osd_data /var/lib/ceph/osd/ceph-86 /var/lib/ceph/osd/ceph-86/journal; m& b4 B& o% R/ C5 F' o( y+ G  v
[root@cn09 ~]# ceph -s
7 w4 N2 \, F. R: W    cluster d385bef6-6778-43cc-8755-4e1d5ef5485e
' v0 h/ t( u6 @3 B     health HEALTH_ERR
+ `- J  ^' _! L. W8 N3 N; e            240 pgs degraded. v6 u! L- y, q1 ?  p
            1 pgs inconsistent1 e& _: c- q' }! ^. L
            26 pgs stuck unclean
7 F6 p& ?/ c8 B0 z: {            240 pgs undersized
+ V4 F$ h) C) M3 I9 }" p& S* `            recovery 223763/22917261 objects degraded (0.976%)/ o8 \5 \2 u  l6 g3 @8 K& c
            1 scrub errors
$ k; v: V, w" e% P            1/90 in osds are down$ c/ v$ q( E2 Q, A% A' R
            noout flag(s) set
; ^+ r- u* R5 q- K) E. Q     monmap e3: 3 mons at {}8 p4 ]4 X1 }2 G( I
            election epoch 10, quorum 0,1,2 , G& D, D( p2 G0 E6 t+ j( L+ s$ Q
     osdmap e223: 90 osds: 89 up, 90 in; 240 remapped pgs4 F! j4 d- V; e! z* I
            flags noout,sortbitwise,require_jewel_osds
/ a0 a  p% `0 J$ F' y% A      pgmap v40230185: 8192 pgs, 1 pools, 29512 GB data, 7460 kobjects2 E( W1 M- N, S) [% d
            88344 GB used, 241 TB / 327 TB avail
( Y( e3 L. h# c( I            223763/22917261 objects degraded (0.976%)3 c8 v. ^( Q; J% a/ g3 h! M
                7949 active+clean
9 c9 ~9 }! E1 R5 j9 V1 z) r                 239 active+undersized+degraded
# C+ v% _8 u* [+ {/ X) g8 O                   3 active+clean+scrubbing+deep
0 B0 ?% m$ i% B* S$ ?: U% ]# g                   1 active+undersized+degraded+inconsistent4 A1 h/ ^, D1 S( z8 X9 o
  client io 16286 kB/s rd, 29898 kB/s wr, 3339 op/s rd, 7408 op/s wr
, j  _. X( M. K4 ?! z1 ?[root@cn09 ~]# watch ceph -s& m3 q6 d8 x+ v% x- |
[root@cn09 ~]# systemctl status ceph-osd@86.service
/ t0 U" J6 u( E% ~( y8 A% m1 Cceph-osd@86.service - Ceph object storage daemon
1 ~2 Z, q4 E8 u3 P   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: disabled)  g: q+ I4 M& B/ x  Q
   Active: active (running) since Tue 2023-02-28 09:33:26 CST; 8min ago
# h- b' ^$ h# D9 O# ?  Process: 2460810 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)5 ~. O, k% a3 h; ^
Main PID: 2460821 (ceph-osd)
, d- F* H7 x9 I- q   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@86.service  j; \) b6 M+ |& v
           └─2460821 /usr/bin/ceph-osd -f --cluster ceph --id 86 --setuser ceph --setgroup ceph
2 l0 C" x0 D9 {6 z1 U  v, D- L4 ^/ E7 r$ k; R" J' c/ A3 Z
Feb 28 09:33:26cn09 systemd[1]: Starting Ceph object storage daemon...) A. z2 J7 w3 q" B
Feb 28 09:33:26 cn09 systemd[1]: Started Ceph object storage daemon.4 T" o6 A2 j- E9 U+ l
Feb 28 09:33:26 cn09 ceph-osd[2460821]: starting osd.86 at :/0 osd_data /var/lib/ceph/osd/ceph-86 /var/lib/ceph/osd/ceph-86/journal: O- y& _, e: p  K9 Y3 l$ I1 _
Feb 28 09:33:45 cn09 ceph-osd[2460821]: 2023-02-28 09:33:45.106009 7fe7b91d2f80 -1 leveldb: Compacting leveldb store...
  h0 N. K5 a# h1 p5 dFeb 28 09:33:46 cn09 ceph-osd[2460821]: 2023-02-28 09:33:46.428126 7fe7b91d2f80 -1 leveldb: Finished compacting leveldb store
$ N9 O$ O9 u- CFeb 28 09:33:50 cn09 ceph-osd[2460821]: 2023-02-28 09:33:50.066050 7fe7b91d2f80 -1 osd.86 219 log_to_monitors {default=true}
2 ]; K- |% y. k+ p% E[root@cn09 ~]# watch ceph -s' o: c( k( a* }' ~& S, c/ a# k
[root@cn09 ~]# ceph osd unset noout . \0 q4 k8 E# G" f$ ?
unset noout
& W$ x1 M, @$ D% s, o5 h+ F[root@cn09 ~]# watch ceph -s
% C( m* e) i4 I: I7 V$ T7 y; M+ H# T4 V

; k& A4 y; c/ V9 x3 g( a* B
4 S" _4 e- S1 y- C# h3 O, j* `! |

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2023-11-24 00:06:43 | 显示全部楼层
解决过程,发现无发正常同步,通过检查调整参数,达到修复的目的:& T) d# k7 H$ y: I. K
[root compute1 ~]# ceph daemon osd.0 config show |egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" 5 v% c2 e  V6 p: k* L
    "osd_max_backfills": "1",
2 o5 C9 M2 M+ s8 c. h, n3 c    "osd_recovery_max_active": "1",
. W) n7 q0 w) f' N- K    "osd_recovery_op_priority": "0",
/ O% x  k: }; p"osd_recovery_op_priority": "0",  默认是10
1 t* O: _/ G: C) o
2 H: d+ N0 L6 c) C2 P# e+ F* S1 v修改为默认值10  
9 J% v% f7 ?% T0 W9 S# g4 ^; ^  a: M9 {" Q
[root compute1 ~]# ceph tell osd.* injectargs osd_recovery_op_priority=10  v4 a+ p: e7 ]' P# i8 R- h
osd.0: osd_recovery_op_priority = '10' (unchangeable)   s; m. {/ D: z8 H4 p" X1 }
osd.1: osd_recovery_op_priority = '10' (unchangeable)
% Z+ q' _( k, c' M' i! n! |osd.2: osd_recovery_op_priority = '10' (unchangeable)
/ ~+ l, I+ r! S! S- X! Rosd.3: osd_recovery_op_priority = '10' (unchangeable)
; Q4 L% |" }; Sosd.4: osd_recovery_op_priority = '10' (unchangeable)
, z2 ]- v; W: W  {3 M% L( oosd.5: osd_recovery_op_priority = '10' (unchangeable) + o; ~; G8 Q& f2 k. ~
osd.6: osd_recovery_op_priority = '10' (unchangeable) * f* t+ v$ N  u7 [1 e
osd.7: osd_recovery_op_priority = '10' (unchangeable)
+ j  Z# ^" \" R% J5 E! F: Nosd.8: osd_recovery_op_priority = '10' (unchangeable)
( T% e( s3 F4 \) s; ?* Mosd.9: osd_recovery_op_priority = '10' (unchangeable) 5 a3 [6 ]7 y0 S8 R
osd.10: osd_recovery_op_priority = '10' (unchangeable) 2 J+ |* m+ t3 j" Y3 }8 z
osd.11: osd_recovery_op_priority = '10' (unchangeable) 3 Z- C/ |) R, p
osd.12: osd_recovery_op_priority = '10' (unchangeable) 3 `' y4 x6 T' [9 I& l
osd.13: osd_recovery_op_priority = '10' (unchangeable) $ h* U# x8 h- L
osd.14: osd_recovery_op_priority = '10' (unchangeable)
9 |9 |' ^' S; t1 [( posd.15: osd_recovery_op_priority = '10' (unchangeable)
( h/ \4 o' m& @1 l% D/ Eosd.16: osd_recovery_op_priority = '10' (unchangeable)
$ I& _  L- l. c& ]osd.17: osd_recovery_op_priority = '10' (unchangeable) 1 S) u/ @! T  a4 C% m4 Y
osd.18: osd_recovery_op_priority = '10' (unchangeable) + p& C) r% ?+ x  G) y
osd.19: osd_recovery_op_priority = '10' (unchangeable) ) q) z  |! k$ `5 N# e; c
osd.20: osd_recovery_op_priority = '10' (unchangeable) # [0 C8 I; u, `, {
osd.21: osd_recovery_op_priority = '10' (unchangeable) . q4 q4 S& j+ v" }
osd.22: osd_recovery_op_priority = '10' (unchangeable)
) w5 f$ ^+ w5 s; {: Zosd.23: osd_recovery_op_priority = '10' (unchangeable)
9 q- B/ n& L* u+ H' z1 y4 d2 o- qosd.24: osd_recovery_op_priority = '10' (unchangeable) , Y% O* V. U# y2 l" S
osd.25: osd_recovery_op_priority = '10' (unchangeable)   E: h1 e" W  u
osd.26: osd_recovery_op_priority = '10' (unchangeable) + l/ F' _) o. U! m
osd.27: osd_recovery_op_priority = '10' (unchangeable) - r. T# f9 D8 D
osd.28: osd_recovery_op_priority = '10' (unchangeable)
; Z5 G& [) `* i5 aosd.29: osd_recovery_op_priority = '10' (unchangeable)
: o+ ?6 m2 Q) t2 \: Z7 Wosd.30: osd_recovery_op_priority = '10' (unchangeable)
' H) A- L9 h7 k8 j+ _- X; Xosd.31: osd_recovery_op_priority = '10' (unchangeable)
2 e0 q. G6 r0 J0 b6 Kosd.32: osd_recovery_op_priority = '10' (unchangeable) * q7 ]+ U1 u7 _: \; y6 L" S
osd.33: osd_recovery_op_priority = '10' (unchangeable)
% X7 w0 s( ^* n. Cosd.34: osd_recovery_op_priority = '10' (unchangeable)
9 m% T  |2 `0 Rosd.35: osd_recovery_op_priority = '10' (unchangeable)
. I, t9 ~& J# l0 x8 |3 e2 m% Posd.36: osd_recovery_op_priority = '10' (unchangeable) $ x0 k) _$ g4 H) I0 j' T7 \3 f1 }) [$ S" ~
osd.37: osd_recovery_op_priority = '10' (unchangeable)
: ]2 c0 I, i8 xosd.38: osd_recovery_op_priority = '10' (unchangeable)
/ B  e/ V. Q" X1 L. Z' Fosd.39: osd_recovery_op_priority = '10' (unchangeable)
: k& l  i' P% q+ a# ^osd.40: osd_recovery_op_priority = '10' (unchangeable)
* t2 E& Z  e9 f: h/ c+ Gosd.41: osd_recovery_op_priority = '10' (unchangeable)
3 F. Y! }/ O6 _/ Gosd.42: osd_recovery_op_priority = '10' (unchangeable)
2 N0 q% V& V0 Fosd.43: osd_recovery_op_priority = '10' (unchangeable)
" y. G0 c+ c$ ~2 j, Posd.44: osd_recovery_op_priority = '10' (unchangeable)
- F6 i7 R8 W$ f5 k, h: v4 Sosd.45: osd_recovery_op_priority = '10' (unchangeable)
; F* \$ @  O2 y2 |- Aosd.46: osd_recovery_op_priority = '10' (unchangeable) ( V. [  w4 ^5 l  J* x( s; j
osd.47: osd_recovery_op_priority = '10' (unchangeable) 1 _. t( W1 n3 A+ j4 x
osd.48: osd_recovery_op_priority = '10' (unchangeable)
( ~# g% e. _6 |+ t8 rosd.49: osd_recovery_op_priority = '10' (unchangeable)
% R$ j9 A  Q6 w5 @osd.50: osd_recovery_op_priority = '10' (unchangeable)
4 Y4 @/ _2 V" G/ R) Posd.51: osd_recovery_op_priority = '10' (unchangeable) / l+ @% j/ _6 L, S8 ~
osd.52: osd_recovery_op_priority = '10' (unchangeable) & X: I, B7 Z/ r
osd.53: osd_recovery_op_priority = '10' (unchangeable) & {7 j: s! u' R7 V5 B
Error ENXIO: problem getting command descriptions from osd.54# P9 W& ~; I8 u
osd.54: problem getting command descriptions from osd.54
6 v5 W" x; c! V$ |- sosd.55: osd_recovery_op_priority = '10' (unchangeable)
' o% a, M. F1 _; u+ g" K" a1 f$ Gosd.56: osd_recovery_op_priority = '10' (unchangeable)
9 z7 q' N0 b/ {7 q$ k" Aosd.57: osd_recovery_op_priority = '10' (unchangeable)
! s3 O! A9 w/ i4 N" |2 mosd.58: osd_recovery_op_priority = '10' (unchangeable) # K, C# `, j: p! L, u# L# t/ g
osd.59: osd_recovery_op_priority = '10' (unchangeable)
( @  o& r2 L7 |& z" eosd.60: osd_recovery_op_priority = '10' (unchangeable)
% L2 w% x0 o" D6 w/ I9 j: G& posd.61: osd_recovery_op_priority = '10' (unchangeable)
4 L: h/ i" H' [) o/ Wosd.62: osd_recovery_op_priority = '10' (unchangeable)
. J5 E. g' i& H5 U3 |  r4 b( \osd.63: osd_recovery_op_priority = '10' (unchangeable) 5 L+ _9 j( w5 |( k& y
osd.64: osd_recovery_op_priority = '10' (unchangeable)
- f6 f4 H  [- q7 _  ?osd.65: osd_recovery_op_priority = '10' (unchangeable) 7 S4 \2 F) g9 F
osd.66: osd_recovery_op_priority = '10' (unchangeable)
* m  T8 |, d  M- ?0 fosd.67: osd_recovery_op_priority = '10' (unchangeable)
# P4 T# U/ E6 josd.68: osd_recovery_op_priority = '10' (unchangeable)
4 E9 o$ t' J, ^- w. H$ _osd.69: osd_recovery_op_priority = '10' (unchangeable)
) `3 j9 u: b) F. N+ Q. x$ }osd.70: osd_recovery_op_priority = '10' (unchangeable)
1 p$ }2 o4 C; b' Dosd.71: osd_recovery_op_priority = '10' (unchangeable)
% c& @# ]5 I5 _2 v0 Qosd.72: osd_recovery_op_priority = '10' (unchangeable)
+ z+ R  D2 ]7 i1 L  Rosd.73: osd_recovery_op_priority = '10' (unchangeable)
. N4 L; r% B  T( p; c) E/ S- j, u. Nosd.74: osd_recovery_op_priority = '10' (unchangeable)
7 B( J  Y0 o0 D9 n% @0 l3 B+ A: M+ O% Cosd.75: osd_recovery_op_priority = '10' (unchangeable) : S  d3 t: v& C, x$ I% v. T
osd.76: osd_recovery_op_priority = '10' (unchangeable)
- O4 Z3 T( h- [! r; ?osd.77: osd_recovery_op_priority = '10' (unchangeable) 7 L8 g- }8 G7 E( l, V
osd.78: osd_recovery_op_priority = '10' (unchangeable)
; `0 l3 a1 T1 E0 ^) losd.79: osd_recovery_op_priority = '10' (unchangeable) # m- a/ P0 ~% o' \4 T$ y2 e
osd.80: osd_recovery_op_priority = '10' (unchangeable)
# ]! D3 E  h- T1 \osd.81: osd_recovery_op_priority = '10' (unchangeable)
/ M* P% p! I  }+ }6 Kosd.82: osd_recovery_op_priority = '10' (unchangeable)
4 q8 g4 l% H& C8 n7 Eosd.83: osd_recovery_op_priority = '10' (unchangeable) ! t! \- T3 a; T' Z9 m
osd.84: osd_recovery_op_priority = '10' (unchangeable)
$ }9 n* v- j1 M7 N. h+ ?8 Z9 v5 U* gosd.85: osd_recovery_op_priority = '10' (unchangeable)
+ w; I- ?- ?- O& }# K+ eosd.86: osd_recovery_op_priority = '10' (unchangeable) # _; F4 I) S8 g8 O8 A. ~
osd.87: osd_recovery_op_priority = '10' (unchangeable)
( o' e% L' Q; Yosd.88: osd_recovery_op_priority = '10' (unchangeable) $ t+ [$ k8 h, ~1 k4 Z' Q( F
osd.89: osd_recovery_op_priority = '10' (unchangeable)
( ?& s3 x: j7 y& w2 A4 Losd.90: osd_recovery_op_priority = '10' (unchangeable)
& B* N+ Z; t$ `7 Losd.91: osd_recovery_op_priority = '10' (unchangeable) % T! G/ G+ O( w4 R* i
osd.92: osd_recovery_op_priority = '10' (unchangeable)
/ R5 \& Q! `1 mosd.93: osd_recovery_op_priority = '10' (unchangeable)
- I. |( z$ l# A% u8 r7 zosd.94: osd_recovery_op_priority = '10' (unchangeable) " s! Z( g& r0 T# z2 N2 B
osd.95: osd_recovery_op_priority = '10' (unchangeable)
* r' }+ s& j' v6 r' I: P) oosd.96: osd_recovery_op_priority = '10' (unchangeable) % l- j: t* r# e1 G" L
osd.97: osd_recovery_op_priority = '10' (unchangeable) & p2 Z" U6 g" m/ A6 m) [
osd.98: osd_recovery_op_priority = '10' (unchangeable)
- ]$ k# [' s. sosd.99: osd_recovery_op_priority = '10' (unchangeable) " z( A! a  @1 r  R( o
osd.100: osd_recovery_op_priority = '10' (unchangeable)
5 [' ]' k  J: ]- {. K/ K$ Y, Rosd.101: osd_recovery_op_priority = '10' (unchangeable)
* x: f6 E' J) v9 O( S" ~" Q% Nosd.102: osd_recovery_op_priority = '10' (unchangeable)
# S( S: k3 B- S! Nosd.103: osd_recovery_op_priority = '10' (unchangeable) & X+ `  N: D/ O7 M. D
osd.104: osd_recovery_op_priority = '10' (unchangeable)   W- M/ {$ c, c# b
osd.105: osd_recovery_op_priority = '10' (unchangeable) / F  m* n- T2 s! S. D
osd.106: osd_recovery_op_priority = '10' (unchangeable) 6 [  J4 b. h) G$ ~6 a. Z
osd.107: osd_recovery_op_priority = '10' (unchangeable) % @# e5 f; d) d- f: X
osd.108: osd_recovery_op_priority = '10' (unchangeable) 3 S; ?. k1 B- @  ?
osd.109: osd_recovery_op_priority = '10' (unchangeable) , D) Q& D2 R5 v* I' t
osd.110: osd_recovery_op_priority = '10' (unchangeable) ! ?. l  ^& V& L, _# v
osd.111: osd_recovery_op_priority = '10' (unchangeable)
# N+ n+ @; i2 n$ `osd.112: osd_recovery_op_priority = '10' (unchangeable)
; y8 y" L5 @$ k& r' d. posd.113: osd_recovery_op_priority = '10' (unchangeable) $ k& i0 I( _, {6 I
osd.114: osd_recovery_op_priority = '10' (unchangeable)
) z$ r, l" ^$ Aosd.115: osd_recovery_op_priority = '10' (unchangeable)
8 i% s1 b8 T7 F$ b4 |  Nosd.116: osd_recovery_op_priority = '10' (unchangeable)
# Y9 r& c! |8 x! D7 i3 l  i7 t3 r3 xosd.117: osd_recovery_op_priority = '10' (unchangeable) - o9 Y8 g4 y' a: {. C5 v
osd.118: osd_recovery_op_priority = '10' (unchangeable)
; R4 S% _, ?: x" y5 T6 m  Zosd.119: osd_recovery_op_priority = '10' (unchangeable) # o; j8 W( k7 p, l
osd.120: osd_recovery_op_priority = '10' (unchangeable) 9 h: T& O% `5 h  W
osd.121: osd_recovery_op_priority = '10' (unchangeable)
+ x+ @* v% a* N1 d6 h, `osd.122: osd_recovery_op_priority = '10' (unchangeable) 1 b1 O; F& `; D4 r4 N5 Q1 D7 n
osd.123: osd_recovery_op_priority = '10' (unchangeable)
( j3 [; R/ s9 V3 pError ENXIO: problem getting command descriptions from osd.124# b1 }5 |: z( U) X* y1 o
osd.124: problem getting command descriptions from osd.1243 F. u( J  e5 N
osd.125: osd_recovery_op_priority = '10' (unchangeable) % N0 Y  T+ ?) |
osd.126: osd_recovery_op_priority = '10' (unchangeable)
  {. n$ M( m2 U) O5 a5 a: gError ENXIO: problem getting command descriptions from osd.127
* \5 \- D% c/ `1 Eosd.127: problem getting command descriptions from osd.127  d$ Z2 @' b- N4 ^% R4 M9 u0 i7 N" @
osd.128: osd_recovery_op_priority = '10' (unchangeable) $ I( {2 n# C8 p/ x- n$ ^, F' p
osd.129: osd_recovery_op_priority = '10' (unchangeable) 0 V4 A$ K/ n+ S' d. q
osd.130: osd_recovery_op_priority = '10' (unchangeable) " V& ]6 N  X& h" o: G3 {
You have new mail in /var/spool/mail/root
9 N: b/ Z+ I$ O( @
7 i2 Y; q! D4 K' S所有的osd都修改:; f5 s* f( v5 S  _2 h

+ S4 v& e' s: |0 \5 `执行过ceph pg repair pg-id+ P& f2 S- Y' ~, Q

4 \5 r6 C, ?5 j: ~7 k
8 F7 p$ k7 V. u状态出现修改过程:: R% r. }  V& A
[root compute1 ~]# ceph s
0 j# N, r  L* \0 T    cluster 70d27aec742e4a95b000-cf37ebba35d0
( W6 B! f. [$ O& q3 T2 B7 j     health HEALTH_ERR
* f& e8 _/ h3 c/ r2 Z. q            1 pgs inconsistent
" E" W$ o, Z4 D2 }7 u% p            1 pgs repair6 m! s: C: F/ Q, o1 D* t
            1 scrub errors) T: R4 q! p# ?  Q! P
     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}% o* G' W- L- K
            election epoch 332, quorum 0,1,2 compute1,compute2,compute39 r6 A4 ~( }1 A2 Z, ]
     osdmap e26725: 131 osds: 128 up, 128 in
, Z: L# j5 g$ }5 N8 z  y; V* E            flags sortbitwise,require_jewel_osds3 g/ G; E0 I$ G' n- C
      pgmap v169370679: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects
4 h/ I! _' X8 y' S            309 TB used, 155 TB / 465 TB avail6 M3 A9 P1 _6 O0 M
               10229 active+clean
- ]. e! I$ s( r  \                   9 active+clean+scrubbing+deep) {8 ~2 a3 @4 {! ~5 n" E+ q' W5 m
                   1 active+clean+scrubbing7 G: M# u' a" o. m1 q! W
                   1 active+clean+scrubbing+deep+inconsistent+repair
$ R/ K2 L: R0 r& p6 C' {! k" v  client io 91086 kB/s rd, 67615 kB/s wr, 3119 op/s rd, 3569 op/s wr  k% t, f  R+ o9 L' h. M
[root compute1 ~]# ceph s
0 |4 A# I- A9 A" I: |: u9 @    cluster 70d27aec742e4a95b000-cf37ebba35d0
* B' r/ j4 ~0 D' ^& {' {     health HEALTH_ERR5 ~$ d& N% p3 @3 R8 ]0 N, V3 C
            1 pgs inconsistent% _3 a  S0 C/ w4 S$ `* b! ~
            1 pgs repair- v! [3 G8 z3 @1 K! A% g! ]
            1 scrub errors
* h. d* ?2 ?% x5 A     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}
  S4 n6 S9 ^, X* ~) g            election epoch 332, quorum 0,1,2 compute1,compute2,compute3
9 [" v, {" e5 S  d' E) }" x     osdmap e26725: 131 osds: 128 up, 128 in6 L7 d/ j8 [, }
            flags sortbitwise,require_jewel_osds9 ~) M0 D: T. e9 l+ m4 ?
      pgmap v169370681: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects+ V0 E5 [8 ?- q; c7 ~
            309 TB used, 155 TB / 465 TB avail' E6 e7 ]4 z: y- A: y% y
               10230 active+clean5 O0 }. w& J, [, a% a
                   8 active+clean+scrubbing+deep
5 Z& Z7 U/ M& F& \+ |                   1 active+clean+scrubbing
3 u* x* W5 A5 w! _! W0 h                   1 active+clean+scrubbing+deep+inconsistent+repair
- Q: f6 w% G9 v6 f# n' T0 v  client io 104 MB/s rd, 67033 kB/s wr, 3479 op/s rd, 5013 op/s wr
" R" y: |9 T* }; k[root compute1 ~]# ceph s% U+ m' C% h) ?% r
    cluster 70d27aec742e4a95b000-cf37ebba35d0
5 w% h6 j& x) h; c) c& O$ [     health HEALTH_ERR
4 i  s% F; O7 |' H6 L6 r# I- y            1 pgs inconsistent
$ n0 p" N; A9 D( P8 U8 _+ }            1 pgs repair0 f( @/ ~5 p( `5 g2 V6 \+ ]
            1 scrub errors3 ^+ Y9 F. u7 @& m
     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}
! @! X$ _; G& R+ ?+ D; }            election epoch 332, quorum 0,1,2 compute1,compute2,compute33 R2 y6 L6 n& t9 Q/ [( u
     osdmap e26725: 131 osds: 128 up, 128 in
: V$ l& ~5 }0 @6 {7 P8 J* T6 H            flags sortbitwise,require_jewel_osds
+ G- }- w# n% _# q' M; ~, x0 D0 i      pgmap v169370783: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects1 w" l! Q8 n7 g8 c* L6 ^
            309 TB used, 155 TB / 465 TB avail
, M: ~2 [, B; b% q, V  @1 _/ }' h               10232 active+clean
+ i, F/ X- {! n                   6 active+clean+scrubbing+deep
) E0 k; H0 l! Q7 @5 ]1 R3 K                   1 active+clean+scrubbing8 l! s7 Z- e6 k  l# A
                   1 active+clean+scrubbing+deep+inconsistent+repair9 P; X3 }# ^4 F2 z  I* Y
  client io 147 MB/s rd, 57692 kB/s wr, 4584 op/s rd, 5944 op/s wr
  j9 e& T3 f& B/ u$ L+ |You have mail in /var/spool/mail/root
+ ?/ z3 \/ P% H( V) A[root compute1 ~]# ceph s0 |! ?3 ]$ M8 i! h: c' H  h0 v
    cluster 70d27aec742e4a95b000-cf37ebba35d0/ h8 |3 d6 Y4 t, P' M: N5 f9 @
     health HEALTH_ERR
+ W) Y/ W( ~& u& m6 y2 s+ ^            1 pgs inconsistent
' V& e' u# K; A1 @& d4 ?            1 pgs repair
/ ~' y  [. p  z, r$ X, u0 h, ]            1 scrub errors
" x. S0 F8 z! a     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}
+ ^+ \* V/ t) [- P% G            election epoch 332, quorum 0,1,2 compute1,compute2,compute3
0 b0 G0 h& [0 k# G- [     osdmap e26725: 131 osds: 128 up, 128 in( L" M, }3 {1 `3 K
            flags sortbitwise,require_jewel_osds
$ l( u' X$ j* a+ z      pgmap v169370785: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects! h" s; {/ A  o7 E% z
            309 TB used, 155 TB / 465 TB avail
7 [. A$ F8 C/ y' W. N               10232 active+clean
; F& N9 R( u2 |                   6 active+clean+scrubbing+deep
3 |0 g- i+ Z+ r; h( X6 o: t) g                   1 active+clean+scrubbing
/ l- a. ~% r1 r                   1 active+clean+scrubbing+deep+inconsistent+repair
. B1 w3 `6 s0 C& U- G& i4 J8 H( A  client io 104 MB/s rd, 49581 kB/s wr, 2830 op/s rd, 3282 op/s wr  e4 b/ D1 k! v; m: C
[root compute1 ~]# ceph s! s! S! A9 Y8 x
    cluster 70d27aec742e4a95b000-cf37ebba35d0- ?' d9 f1 Q; ~6 c, r0 c
     health HEALTH_ERR
* b/ Q# b. L+ e, ^% _! b; j& g4 _            1 pgs inconsistent
5 e4 ]& J0 {( B, x            1 pgs repair
! M; {" q& y) c7 l            1 scrub errors
$ \- j* }& r3 M. S0 G     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}
" z8 U0 `: E' t2 L# f* l1 L9 H            election epoch 332, quorum 0,1,2 compute1,compute2,compute3
0 P% }& B& X. M* Q$ ^1 I     osdmap e26725: 131 osds: 128 up, 128 in
$ ^6 `# |3 A* k$ f  q            flags sortbitwise,require_jewel_osds
1 E+ c7 q0 u3 q, y# V      pgmap v169370786: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects
9 [. H& O0 O( r- h3 U            309 TB used, 155 TB / 465 TB avail1 Q6 m: O* _# V9 p0 L) j
               10232 active+clean) y3 ?- M! a# K+ S' w
                   6 active+clean+scrubbing+deep: \( Z2 |, h6 ^, A+ c6 F' e" l: v% I
                   1 active+clean+scrubbing) C+ L. G# Q) f- H+ k$ @4 B+ H, V$ p
                   1 active+clean+scrubbing+deep+inconsistent+repair
+ j8 n, A' N( U/ T( _) r4 L6 X  client io 95762 kB/s rd, 44372 kB/s wr, 2258 op/s rd, 2660 op/s wr' {/ M& ~0 @1 v7 w  [# V6 r

- R( t6 @6 i+ s: ~: n0 z9 E; Q6 I! [6 Y+ {; w# m1 f$ r' v) S
[root compute1 ~]# ceph daemon osd.0 config show |egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills"
) p* G6 ~3 V* s- F1 d* @    "osd_max_backfills": "1",) r0 s8 H3 X& }  s+ d4 Z
    "osd_recovery_max_active": "1",' a, K  b: C: f; `( {
    "osd_recovery_op_priority": "10",3 `* r$ I+ F3 E! L& M
[root compute1 ~]# ceph daemon osd.0 config show |egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" , }8 e% t) O2 |
    "osd_max_backfills": "1",
( O. f' q/ x4 c( F+ d% Y: r    "osd_recovery_max_active": "1",
; _2 B) w2 t  w    "osd_recovery_op_priority": "10",
3 x( @' [% f: V[root compute1 ~]# ceph s% Z7 t: }$ k% S3 b' s
    cluster 70d27aec742e4a95b000-cf37ebba35d0
' h9 W; R5 z5 G) I. d6 x$ t     health HEALTH_ERR, K* N& }  {. W; [+ n* a  m
            1 pgs inconsistent
3 J+ [. v  t$ y) F/ U            1 pgs repair
. t  ^/ d# H6 u" w! e- c+ k            1 scrub errors
7 g( b; x& [* U6 K     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}3 d9 ]& W* ^- W& z: A# g
            election epoch 332, quorum 0,1,2 compute1,compute2,compute3
$ U+ |7 |2 j0 h9 @7 o     osdmap e26725: 131 osds: 128 up, 128 in* G: S2 n8 H( l$ D; A
            flags sortbitwise,require_jewel_osds8 F# j, T* F$ D6 P1 z8 Q+ @" M9 i
      pgmap v169370799: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects
  {# i) z4 p+ ^# [" i            309 TB used, 155 TB / 465 TB avail# K+ h% m0 {" c! d9 f* V: _% t# g
               10232 active+clean
$ l3 H3 K# _9 {8 M                   6 active+clean+scrubbing+deep
% w5 v; k5 g0 Q2 D; K( C# r                   1 active+clean+scrubbing6 l' G- |# e6 ~8 m9 ~7 ]3 _1 I
                   1 active+clean+scrubbing+deep+inconsistent+repair6 ^% g* N  t  o# y7 p/ D5 M2 h: X
  client io 134 MB/s rd, 117 MB/s wr, 3865 op/s rd, 4949 op/s wr
0 p5 {0 @! j$ `. E* S
) N/ e0 _; f. V% {8 [" \! X
$ w% n6 ]+ g0 n[root compute1 ~]# ceph daemon osd.0 config show |egrep "osd_recovery_max_active|osd_recovery_op_priority|osd_max_backfills" ' n' ?& H/ q( n% ?6 w
    "osd_max_backfills": "1",
8 }$ E' o0 w" s  t1 J9 `    "osd_recovery_max_active": "1",
+ V! A# D# j/ n" a8 I1 t- ^    "osd_recovery_op_priority": "10",
$ K4 E- p5 k" u7 V
5 _! W) N9 N! l  C; \1 g5 H$ k( [8 _  g2 |3 z
# a( s* u( a8 M  `
[root compute1 ~]# ceph s* A: e6 E0 P# n# f
    cluster 70d27aec742e4a95b000-cf37ebba35d0
) W9 ?: D+ U4 r     health HEALTH_OK
" j+ h7 m$ B4 R. O( B5 E7 I% e     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}
* S' V4 P$ z1 a            election epoch 332, quorum 0,1,2 compute1,compute2,compute36 |5 ^  l6 l, d/ K* @
     osdmap e26725: 131 osds: 128 up, 128 in, p3 m9 G. h, k4 v1 I
            flags sortbitwise,require_jewel_osds
- P) g7 |7 @; B6 m" z& B$ {      pgmap v169370916: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects
. q9 I2 |# \  c6 E8 }! q) h( }            309 TB used, 155 TB / 465 TB avail+ c6 t2 a/ E! j: v8 h$ r
               10233 active+clean
1 }6 b5 g* y5 a6 D3 z# n                   6 active+clean+scrubbing+deep
. v7 d9 k8 `! U; u8 d                   1 active+clean+scrubbing
) J; F$ g' q# f9 H* z; |$ N; m  client io 195 MB/s rd, 234 MB/s wr, 4404 op/s rd, 5993 op/s wr
6 I4 @! h' s- N6 pYou have mail in /var/spool/mail/root: e0 `9 d2 h. [

0 h* u" \( M* K( N; P[root compute1 ~]# ceph s$ }, m8 r" V; e3 T4 b/ J; |
    cluster 70d27aec742e4a95b000-cf37ebba35d0  r: X) C( E9 s1 [2 l* `0 F
     health HEALTH_OK* @1 _, ^9 s- \! ]1 q( G
     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}
9 ^; J8 a7 }' w  M            election epoch 332, quorum 0,1,2 compute1,compute2,compute3
" A2 e" A! Q" n0 E     osdmap e26725: 131 osds: 128 up, 128 in4 l/ g( `6 A1 {5 A
            flags sortbitwise,require_jewel_osds6 y' c! U9 P  x/ f
      pgmap v169370921: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects
1 m5 R# b2 ?" U+ \            309 TB used, 155 TB / 465 TB avail
. w+ n, v; }2 m& A3 Y               10233 active+clean
1 U9 q2 y" z! k1 P( X                   6 active+clean+scrubbing+deep
; J9 e( Y( O# ?- x$ L! N$ I                   1 active+clean+scrubbing
* T# t9 A# ^6 Z0 c9 L8 A( }  client io 257 MB/s rd, 358 MB/s wr, 5025 op/s rd, 6587 op/s wr5 ]; h' z' U! V( D, y; X2 w& S: M$ ^5 u

* q8 m, T% A) K& c恢复正常:
7 u. k* W, c9 W; ?[root compute1 ~]# ceph s0 k* a- M8 r% L% J
    cluster 70d27aec742e4a95b000-cf37ebba35d0
6 s  V% c2 V3 I2 m  o9 p# f     health HEALTH_OK
) h* [6 D9 E/ o. O* ]- `7 K     monmap e3: 3 mons at {compute1=176.12.132.5:6789/0,compute2=176.12.132.6:6789/0,compute3=176.12.132.7:6789/0}
( a4 i! ^/ J  b: e# j% q  W            election epoch 332, quorum 0,1,2 compute1,compute2,compute37 S6 ^& W5 d. _* Y  X1 Z
     osdmap e26725: 131 osds: 128 up, 128 in- C( d' `2 R5 I& N2 p
            flags sortbitwise,require_jewel_osds/ r* L3 @2 c5 j5 g) o
      pgmap v169370922: 10240 pgs, 1 pools, 103 TB data, 27242 kobjects
; l4 u/ a/ v' N            309 TB used, 155 TB / 465 TB avail0 R# w( o  h& @, j. B7 _! W4 h! Y
               10233 active+clean
/ s7 g8 A6 c  Q7 t7 o                   6 active+clean+scrubbing+deep
; H' [. H4 }+ S; m# C                   1 active+clean+scrubbing& }  ~$ G4 ^( @0 t' Z, E
  client io 254 MB/s rd, 390 MB/s wr, 5522 op/s rd, 6826 op/s wr
$ r2 |4 f/ T. ?& a* Q6 q4 a2 }3 s0 R
+ |" h$ H  l# y' O; N5 N) }6 w' ^3 l7 B$ K* Q! p! Q
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-11 23:05 , Processed in 0.020775 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表