浏览此站
联系我们相册切换到窄版 Language

 找回密码
 注册
易陆发现论坛»论坛 › 云计算开源区 › ceph分布式存储 › 处理过程osd down掉了，服务状态正常HEALTH_WARN 2 osds ...
返回列表 发新帖
查看: 2076|回复: 3

处理过程osd down掉了，服务状态正常HEALTH_WARN 2 osds down; Reduced data availability: 29 pgs

[复制链接]
admin

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2021-10-27 10:26:02 | 显示全部楼层 |阅读模式
ceph health detail
' |  i( [5 G" L" p- X' c3 \+ RHEALTH_WARN 2 osds down; Reduced data availability: 29 pgs inactive
- [& Q5 L. T  f; C/ S$ kOSD_DOWN 2 osds down
6 U+ i' w0 ]' x, f9 v    osd.6 (root=hdd,host=hdd-ceph1) is down
1 K# e" s. g% b( a. G    osd.11 (root=hdd,host=hdd-ceph1) is down/ e. B2 M, Z+ d! c: i6 a) o
PG_AVAILABILITY Reduced data availability: 29 pgs inactive
8 h. n7 \5 K% O, b    pg 11.1 is stuck inactive for 81874.695636, current state unknown, last acting []$ D6 d$ z6 D; y2 i
    pg 11.5 is stuck inactive for 81874.695636, current state unknown, last acting []
# f; P  w* |+ i# ]) ?3 I    pg 11.22 is stuck inactive for 81874.695636, current state unknown, last acting []
+ D2 C: D. Z& x8 F& u2 I7 T    pg 11.31 is stuck inactive for 81874.695636, current state unknown, last acting []
  Z; [$ |. R' ]  e4 i1 Z    pg 11.3b is stuck inactive for 81874.695636, current state unknown, last acting []
9 [& M  c! W) \& A: p  N    pg 11.5d is stuck inactive for 81874.695636, current state unknown, last acting []
7 q, u: X; z, Y6 i1 w! w+ A! ?    pg 11.5f is stuck inactive for 81874.695636, current state unknown, last acting []4 }' I( J7 v3 l
    pg 11.63 is stuck inactive for 81874.695636, current state unknown, last acting []" Z/ H! l7 k& i7 i
    pg 11.67 is stuck inactive for 81874.695636, current state unknown, last acting []- I* W6 H# ^4 l7 u4 d' V8 J
    pg 11.79 is stuck inactive for 81874.695636, current state unknown, last acting []# f' n9 `. h* L6 @
    pg 11.83 is stuck inactive for 81874.695636, current state unknown, last acting []. \( m! O1 b9 w4 c9 s7 j
    pg 11.90 is stuck inactive for 81874.695636, current state unknown, last acting []; }6 N( o/ X; p2 F6 G
    pg 11.91 is stuck inactive for 81874.695636, current state unknown, last acting []
- W4 q- @- Z7 Z: Z0 `& d) E7 t    pg 11.93 is stuck inactive for 81874.695636, current state unknown, last acting []
0 ~" _6 N3 _) K3 i    pg 11.a1 is stuck inactive for 81874.695636, current state unknown, last acting []
  J: U* ~4 o# ~* [' X. z" }2 Y    pg 11.a4 is stuck inactive for 81874.695636, current state unknown, last acting []+ w- Z3 S3 d# u. i' ^  m# ^
    pg 11.aa is stuck inactive for 81874.695636, current state unknown, last acting []8 v# N4 ?& }: V+ t( {
    pg 11.b3 is stuck inactive for 81874.695636, current state unknown, last acting []3 `; i% x# `) y$ a$ M2 G
    pg 11.b6 is stuck inactive for 81874.695636, current state unknown, last acting []+ q7 }' e! f. s8 F6 d3 k9 [8 Z
    pg 11.b8 is stuck inactive for 81874.695636, current state unknown, last acting []
0 c' b; h2 G" i! l2 t    pg 11.ca is stuck inactive for 81874.695636, current state unknown, last acting [], ]0 G, R* Y  O$ j$ `; ~  N
    pg 11.cf is stuck inactive for 81874.695636, current state unknown, last acting []
9 @, i# W1 h! G# `% a1 Y7 x    pg 11.da is stuck inactive for 81874.695636, current state unknown, last acting [], p/ O5 Q- n0 h" W

4 {7 s- W  B+ E7 U6 P3 Y) t  x* q. A) Q

# |% k- t2 A3 P# F* x( B' {4 w7 a( |3 g7 C: Y
    pg 11.e6 is stuck inactive for 81874.695636, current state unknown, last acting []) ?% b6 c9 L  h4 k* a
    pg 11.e8 is stuck inactive for 81874.695636, current state unknown, last acting []6 W* T. B* h: x6 z
    pg 11.ec is stuck inactive for 81874.695636, current state unknown, last acting []
) F& P* M1 |; G  K9 F    pg 11.ef is stuck inactive for 81874.695636, current state unknown, last acting []) X4 K( Q% n% W5 Y) `
    pg 11.fa is stuck inactive for 81874.695636, current state unknown, last acting []
2 N' B5 i0 e2 w- t    pg 11.fb is stuck inactive for 81874.695636, current state unknown, last acting []  z, x$ ?% ]3 r) \6 m

+ N! E2 \3 I7 w8 cpg好像卡住了。
0 N* t. w* U0 F* H" G# z
# y5 l/ L/ b6 q' Z$ R8 ]' V# i/ U# J
日志刷新很快：9 O$ |& o2 e* |" O: Z4 b7 W
% w, i1 r$ V, J" r4 D) X
2021-10-27 10:22:59.527 7f9b2d0f5700  1 osd.6 pg_epoch: 23810 pg[10.1c( v 4595'732 (0'0,4595'732] local-lis/les=13109/131=[13109,23810)/3 crt=4595'732 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [8] -> [14,8], acting [8] -> [14,ting 46110878540316671993 e* x5 m0 r& H" I* M5 q. Y
2021-10-27 10:22:59.528 7f9b2e0f7700  1 osd.6 pg_epoch: 23739 pg[8.49d( v 4595'444 (0'0,4595'444] local-lis/les=13081/130081,23739)/5 crt=4595'444 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14] -> [], acting [14] -> [], acting1087854031667199
) p9 |% p/ i# m+ L9 `" l+ i2021-10-27 10:22:59.539 7f9b2e8f8700  1 osd.6 pg_epoch: 23770 pg[8.339( v 4595'513 (0'0,4595'513] local-lis/les=13099/13113099,23770)/9 crt=4595'513 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14,8] -> [14], acting [14,8] -> [1acting 4611087854031667199
5 W. M# f, ^5 f( m! z4 Z2021-10-27 10:22:59.540 7f9b2e0f7700  1 osd.6 pg_epoch: 23746 pg[8.49d( v 4595'444 (0'0,4595'444] local-lis/les=13081/13013081,23746)/5 crt=4595'444 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [] -> [14], acting [] -> [14], acti1087854031667199
: p8 q& X$ f& n2021-10-27 10:22:59.541 7f9b2c8f4700  1 osd.6 pg_epoch: 23808 pg[8.4aa( v 4595'411 (0'0,4595'411] local-lis/les=13114/131i=[13112,23808)/11 crt=4595'411 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [22,8,14] -> [22,8], acting [2254031667199 upacting 46110878540316671990 [. k( @' ~# R$ s% E3 b
2021-10-27 10:22:59.544 7f9b2c8f4700  1 osd.6 pg_epoch: 23810 pg[8.4aa( v 4595'411 (0'0,4595'411] local-lis/les=13114/1310 pi=[13112,23810)/10 crt=4595'411 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [22,8] -> [22,8,14], acting 87854031667199 upacting 4611087854031667199
# Q! w) F9 g2 |5 S1 i2021-10-27 10:22:59.545 7f9b2e8f8700  1 osd.6 pg_epoch: 23773 pg[8.339( v 4595'513 (0'0,4595'513] local-lis/les=13099/131=[13099,23773)/9 crt=4595'513 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14] -> [14,8], acting [14] -> [1upacting 4611087854031667199
1 V. L* q8 S  z( w% ?: C% |2021-10-27 10:22:59.546 7f9b2e0f7700  1 osd.6 pg_epoch: 23749 pg[8.49d( v 4595'444 (0'0,4595'444] local-lis/les=13081/130i=[13081,23749)/5 crt=4595'444 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14] -> [23,14], acting [14] -> 99 upacting 4611087854031667199! y" ^( _- X( [5 z" @6 Z
2021-10-27 10:22:59.566 7f9b2d0f5700  1 osd.6 pg_epoch: 23838 pg[10.1c( v 4595'732 (0'0,4595'732] local-lis/les=13109/13113109,23838)/3 crt=4595'732 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14,8] -> [14], acting [14,8] -> [1acting 46110878540316671999 u( i' e" i7 f! L& a& @; z
2021-10-27 10:22:59.566 7f9b2d0f5700  1 osd.6 pg_epoch: 23838 pg[10.1c( v 4595'732 (0'0,4595'732] local-lis/les=13109/13113109,23838)/3 crt=4595'732 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray+ i) M' g6 |2 s8 |) @# E" N
2021-10-27 10:22:59.597 7f9b2c8f4700  1 osd.6 pg_epoch: 23838 pg[8.4aa( v 4595'411 (0'0,4595'411] local-lis/les=13114/131pi=[13112,23838)/11 crt=4595'411 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [22,8,14] -> [22,14], acting [87854031667199 upacting 4611087854031667199
. t  a% ?- g' ^* Z- x3 G2021-10-27 10:22:59.597 7f9b2c8f4700  1 osd.6 pg_epoch: 23838 pg[8.4aa( v 4595'411 (0'0,4595'411] local-lis/les=13114/131pi=[13112,23838)/11 crt=4595'411 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
( m4 h7 e7 W3 I6 s% b$ N. N4 j( e2021-10-27 10:22:59.606 7f9b2d0f5700  1 osd.6 pg_epoch: 23739 pg[8.765( v 4595'410 (0'0,4595'410] local-lis/les=13094/130094,23739)/3 crt=4595'410 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14] -> [], acting [14] -> [], acting1087854031667199
8 g/ ^4 a$ }. C: k; ~7 Z5 z2 G2021-10-27 10:22:59.612 7f9b2e8f8700  1 osd.6 pg_epoch: 23808 pg[8.339( v 4595'513 (0'0,4595'513] local-lis/les=13099/1313099,23808)/9 crt=4595'513 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14,8] -> [8], acting [14,8] -> [8],g 4611087854031667199
& w- v5 c2 I! d, n& c- i% e2021-10-27 10:22:59.617 7f9b2e8f8700  1 osd.6 pg_epoch: 23810 pg[8.339( v 4595'513 (0'0,4595'513] local-lis/les=13099/131=[13099,23810)/9 crt=4595'513 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [8] -> [14,8], acting [8] -> [14,ting 4611087854031667199
* j9 L; `' D6 i- b% \( g2021-10-27 10:22:59.620 7f9b2d8f6700  1 osd.6 pg_epoch: 23808 pg[8.38b( v 4595'408 (0'0,4595'408] local-lis/les=13099/13113099,23808)/12 crt=4595'408 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [23,14] -> [23], acting [23,14] -> upacting 4611087854031667199- p( b& Y5 i3 ~4 M) n
2021-10-27 10:22:59.622 7f9b2d0f5700  1 osd.6 pg_epoch: 23746 pg[8.765( v 4595'410 (0'0,4595'410] local-lis/les=13094/13013094,23746)/3 crt=4595'410 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [] -> [14], acting [] -> [14], acti1087854031667199+ J0 Q7 x3 D6 |! _8 K! w
2021-10-27 10:22:59.624 7f9b2d8f6700  1 osd.6 pg_epoch: 23810 pg[8.38b( v 4595'408 (0'0,4595'408] local-lis/les=13099/131i=[13099,23810)/12 crt=4595'408 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [23] -> [23,14], acting [23] ->199 upacting 46110878540316671997 T6 x  k% r. h5 w  E+ Z* h
8 d! t6 \. G& v3 w

6 |* b( N6 Z; e9 N4 c重启两个pg所在的osd
, b9 A( Z; @* F4 Y4 Rsystemctl ceph-osd@x.service* Y8 [& b( `' H- H

& f- q! ~# F) x* R, Z1 g
回复

举报

admin

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-10-27 10:28:57 | 显示全部楼层
[root@ceph1 ~]# systemctl status ceph-osd@6.service / R& }# x! Q/ R+ [; o1 C7 `
● ceph-osd@6.service - Ceph object storage daemon osd.6
" W' j. I* d  y7 |" Y/ J- p   Loaded: loaded ([url=]/usr/lib/systemd/system/ceph-osd@.service[/url]; enabled-runtime; vendor preset: disabled)
# Y8 @, R" h  R! }/ Q   Active: active (running) since Wed 2021-10-27 10:22:44 CST; 11min ago
5 x& q7 X' Z) P2 ]  Process: 6169 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
; R" K1 S3 L, C& e3 d, X Main PID: 6175 (ceph-osd)
% r5 k% g9 p' i9 l9 V  ^   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service7 K  j$ H- r, {; L1 S
           └─6175 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph' f+ ?; L- K! g) y$ _3 S/ V
Oct 27 10:22:44 ceph1 systemd[1]: Starting Ceph object storage daemon osd.6.../ o$ D: w' @- N9 C  U
Oct 27 10:22:44 ceph1 systemd[1]: Started Ceph object storage daemon osd.6.
0 D6 A( \4 o+ e7 T: Z# iOct 27 10:22:55 ceph1 ceph-osd[6175]: 2021-10-27 10:22:55.875 7f9b532a0a80 -1 osd.6 23718 log_to_monitors {default=true}/ e9 s# }) {  K; l9 ?6 G3 ^
Oct 27 10:29:37 ceph1 ceph-osd[6175]: 2021-10-27 10:29:37.277 7f9b45125700 -1 osd.6 25440 set_numa_affinity unab...ectory  ~* ^3 Y5 E1 y1 l, a; ]8 R5 b
Hint: Some lines were ellipsized, use -l to show in full.; e$ r# `* z; N2 q4 w

$ o( }% |0 ?/ u, S! ]状态正常，但是ceph health deatil却不正常：$ W5 s* w' W% u; p
2 osds down/ Z" }. B# C+ v0 D9 C
            Reduced data availability: 29 pgs inactive9 B' M: M, R- n
1 n" k1 G# a( Q! K/ @" J

8 S/ q0 Q! Y8 m; ]9 {( X  P" E看到网上的处理过程：5 R2 w# O2 D6 r( v% b

. b2 h: b& `3 `+ Y+ k% r
显示这3个 pg 卡住了
运行 pg query, 查看该 pg 的具体信息, 显示:
[root@controller ~]# ceph pg 3.1e query
+ k* t+ e* L# [
6 `% O4 p7 V& a& j6 u
4 }' `* t( b/ |# p5 v, c1 s0 ~3 MError ENOENT: i don't have pgid 3.1e
) H7 b# a5 y3 o( Z6 j( W- Y: Z; o0 Z4 ~( \9 e8 d
无法找到该 pg id.
运行 pg dump_stuck unclean, 显示:
" {- L* q; G1 y; V4 C2 V/ v
分析
看来是这几个 pgid 彻底找不到了. 我的 osd pool 有三个, 分别叫 l1 (1副本), l2 (2副本), l3 (3副本).
估计可能是之前写入 1 副本的数据由于硬盘挂掉导致的数据丢失.
既然是1副本, 也不要求数据可靠性了. 并且本身存储的也是一些下载到一半的数据, 也没什么关系
修正
通过阅读 CEPH 官方 PG troubleshooting 文档 , 发现了解决方案:
POOL SIZE = 1
If you have the osd pool default size set to 1, you will only have one copy of the object. OSDs rely on other OSDs to tell them which objects they should have.
If a first OSD has a copy of an object and there is no second copy, then no second OSD can tell the first OSD that it should have that copy. For each placement group mapped to the first OSD (see ceph pg dump),
* w8 W* t, Z& F1 z) n
you can force the first OSD to notice the placement groups it needs by running:
ceph osd force-create-pg <pgid>
即, 多 osd 副本可以互相通知 pg 信息, 但是单副本就会丢, 为了恢复这个pg, 我们可以强行创建它.
[root@controller ~]# ceph osd force-create-pg 3.1e  ^1 u# X5 |) i2 x+ ]$ {3 n

6 j, N8 M+ L' X0 Z; w3 o$ Q8 z! C' B7 V9 c2 x9 K; E2 q4 J
Error EPERM: This command will recreate a lost (as in data lost)
. x) B1 x; z' A4 a/ C! i9 @  HPG with data in it, such that the cluster will give up ever trying to recover the lost data.  $ q; W2 T! K6 q" }1 h. t, M9 L, I
" t" ?6 g6 C8 Y9 F$ M/ c" h; V

- _( q( Z$ y" `8 e8 U; c: j  d+ yDo this only if you are certain that all copies of the PG are in fact lost and you are willing to accept that the data is permanently destroyed. & I0 x/ e) X: Q
Pass --yes-i-really-mean-it to proceed. + d  s% _7 h6 n3 Z
5 ?( |9 L+ r# M1 q+ L6 F" `

3 }, A) ~+ W) l) {' S
运行创建命令, 提示, 运行会永久的丢失该 pg 的数据, 需要加上 --yes-i-really-mean-it.
[backcolor=rgb(245, 245, 245) !important][url=]https://common.cnblogs.com/images/copycode.gif[/url]# D( d3 r; B" D& d
2 _; O( u+ r2 s* x0 @2 n
! _0 i# {& d) E6 G$ o, h' y  S
[root@controller ~]# ceph osd force-create-pg 3.1e --yes-i-really-mean-it
. g+ V" D: h2 [+ q
' H9 S5 v. V4 e* T
$ g$ \3 O0 h$ H) [4 \7 ^pg 3.1e now creating, ok3 L, n* g! l* f; Q/ u) }
# C' b% ^6 t' K; O  p3 K+ U$ P

+ B7 p. z4 V- J: Z[root@controller ~]# ceph osd force-create-pg 3.b --yes-i-really-mean-it& Q8 {; C: B& j0 _: }
+ y. O) P+ c% P% L
0 o- n6 n: B: ~1 [/ N
pg 3.b now creating, ok
; N/ L$ p3 A" ?* p0 z: \- Z1 P8 k0 ~, v+ D% d4 ]0 q( O
# u6 o- x; j3 d; s6 {. O) @0 T
[root@controller ~]# ceph osd force-create-pg 3.4 --yes-i-really-mean-it
. Z. E# w' Z4 w& Z5 m' v
3 l$ K( d0 H) m4 Z8 o3 L) s! R
. @- t6 f+ B1 N3 c  R' i  Lpg 3.4 now creating, ok
) U' F" g$ V1 w7 u
6 z! |8 `) H1 C2 W$ i) Y9 [9 m感觉有些可行性，于是直接走最后一步：+ h+ F! X+ J+ x, P( }( b

- u! Y+ C& x( y[root@ceph1 ~]# ceph osd force-create-pg 11.1 --yes-i-really-mean-it
$ m5 j; D) ?6 F  y# r4 zpg 11.1 now creating, ok
8 r1 {4 n0 X' x9 [' Y- p$ }[root@ceph1 ~]# , `& t8 A$ Q4 P" {
[root@ceph2 ~]# ceph -s
0 C) l) x2 X' M' k  K  cluster:
, {# n# q6 z" W/ k. J9 f" r" B5 Z6 n    id:     4d8d7309-ad9e-4566-bee5-69b8d805dd571 C' Y, E/ v7 I: a# o' M
    health: HEALTH_OK
; w7 v' x+ i7 Q* M/ o5 z4 ?3 Q
% `, B5 j+ z- ?, `4 U' U- X  services:0 h% r* I3 T+ B
    mon: 5 daemons, quorum ceph1,ceph2,ceph3,compute1,compute2 (age 28m)
$ e4 {6 Q' V9 l5 k1 e7 ^% K    mgr: ceph2(active, since 22h), standbys: compute3, compute2, compute1, ceph3, ceph10 H4 D: W6 k) i3 ]! }# \
    osd: 36 osds: 36 up (since 47s), 36 in (since 10m); E2 z' \0 M2 k

9 _0 m2 A! k0 x# z0 S  data:0 ]: A& x2 p# N* f  H; }4 A
    pools:   9 pools, 5888 pgs
- {6 P9 I; d% m: V, y4 h5 {    objects: 143.19k objects, 549 GiB$ t8 {- u; \( `( q# r- g2 B
    usage:   5.9 TiB used, 127 TiB / 133 TiB avail+ \; N, W  [5 g: y- W
    pgs:     5888 active+clean
, _( O( M7 z4 A- P8 P ) i7 Q' q. `6 Y+ i* s
  io:/ O8 q. N8 X! u# B1 p
    client:   258 KiB/s rd, 35 KiB/s wr, 224 op/s rd, 6 op/s wr
$ b# |) E# y3 d* ~, Q5 a8 G
! N7 \) ^6 Y6 q, k, Z1 {3 V
5 N+ m# l7 {* S' G4 P( V' G  v, X0 D% U2 C) k
没想到ok状态了。3 T+ @3 n( |2 `8 D+ Y, Y! [& a
2 p+ f8 ?: V+ z! I# p/ N8 U
# w% o7 |$ H- b4 [" x5 o
! B# s) z; E1 R& G8 h5 k
回复

举报

admin

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-10-27 10:33:21 | 显示全部楼层
[root@ceph1 ~]# ceph osd force-create-pg 11.5 --yes-i-really-mean-it
6 r' J" D9 o) }& \! wpg 11.5 now creating, ok
! s' s" @" H: e+ p( V' M/ L[root@ceph1 ~]# ceph osd force-create-pg 11.22 --yes-i-really-mean-it. I) S2 `# [, P' M
pg 11.22 now creating, ok( Z  `* g# V# c* k" ]1 `

  h" a3 \/ Y: w1 O* ~* F4 g. ^( Y* h
6 h! l9 U8 A7 X0 {" U4 L
ceph 修复成功。$ I/ [" B% i' ]: t, ~/ J* z: g- o
回复

举报

admin

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-10-27 10:40:56 | 显示全部楼层
[root@ceph1 ~]# ceph pg dump_stuck
* A/ _6 s2 ]* Y; y) C, s( Y: ndegraded    inactive    stale       unclean     undersized  
回复

举报

返回列表 发新帖
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-12 01:01 , Processed in 0.018757 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表