处理过程osd down掉了，服务状态正常HEALTH_WARN 2 osds down; Reduced data availability: 29 pgs

admin

ceph health detail
HEALTH_WARN 2 osds down; Reduced data availability: 29 pgs inactive
8 Y% T8 E  Z' F* XOSD_DOWN 2 osds down
    osd.6 (root=hdd,host=hdd-ceph1) is down
$ S8 G' q; U/ ~9 X  _9 ]    osd.11 (root=hdd,host=hdd-ceph1) is down
PG_AVAILABILITY Reduced data availability: 29 pgs inactive
    pg 11.1 is stuck inactive for 81874.695636, current state unknown, last acting []
7 ^9 |  d% O& [- s3 e    pg 11.5 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.22 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.31 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.3b is stuck inactive for 81874.695636, current state unknown, last acting []
; t( m8 I* w. M$ r9 `: F    pg 11.5d is stuck inactive for 81874.695636, current state unknown, last acting []
; C& }  F. X' a7 k# q    pg 11.5f is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.63 is stuck inactive for 81874.695636, current state unknown, last acting []
: E: J: ~1 q; J: @4 B# a& m    pg 11.67 is stuck inactive for 81874.695636, current state unknown, last acting []
! Y. d0 q# @: x# r+ |5 `    pg 11.79 is stuck inactive for 81874.695636, current state unknown, last acting []
2 u# G; l% W: E% G    pg 11.83 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.90 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.91 is stuck inactive for 81874.695636, current state unknown, last acting []
( [) Y9 j) ~, z" m9 h; t" c    pg 11.93 is stuck inactive for 81874.695636, current state unknown, last acting []
( B. Y8 V+ Y. w- Z2 \6 M+ \    pg 11.a1 is stuck inactive for 81874.695636, current state unknown, last acting []
! l* B0 A5 W8 q    pg 11.a4 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.aa is stuck inactive for 81874.695636, current state unknown, last acting []
; N: x% ~8 e/ \- M! E2 @% H5 d    pg 11.b3 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.b6 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.b8 is stuck inactive for 81874.695636, current state unknown, last acting []
9 W: y6 w4 R: }$ V8 E    pg 11.ca is stuck inactive for 81874.695636, current state unknown, last acting []
# G  k/ ]5 t% @6 W& x    pg 11.cf is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.da is stuck inactive for 81874.695636, current state unknown, last acting []
( w# L, h- K& X

1 E7 |$ j) i8 s6 \: V9 M! |
8 A8 z8 r; e3 _# q) n6 V
/ `* t4 n" ]7 v; g" H    pg 11.e6 is stuck inactive for 81874.695636, current state unknown, last acting []
    pg 11.e8 is stuck inactive for 81874.695636, current state unknown, last acting []
) {( J2 h& D, g: _) g/ |    pg 11.ec is stuck inactive for 81874.695636, current state unknown, last acting []
7 t+ m* ^: p. W8 m    pg 11.ef is stuck inactive for 81874.695636, current state unknown, last acting []
6 P# h! j- S# [0 A2 {    pg 11.fa is stuck inactive for 81874.695636, current state unknown, last acting []
  k3 l5 t1 S  |: B1 K; A6 S    pg 11.fb is stuck inactive for 81874.695636, current state unknown, last acting []

3 W' A" b1 q1 Qpg好像卡住了。
' k, I% z4 X) a( C0 w
  r( X. C$ K9 y0 ]$ u
日志刷新很快：

2021-10-27 10:22:59.527 7f9b2d0f5700  1 osd.6 pg_epoch: 23810 pg[10.1c( v 4595'732 (0'0,4595'732] local-lis/les=13109/131=[13109,23810)/3 crt=4595'732 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [8] -> [14,8], acting [8] -> [14,ting 4611087854031667199
2021-10-27 10:22:59.528 7f9b2e0f7700  1 osd.6 pg_epoch: 23739 pg[8.49d( v 4595'444 (0'0,4595'444] local-lis/les=13081/130081,23739)/5 crt=4595'444 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14] -> [], acting [14] -> [], acting1087854031667199
( k, b( v5 I( z2 z6 a0 _2021-10-27 10:22:59.539 7f9b2e8f8700  1 osd.6 pg_epoch: 23770 pg[8.339( v 4595'513 (0'0,4595'513] local-lis/les=13099/13113099,23770)/9 crt=4595'513 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14,8] -> [14], acting [14,8] -> [1acting 4611087854031667199
; a3 \" D9 |# [, N2 d* R2021-10-27 10:22:59.540 7f9b2e0f7700  1 osd.6 pg_epoch: 23746 pg[8.49d( v 4595'444 (0'0,4595'444] local-lis/les=13081/13013081,23746)/5 crt=4595'444 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [] -> [14], acting [] -> [14], acti1087854031667199
0 S) B! {# [: b2021-10-27 10:22:59.541 7f9b2c8f4700  1 osd.6 pg_epoch: 23808 pg[8.4aa( v 4595'411 (0'0,4595'411] local-lis/les=13114/131i=[13112,23808)/11 crt=4595'411 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [22,8,14] -> [22,8], acting [2254031667199 upacting 4611087854031667199
2021-10-27 10:22:59.544 7f9b2c8f4700  1 osd.6 pg_epoch: 23810 pg[8.4aa( v 4595'411 (0'0,4595'411] local-lis/les=13114/1310 pi=[13112,23810)/10 crt=4595'411 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [22,8] -> [22,8,14], acting 87854031667199 upacting 4611087854031667199
2021-10-27 10:22:59.545 7f9b2e8f8700  1 osd.6 pg_epoch: 23773 pg[8.339( v 4595'513 (0'0,4595'513] local-lis/les=13099/131=[13099,23773)/9 crt=4595'513 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14] -> [14,8], acting [14] -> [1upacting 4611087854031667199
* c, }8 G) a- h2 _2021-10-27 10:22:59.546 7f9b2e0f7700  1 osd.6 pg_epoch: 23749 pg[8.49d( v 4595'444 (0'0,4595'444] local-lis/les=13081/130i=[13081,23749)/5 crt=4595'444 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14] -> [23,14], acting [14] -> 99 upacting 4611087854031667199
2021-10-27 10:22:59.566 7f9b2d0f5700  1 osd.6 pg_epoch: 23838 pg[10.1c( v 4595'732 (0'0,4595'732] local-lis/les=13109/13113109,23838)/3 crt=4595'732 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14,8] -> [14], acting [14,8] -> [1acting 4611087854031667199
( }' r. [+ t" R  {" m2021-10-27 10:22:59.566 7f9b2d0f5700  1 osd.6 pg_epoch: 23838 pg[10.1c( v 4595'732 (0'0,4595'732] local-lis/les=13109/13113109,23838)/3 crt=4595'732 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
! f; M. N( K- ?. @+ g2021-10-27 10:22:59.597 7f9b2c8f4700  1 osd.6 pg_epoch: 23838 pg[8.4aa( v 4595'411 (0'0,4595'411] local-lis/les=13114/131pi=[13112,23838)/11 crt=4595'411 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [22,8,14] -> [22,14], acting [87854031667199 upacting 4611087854031667199
/ @5 T: v& c! x2021-10-27 10:22:59.597 7f9b2c8f4700  1 osd.6 pg_epoch: 23838 pg[8.4aa( v 4595'411 (0'0,4595'411] local-lis/les=13114/131pi=[13112,23838)/11 crt=4595'411 lcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray
2021-10-27 10:22:59.606 7f9b2d0f5700  1 osd.6 pg_epoch: 23739 pg[8.765( v 4595'410 (0'0,4595'410] local-lis/les=13094/130094,23739)/3 crt=4595'410 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14] -> [], acting [14] -> [], acting1087854031667199
2021-10-27 10:22:59.612 7f9b2e8f8700  1 osd.6 pg_epoch: 23808 pg[8.339( v 4595'513 (0'0,4595'513] local-lis/les=13099/1313099,23808)/9 crt=4595'513 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [14,8] -> [8], acting [14,8] -> [8],g 4611087854031667199
2021-10-27 10:22:59.617 7f9b2e8f8700  1 osd.6 pg_epoch: 23810 pg[8.339( v 4595'513 (0'0,4595'513] local-lis/les=13099/131=[13099,23810)/9 crt=4595'513 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [8] -> [14,8], acting [8] -> [14,ting 4611087854031667199
9 _- b; [' K& [* N2021-10-27 10:22:59.620 7f9b2d8f6700  1 osd.6 pg_epoch: 23808 pg[8.38b( v 4595'408 (0'0,4595'408] local-lis/les=13099/13113099,23808)/12 crt=4595'408 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [23,14] -> [23], acting [23,14] -> upacting 4611087854031667199
2021-10-27 10:22:59.622 7f9b2d0f5700  1 osd.6 pg_epoch: 23746 pg[8.765( v 4595'410 (0'0,4595'410] local-lis/les=13094/13013094,23746)/3 crt=4595'410 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [] -> [14], acting [] -> [14], acti1087854031667199
+ G  ~. w3 Z( h$ p2021-10-27 10:22:59.624 7f9b2d8f6700  1 osd.6 pg_epoch: 23810 pg[8.38b( v 4595'408 (0'0,4595'408] local-lis/les=13099/131i=[13099,23810)/12 crt=4595'408 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [23] -> [23,14], acting [23] ->199 upacting 4611087854031667199
  |& J" s' d3 a0 ~# M, F; S* B
4 p) s& Y7 @1 H5 g3 g
# {1 _0 f$ {+ O% d重启两个pg所在的osd
systemctl ceph-osd@x.service
; M3 R; l1 V) N3 N
) d: p' L) \( C7 m% D" c6 \3 \( O

admin

[root@ceph1 ~]# systemctl status ceph-osd@6.service
( M: R. ]: }+ e; r● ceph-osd@6.service - Ceph object storage daemon osd.6
& o1 V/ B+ V( D   Loaded: loaded ([url=]/usr/lib/systemd/system/ceph-osd@.service[/url]; enabled-runtime; vendor preset: disabled)
   Active: active (running) since Wed 2021-10-27 10:22:44 CST; 11min ago
% n6 ^4 V6 K4 \- I, q  Process: 6169 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i (code=exited, status=0/SUCCESS)
% j4 v. t, o' E. X7 L# y! h: q9 Z* f" b Main PID: 6175 (ceph-osd)
/ `7 S6 T$ @. m9 }   CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
           └─6175 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
Oct 27 10:22:44 ceph1 systemd[1]: Starting Ceph object storage daemon osd.6...
Oct 27 10:22:44 ceph1 systemd[1]: Started Ceph object storage daemon osd.6.
7 ~, \$ m  k# c+ S4 E  ~3 sOct 27 10:22:55 ceph1 ceph-osd[6175]: 2021-10-27 10:22:55.875 7f9b532a0a80 -1 osd.6 23718 log_to_monitors {default=true}
Oct 27 10:29:37 ceph1 ceph-osd[6175]: 2021-10-27 10:29:37.277 7f9b45125700 -1 osd.6 25440 set_numa_affinity unab...ectory
5 ~. a5 I0 K. E/ [7 @' iHint: Some lines were ellipsized, use -l to show in full.
, L: j( u, o4 Z& t
状态正常，但是ceph health deatil却不正常：
2 osds down
            Reduced data availability: 29 pgs inactive

5 v4 p, x& k, P1 g/ B
9 J+ ?: O4 [" o3 S! Q( W看到网上的处理过程：

8 [: V& r# R* c7 ~7 b

显示这3个 pg 卡住了

运行 pg query, 查看该 pg 的具体信息, 显示:

[root@controller ~]# ceph pg 3.1e query
! D7 F3 J; n4 B; ]1 c' x
& k9 @. T1 }: Z( ~/ X4 U
Error ENOENT: i don't have pgid 3.1e
: a0 q. `1 ?1 }, m% ?, U# f
/ x! K+ b# |5 v# b4 Y

无法找到该 pg id.

运行 pg dump_stuck unclean, 显示:

  g3 b  z& \+ {2 `9 D5 P

分析

看来是这几个 pgid 彻底找不到了. 我的 osd pool 有三个, 分别叫 l1 (1副本), l2 (2副本), l3 (3副本).

估计可能是之前写入 1 副本的数据由于硬盘挂掉导致的数据丢失.

既然是1副本, 也不要求数据可靠性了. 并且本身存储的也是一些下载到一半的数据, 也没什么关系

修正

通过阅读 CEPH 官方 PG troubleshooting 文档 , 发现了解决方案:

POOL SIZE = 1

If you have the osd pool default size set to 1, you will only have one copy of the object. OSDs rely on other OSDs to tell them which objects they should have.

If a first OSD has a copy of an object and there is no second copy, then no second OSD can tell the first OSD that it should have that copy. For each placement group mapped to the first OSD (see ceph pg dump),

you can force the first OSD to notice the placement groups it needs by running:

ceph osd force-create-pg <pgid>

即, 多 osd 副本可以互相通知 pg 信息, 但是单副本就会丢, 为了恢复这个pg, 我们可以强行创建它.

[root@controller ~]# ceph osd force-create-pg 3.1e
, A1 G' N. L1 l

Error EPERM: This command will recreate a lost (as in data lost)
PG with data in it, such that the cluster will give up ever trying to recover the lost data.  
/ M. Z7 |! r. e  Z8 u* Q

Do this only if you are certain that all copies of the PG are in fact lost and you are willing to accept that the data is permanently destroyed.
' r6 P, S/ c, z) i/ o Pass --yes-i-really-mean-it to proceed.

$ K/ l- |- F9 u/ D

运行创建命令, 提示, 运行会永久的丢失该 pg 的数据, 需要加上 --yes-i-really-mean-it.

[backcolor=rgb(245, 245, 245) !important][url=]https://common.cnblogs.com/images/copycode.gif[/url]


) [$ m% n+ M$ ^" _! r[root@controller ~]# ceph osd force-create-pg 3.1e --yes-i-really-mean-it

& g& d" i, R% T1 u
1 l) E( ]6 S3 }pg 3.1e now creating, ok


; J! L/ M5 a7 |. e% n: F[root@controller ~]# ceph osd force-create-pg 3.b --yes-i-really-mean-it

6 U6 Y! }: G' f; ]0 i/ ^
pg 3.b now creating, ok
# B" f# q6 _& ]  I

! t3 {- @" u4 I7 M: ~[root@controller ~]# ceph osd force-create-pg 3.4 --yes-i-really-mean-it

! ]2 l: C' Y6 u8 V
pg 3.4 now creating, ok

感觉有些可行性，于是直接走最后一步：

. z. j' Y6 D; T9 y3 u9 U  d" P[root@ceph1 ~]# ceph osd force-create-pg 11.1 --yes-i-really-mean-it
pg 11.1 now creating, ok
+ Z' t$ ?+ _1 J- j' x9 }[root@ceph1 ~]#
[root@ceph2 ~]# ceph -s
  F$ h, D- b: Q! I9 D- m3 m' _  cluster:
    id:     4d8d7309-ad9e-4566-bee5-69b8d805dd57
- D# E1 o" G# Y7 F    health: HEALTH_OK

$ z0 ?* K+ f' l% H, }+ a9 S' l  services:
    mon: 5 daemons, quorum ceph1,ceph2,ceph3,compute1,compute2 (age 28m)
$ U- s* f, J1 \6 T: Q    mgr: ceph2(active, since 22h), standbys: compute3, compute2, compute1, ceph3, ceph1
    osd: 36 osds: 36 up (since 47s), 36 in (since 10m)

  data:
    pools:   9 pools, 5888 pgs
: p: p+ e1 E2 {" Y7 U: q0 Q    objects: 143.19k objects, 549 GiB
1 S0 K& J/ T0 M. o& @4 ~3 w    usage:   5.9 TiB used, 127 TiB / 133 TiB avail
$ G  V( N2 T$ t    pgs:     5888 active+clean
4 z, e& _5 D/ D0 Z
( |2 g5 Z$ d: I  io:
8 B7 l  x6 C8 H7 O    client:   258 KiB/s rd, 35 KiB/s wr, 224 op/s rd, 6 op/s wr
$ m" Y. l: D2 B' L# O+ w


没想到ok状态了。
5 z+ O# h4 P% v2 ^- s; m0 h
5 y2 c2 J/ \7 I. D+ J( o5 @
/ ~0 L2 i6 Y: \7 n0 C- E7 S6 i

admin

[root@ceph1 ~]# ceph osd force-create-pg 11.5 --yes-i-really-mean-it
pg 11.5 now creating, ok
5 u4 M6 W. R5 z& B! D[root@ceph1 ~]# ceph osd force-create-pg 11.22 --yes-i-really-mean-it
pg 11.22 now creating, ok
/ ]' {& ^0 M2 N6 H+ w
4 f! O& t" H7 b: m( q, Q& {2 u
' n) j# v  u; {, W6 @* l
ceph 修复成功。
6 }: d- d1 j; H7 c$ R8 p

admin

[root@ceph1 ~]# ceph pg dump_stuck
degraded    inactive    stale       unclean     undersized