ceph分布式集群出现pgunfound objects处理过程记录

admin

当我们执行ceph  health detail 命令时报错：
* ?' N& s9 f0 G8 r[root@mon1 ~]# ceph health detail
& B7 J- ?8 ^" A% kHEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged; noout flag(s) set; 1/16745475 objects unfound (0.000%); Degraded data redundancy: 2/33490950 objects degraded (0.000%), 1 pg degraded
' w: a" B3 s6 ?  @" w& i: gFS_DEGRADED 1 filesystem is degraded
    fs cephfs is degraded
MDS_DAMAGE 1 mds daemon damaged
/ U/ x) C! b; ]; C8 i* z! o/ H" ^( G    fs cephfs mds.0 is damaged
OSDMAP_FLAGS noout flag(s) set
" b$ g* o( K  I- N. a) HOBJECT_UNFOUND 1/16745475 objects unfound (0.000%)
    pg 7.123 has 1 unfound objects
PG_DEGRADED Degraded data redundancy: 2/33490950 objects degraded (0.000%), 1 pg degraded
    pg 7.123 is active+recovery_wait+degraded, acting [5,15], 1 unfound

- h$ }3 P- c& q/ ^) ], c$ _2 G确认哪些对象找不到了：
9 {( N1 c9 v+ Q2 P. S1 L[root@mon1 ~]# ceph pg 7.123 list_missing
{
0 o$ G% `3 i6 v+ H. y; U5 U    "offset": {
1 R. e' k; R6 |( y* {. s        "oid": "",
        "key": "",
        "snapid": 0,
        "hash": 0,
        "max": 0,
- \1 s  v8 A" X4 \' U1 M        "pool": -9223372036854775808,
8 U0 \9 v3 }* A( `  J0 H% D. S        "namespace": ""
    },
) e8 q5 l+ r; H; k$ ?( \& q9 C    "num_missing": 1,
  u# s" ~3 t% t    "num_unfound": 1,
    "objects": [
( c, ?9 t; N# S3 N% l( w% {        {
            "oid": {
                "oid": "1000b73777d.00000000",
  Y, k, `) x' Q/ h                "key": "",
- C; v: J9 F* r$ G1 ?2 E* X% x                "snapid": -2,
                "hash": 4108226851,
                "max": 0,
  g) S2 \4 b2 ]+ d                "pool": 7,
                "namespace": ""
5 z7 V/ m* M% s            },
            "need": "13279'1",
1 d: d) ?9 x/ p( `5 G# J. J3 ]            "have": "0'0",
            "flags": "none",
! d# a! o$ O4 @& y            "locations": []
        }
    ],
    "more": false
}
  Y5 C& Q9 }1 o
; n3 z3 Z+ L0 H; A' W5 C- v如果在一次查询里列出的对象太多， more 这个字段将为 true ，你就可以查询更多。
3 d) j( a8 C0 A5 \其次，你可以找出哪些 OSD 上探测到、或可能包含数据：
  f/ D6 Z9 b, p9 s

ceph pg 7.123 query



[root@mon1 ~]# watch ceph -s
+ |% c8 o4 S; X2 J还无法恢复，只有放弃丢失的对象。执行如下命令回滚或删除对象：

ceph pg  {pgname}  mark_unfound_lost revert|delete
- w. M/ W! t7 K1 [8 n
& K6 ]: f: y" L. [& U6 c+ K

• revert选项：回滚到对象的前一个版本
• delete选项：完全删除这个对象
  & f3 M+ L9 F, y4 C

/ M0 @! G* k9 Y" D# F
+ D$ e1 _" y8 S这里采用删除的方式解决：
, S  I' r8 C/ q[root@mon1 ~]# ceph pg 7.123 mark_unfound_lost delete
* j* x9 \7 n. ?. r* e: C& c. ypg has 1 objects unfound and apparently lost marking
0 e- T% M9 K/ R- Z2 D[root@mon1 ~]#
; o+ Z& D1 Z4 [7 f

+ o2 M0 n4 {  _