HEALTH_WARN 1 failed cephadm daemon(s)

admin

HEALTH_WARN 1 failed cephadm daemon(s)
ceph health detail
HEALTH_WARN 2 failed cephadm daemon(s)
[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
. h  }: V9 }% Y' n    daemon alertmanager.controller on controller is in error state
! w4 m  }+ y$ y. K: f( A    daemon grafana.controller on controller is in error state

admin

经过排查，应该是系统层间安装过ceph集群，没有清理干净。新版本还不知道怎么全部清除，还在测试中。

admin

[root@compute01 ~]# ceph status
  Z) ]$ L% R( `9 p- C  cluster:
    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6
    health: HEALTH_WARN
3 M! R- |5 p# `6 L0 O            2 failed cephadm daemon(s)
            clock skew detected on mon.compute01
8 G6 [% A& c3 P$ B5 H+ R0 M
6 c# D: `( P3 s  services:
+ `- x9 {% w6 c0 `) |3 t    mon: 2 daemons, quorum controller,compute01 (age 3h)
    mgr: compute01.getqhn(active, since 3h), standbys: controller.kxfttd
  D4 V5 R) z& X0 G3 B    osd: 3 osds: 3 up (since 3h), 3 in (since 3h)
* k' e) g( Y' c8 \
, Y) j7 r- K' _0 ?+ G  data:
1 Y6 Z) t4 Y( }0 K1 i    pools:   1 pools, 1 pgs
7 h" r3 o; D( P9 ]- z& L1 I# _    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 1.2 TiB / 1.2 TiB avail
    pgs:     1 active+clean
/ G) z9 e0 H( N9 L- U% \9 {
4 O% z6 x* V8 `0 X; i" F! z
[root@compute01 ~]# systemctl status ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6
7 G# q; m7 V! s; ?; w( A, lceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service  
6 y0 p2 Y+ S0 h& p% h" r( v! J[root@compute01 ~]# systemctl status ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6
2 c/ @7 X% ~# \$ [, L$ e2 U" Sceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service  
0 ~$ n" y$ \. `" V3 Z



9 m! o% ^3 X- g8 _[root@compute01 ~]# systemctl disable ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service
" V8 U! t* \- T9 [+ ZRemoved [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service[/url].
. w* w- P8 Q, O/ X" ^( |: a0 T; SRemoved [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service[/url].
Removed [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service[/url].
Removed [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service[/url].
' i- A4 m  ~+ Y' U% x. K

[root@compute01 ~]# cd /var/lib/ceph
; ^1 x5 n/ V! ?2 O" c  I
[root@controller ~]# cd /var/lib/ceph
' f: X4 i6 v! K0 I. o4 e[root@controller ceph]# rm -rf 1e87bca4-e7ce-11eb-aa90-0025908471d6/


4 d: o! G% E* O[root@controller ceph]# ceph status  
  cluster:
2 B+ g, }# ?, Y* }    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6
3 x; Z# y1 U1 V+ N9 A7 i    health: HEALTH_OK

- s: }# P9 H8 [/ g8 B0 q  X  services:
+ }4 i# t2 B; i    mon: 2 daemons, quorum controller,compute01 (age 84s)
. @9 I1 W# P" \) C8 i    mgr: compute01.getqhn(active, since 30s)
    osd: 3 osds: 3 up (since 14s), 3 in (since 4h)
4 C& X7 Z9 e! ^% T, Q
  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
- H' Q& Z* L) b( C" j# D8 m    usage:   3.0 GiB used, 1.2 TiB / 1.2 TiB avail
    pgs:     1 active+clean

问题竟然解决。
2 \- }* B- A- y) B: F0 E* @
8 H+ M& N( s1 A& X0 D( Q9 o, H4 Q

admin

admin 发表于 2021-7-25 20:59
[root@compute01 ~]# ceph status
  k: {3 {# K  _/ a# c5 o  cluster:
4 m2 a, a" i5 M* \% `9 s    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6

" V( w/ e. J, e2 Y+ W4 F+ [可能原因是因为生产了一个不一样的cluster_id导致信息不一致不对称引起。如果一个纯净的系统也出现这个问题，需要排除原因。