HEALTH_WARN 1 failed cephadm daemon(s)

admin

HEALTH_WARN 1 failed cephadm daemon(s)
  e( H9 i% c' w2 O) lceph health detail
HEALTH_WARN 2 failed cephadm daemon(s)
[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
    daemon alertmanager.controller on controller is in error state
+ J* }! n1 x# X8 D8 u    daemon grafana.controller on controller is in error state
# u. k* U: y( A- a. u
) r" C, ^$ ?1 V
* t" K5 y. G+ C0 O1 |

admin

经过排查，应该是系统层间安装过ceph集群，没有清理干净。新版本还不知道怎么全部清除，还在测试中。

admin

[root@compute01 ~]# ceph status
& ~# z. h, a% o$ k$ r; W: w6 V" L  cluster:
" ~" x5 N9 U# f! e    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6
) t, |$ U8 ]2 m& h6 M: J    health: HEALTH_WARN
            2 failed cephadm daemon(s)
            clock skew detected on mon.compute01

/ W  ?: W1 y! o& t5 w* M% p  services:
0 I' C1 k; y0 u. ?3 ~, r3 a% N    mon: 2 daemons, quorum controller,compute01 (age 3h)
    mgr: compute01.getqhn(active, since 3h), standbys: controller.kxfttd
0 _" j2 m& s( f7 {( M6 A) B    osd: 3 osds: 3 up (since 3h), 3 in (since 3h)
1 K  ]3 A% x6 e  y, I
  data:
( g$ J2 n+ ^& }1 i3 F! U    pools:   1 pools, 1 pgs
" g1 t7 `' ]' U, q. C& }    objects: 0 objects, 0 B
" q! i0 s$ J$ B' k, B9 C% M    usage:   3.0 GiB used, 1.2 TiB / 1.2 TiB avail
    pgs:     1 active+clean


* u  k! M2 d0 X! Y8 i$ m3 ?& O[root@compute01 ~]# systemctl status ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6
1 ~7 `+ h" a  g5 y+ Sceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service
* I  }: b- i+ A3 R" c) oceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target
) Q' X* o, n8 f7 rceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service  
$ }6 o# W) X5 I& [  `+ c[root@compute01 ~]# systemctl status ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6
6 N# z* ]$ W6 n" nceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service  




[root@compute01 ~]# systemctl disable ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service
Removed [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service[/url].
- M0 I+ J2 @, L) a8 ]0 \) URemoved [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service[/url].
4 \/ @) T8 `3 m- u4 `( [Removed [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service[/url].
2 _/ g5 y" t; s. |" K. @8 rRemoved [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service[/url].
+ Y& R3 _4 y$ Q5 d/ S0 _2 i7 V

! Z& n" z1 q( ?& B. J" i[root@compute01 ~]# cd /var/lib/ceph
9 }! K2 R/ b, M  K( c
[root@controller ~]# cd /var/lib/ceph
3 S9 l3 C0 O3 h! f' F& W1 w4 f[root@controller ceph]# rm -rf 1e87bca4-e7ce-11eb-aa90-0025908471d6/


' t6 ]3 w; D0 ~) ?[root@controller ceph]# ceph status  
4 r3 s/ u) P! l7 {  cluster:
0 Q# C5 M. D' J/ W3 M! c- S    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6
9 Q& i! v' O. x2 R% B& L    health: HEALTH_OK
! \  ~7 V* q' U( X  F1 g8 W
  services:
, c6 c0 R. D9 r$ q! v9 g    mon: 2 daemons, quorum controller,compute01 (age 84s)
1 S1 ]% j! c7 s* @1 z    mgr: compute01.getqhn(active, since 30s)
* {& ?. w6 P& _' r- C' E    osd: 3 osds: 3 up (since 14s), 3 in (since 4h)

( |9 a$ W, h& p* X& q+ ^  data:
2 g% b% I' {! d4 U- g; g    pools:   1 pools, 1 pgs
) {. B  O! x4 c' @6 a5 f    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 1.2 TiB / 1.2 TiB avail
) X' X' Z1 D$ l! Y! o: C% k    pgs:     1 active+clean
* h. P1 A5 X+ h. T8 G5 W9 U
问题竟然解决。
; t0 d4 }! g* h; G: b$ w& B3 l
9 D7 ]+ y/ E7 X. q8 o- ^# n5 [2 S

admin

admin 发表于 2021-7-25 20:59
; e5 l$ R, L. `; K. \[root@compute01 ~]# ceph status
' e7 Z0 F/ f( x, d9 H" ^, d+ X4 Y  cluster:
- q2 A1 N  C5 m" x1 F; F    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6

可能原因是因为生产了一个不一样的cluster_id导致信息不一致不对称引起。如果一个纯净的系统也出现这个问题，需要排除原因。