HEALTH_WARN 1 failed cephadm daemon(s)

admin

HEALTH_WARN 1 failed cephadm daemon(s)
ceph health detail
: }0 Y$ w, F3 t" ?; }5 fHEALTH_WARN 2 failed cephadm daemon(s)
' n) L# m. i# a  ^* W[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
    daemon alertmanager.controller on controller is in error state
    daemon grafana.controller on controller is in error state
2 t2 d# `" _5 Y8 m/ `4 W( H7 W

admin

经过排查，应该是系统层间安装过ceph集群，没有清理干净。新版本还不知道怎么全部清除，还在测试中。

admin

[root@compute01 ~]# ceph status
0 k7 L% D- R* l4 t# z+ X/ G# E  cluster:
  N( \( ~) n; b    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6
    health: HEALTH_WARN
            2 failed cephadm daemon(s)
6 D. W; k+ `+ z! O- Q' k( b            clock skew detected on mon.compute01
6 z  D* s( u+ k% ^% J
  services:
( ]/ \" H$ m0 a: I% x1 u# K    mon: 2 daemons, quorum controller,compute01 (age 3h)
    mgr: compute01.getqhn(active, since 3h), standbys: controller.kxfttd
9 j$ f5 b4 ]" t: K9 Y% J    osd: 3 osds: 3 up (since 3h), 3 in (since 3h)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 1.2 TiB / 1.2 TiB avail
$ Q! m9 B7 |8 X3 S7 y4 @4 u, l, c1 a    pgs:     1 active+clean


$ e( z$ ~5 b7 N. m9 D& G[root@compute01 ~]# systemctl status ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service  
[root@compute01 ~]# systemctl status ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6
1 \9 _6 \" |+ h* {: sceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service
ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target
: {6 j3 P  K* m& |) I9 Rceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service  
, a! v: N8 Q0 S+ o7 n; X- l8 T
1 X9 u. T/ `4 B: G( M3 o


, o' E6 X- C( V& y1 e3 I3 z. h[root@compute01 ~]# systemctl disable ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service          ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service     ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service
Removed [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@crash.compute01.service[/url].
Removed [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@mgr.compute01.bunbzp.service[/url].
: J( U: e# @" w9 C3 eRemoved [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@node-exporter.compute01.service[/url].
5 ]% h% @* y* B# n: B" iRemoved [url=]/etc/systemd/system/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6.target.wants/ceph-1e87bca4-e7ce-11eb-aa90-0025908471d6@osd.0.service[/url].
6 B0 ]* V" C/ c5 h. V

[root@compute01 ~]# cd /var/lib/ceph
, j5 ]* Y! P. I, o( o
[root@controller ~]# cd /var/lib/ceph
9 @  k& p1 \( j: x1 G[root@controller ceph]# rm -rf 1e87bca4-e7ce-11eb-aa90-0025908471d6/
% s( ~$ _7 }1 b/ y% j+ P7 B$ s: x

[root@controller ceph]# ceph status  
  cluster:
    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6
    health: HEALTH_OK

  services:
    mon: 2 daemons, quorum controller,compute01 (age 84s)
    mgr: compute01.getqhn(active, since 30s)
    osd: 3 osds: 3 up (since 14s), 3 in (since 4h)
- V; P# \" G4 g) h
/ Q3 E, h) U9 |3 G, d  data:
    pools:   1 pools, 1 pgs
; `( C$ ?/ {8 l( A9 m9 b8 E" k! X    objects: 0 objects, 0 B
4 M% _; A, M% ^; ^0 D) t    usage:   3.0 GiB used, 1.2 TiB / 1.2 TiB avail
    pgs:     1 active+clean

问题竟然解决。

admin

admin 发表于 2021-7-25 20:59
* }+ S# k1 e/ P3 F! O- ~+ }[root@compute01 ~]# ceph status
9 f( N. A/ |8 n( K% ?  cluster:
    id:     4c1f752a-ed1a-11eb-8ce5-0025908471d6

1 g/ C  [% n: v  a# E. `可能原因是因为生产了一个不一样的cluster_id导致信息不一致不对称引起。如果一个纯净的系统也出现这个问题，需要排除原因。