ceph 分布式存储15以上版本mon报错解决办法 使用cephadm处理问题

admin

ceph 分布式存储15以上版本mon报错解决办法


; h4 `8 f7 H2 c4 } ceph -s
" c7 T& }; L$ e& k' z1 F  cluster:
    id:     ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9
    health: HEALTH_WARN
/ _% t" C% w7 b7 @5 k8 N* }
7 \5 s0 Y. ^2 U( o5 P9 F$ K! T9 |            1/3 mons down, quorum gm268-2,gm268-3
: K8 ^/ f; |  S: z& E+ @$ K            1 pgs not deep-scrubbed in time
, M) ?3 @! Q% b+ ^$ A: I
0 }: |# l$ k, l" e; }& K* @
/ e; o% \7 |$ ?3 R# {0 a, A4 b解决过程：
$ \" ?9 e' q- l8 s查看：
0 \" B; Z' g2 y5 M4 |$ ceph orch host ls

HOST     ADDR     LABELS       STATUS  

, s" U  o. F) N& g. m* T9 [" hgm268-1  gm268-1  mon mgr mds         

" J8 i9 x* v" Q2 ]# T0 O; S. [; ygm268-2  gm268-2  mon mgr mds         
- t* y3 g& o! x5 f
gm268-3  gm268-3  mon mgr mds         

gm268-4  gm268-4                       
3 ~9 L8 T* q2 P3 `" v! ]6 u
删除mon 标签
" B8 P6 |1 b/ ^; d' d. z; } $ ceph orch host label rm gm268-1 mon

# z  V% G, v2 B: dRemoved label mon from host gm268-1
* k8 ~4 A  ^6 G1 q/ z0 ?. V
- Q1 @0 {( B4 D* ]6 f4 mroot@gm268-1 15:15:44 ~
: D( j- `- ?% ?& c$ c* @
$ ceph orch host ls

; N- z& b1 W' F% p9 ~( pHOST     ADDR     LABELS       STATUS  

gm268-1  gm268-1  mgr mds              
; P1 @6 {. M% o" }
gm268-2  gm268-2  mon mgr mds         


gm268-3  gm268-3  mon mgr mds             


% h4 J3 `0 B4 R. J
$ ceph orch apply mon label:mon
% O% ?+ `) }( z: M9 J

+ t$ n5 X* s1 cScheduled mon update...
0 u8 g/ d( `# j
/ E; l4 @3 z7 V9 |9 D1 q

这样mon就被剔除集群了。

% C; L7 U+ k6 d4 u
0 q" |1 L' I% c9 {+ U( a6 qmon不在报错：
& f$ g& d* N, c5 d8 ^. H3 h* c0 ^8 [ cluster:
    id:     ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9
    health: HEALTH_WARN
            failed to probe daemons or devices
; I6 J* ?3 K- m0 t* L8 H9 \            1 pgs not deep-scrubbed in time

问题解决。

admin

剔除mon之后，会少一个mon，需要添加回来。保证多个mon正常运行：

在标签上标记mon
! g8 |8 a) ~2 A; U; E  A
' K" y: n* Q' \- c$ ceph orch host ls
HOST     ADDR     LABELS       STATUS  
3 F( Z0 p: F! F) {" C: D3 O5 U( xgm268-1  gm268-1  mgr mds              
- F, H' x: \) R! C/ K# W5 y" g: hgm268-2  gm268-2  mon mgr mds         
gm268-3  gm268-3  mon mgr mds         
gm268-4  gm268-4                       
, ^  `: r6 |. Y% q/ W8 egm268-5  gm268-5                       
tp266-1  tp266-1                       
+ k7 y7 [( ~% P- `

$ ceph orch host label add gm268-1 mon
Added label mon to host gm268-1


0 b; M3 M: O4 X& X& B$ m3 C8 I$ ceph orch host ls
HOST     ADDR     LABELS       STATUS  
) x3 r5 c- f4 ?; y3 H, wgm268-1  gm268-1  mgr mds mon         
8 }9 D, `1 W" b6 z' Pgm268-2  gm268-2  mon mgr mds         
. a: |+ m& P2 Z+ mgm268-3  gm268-3  mon mgr mds         
1 b* @7 Z1 \2 D, S0 sgm268-4  gm268-4                       
gm268-5  gm268-5                       
tp266-1  tp266-1                       
1 x- X$ a" D9 S# {! P+ ~) o& F
0 \# W9 i, v% h- k/ X1 l4 |/ V
4 B5 w( w5 S! N9 F0 z! g$ ceph orch apply mon label:mon
& s1 H, k4 t. m8 f- R! gScheduled mon update...



  A& C! ]& W% n; H* r0 Y& D$ ceph orch host ls
HOST     ADDR     LABELS       STATUS  
gm268-1  gm268-1  mgr mds mon         
gm268-2  gm268-2  mon mgr mds         
: }( Z7 Q+ \7 c+ Wgm268-3  gm268-3  mon mgr mds         
gm268-4  gm268-4                       
. b! G. ]5 Y: b1 Q$ ?3 O" ugm268-5  gm268-5                       
tp266-1  tp266-1                       
, G: G8 o; T3 b3 o! ]% u. C; E

% _  |* c, G& Z/ _$ ceph orch daemon add mon gm268-1:10.12.1.2
Error EINVAL: name mon.gm268-1 already in use
' ?1 r5 x0 m+ n+ ^3 f
. U9 A5 l/ M: f. |, @
0 s0 v2 j! R& h6 n. R9 H( |: |& Y7 p$ ceph mon add gm268-1 10.12.1.2
3 x* [! b3 O8 a7 ], Uadding mon.gm268-1 at [v2:10.12.1.2:3300/0,v1:10.12.1.2:6789/0]
+ [4 _6 `! m3 o4 B. A

7 m; f' j$ W2 E; W: ]
$ ceph mon dump
2024-10-17T21:54:06.014+0800 7f4e6d2f7700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
- a# g7 C4 n- Y9 n- {2024-10-17T21:54:06.016+0800 7f4e6caf6700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
: g" ^  w0 k6 w4 y, V# p! B+ W3 Wdumped monmap epoch 11
epoch 11
: d1 C, a- G9 |7 o+ X0 v7 cfsid ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9
last_changed 2024-10-17T13:54:00.781176+0000
1 h+ O; V( {6 T) ]: r2 |created 2023-01-08T11:23:03.268285+0000
' Z6 h9 h/ X- G: R5 d$ ]min_mon_release 15 (octopus)
1 d. w) M* Z: x+ V+ t) p0: [v2:10.12.2.2:3300/0,v1:10.12.2.2:6789/0] mon.gm268-2
1: [v2:10.12.3.2:3300/0,v1:10.12.3.2:6789/0] mon.gm268-3
/ y& D5 Z2 F' A, N2: [v2:10.12.1.2:3300/0,v1:10.12.1.2:6789/0] mon.gm268-1
, Y: v& O- R: [  w: {

0 f4 J: n' P& m# Y8 Croot@gm268-1 21:54:06 ~
$
  K+ l5 a7 |( f- e; }  x7 [9 w. W* o% |
/ S, v1 B. ~* F* \
' H/ B0 J6 f- c


8 C& y8 G( \; a4 M3 a$ z

admin

cephadm bootstrap --mon-ip 10.12.1.2 --fsid ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9 --allow-overwrite
9 ]+ f; j6 _: l+ W1 UVerifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
1 Y6 n* ]& n' r; B# l7 ~/ D/ RUnit chronyd.service is enabled and running
4 G: g  E' f, J  Z! J" yRepeating the final host check...
podman|docker (/usr/bin/podman) is present
! b1 A8 G& ?) Xsystemctl is present
, x$ l6 G9 l2 l3 Flvcreate is present
Unit chronyd.service is enabled and running
2 R! h! q8 b6 K+ IHost looks OK
Cluster fsid: ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9
Verifying IP 10.12.1.2 port 3300 ...
8 ~4 Y- M4 ~: DVerifying IP 10.12.1.2 port 6789 ...
Mon IP 10.12.1.2 is in CIDR network 10.12.0.0/15
$ L! ?0 ^8 Z/ W  J  K( ^" A  sPulling container image quay.io/ceph/ceph:v15...
8 m- T$ J9 R% R3 M) w: [
: D3 p3 |  n( D! _! ?6 z

admin

后来通过删除/var/lib/ceph
' D- V; ]$ _, J# j- f% _5 Urm -rf  /var/lib/ceph/ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9/mon.gm268-1/
3 \7 U4 Q, G+ q, q9 M
5 V& V4 v" {6 ^再重新添加mon既可以：

  ceph orch host label add gm268-1 mon
   

  X4 U4 p( L& H0 }9 O   ceph orch  apply mon  label:mon

  ceph orch host ls
: ~+ Y6 d( C+ h- B) T9 X% v
" n' N; W/ C: a0 ?, T; y
$ ceph mon add gm268-1 10.12.1.2
& g+ X* |/ Y9 Vadding mon.gm268-1 at [v2:10.12.1.2:3300/0,v1:10.12.1.2:6789/0]

( h: M* B; J  p8 z3 Q
3 U$ f+ V/ a, H- J$ ceph mon dump
$ |, a: w* F4 ^2 _2024-10-17T21:54:06.014+0800 7f4e6d2f7700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
, J( |2 v; x! U( B/ ]) q2024-10-17T21:54:06.016+0800 7f4e6caf6700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
dumped monmap epoch 11
5 @) \$ X+ E8 ~) d& J8 ^epoch 11
" g% ?* H# ^( v& q: zfsid ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9
last_changed 2024-10-17T13:54:00.781176+0000
created 2023-01-08T11:23:03.268285+0000
min_mon_release 15 (octopus)
( ^& e9 q4 L/ x1 B9 ^" b2 @8 e$ K9 p0: [v2:10.12.2.2:3300/0,v1:10.12.2.2:6789/0] mon.gm268-2
1: [v2:10.12.3.2:3300/0,v1:10.12.3.2:6789/0] mon.gm268-3
: N3 k# A/ K  Z6 a2: [v2:10.12.1.2:3300/0,v1:10.12.1.2:6789/0] mon.gm268-1
: V% N% q4 z4 O8 l
1 k5 i  c: s  z7 O
. J8 }* @) @+ T+ @" z# n$
完成添加mon节点。

admin

admin 发表于 2024-10-17 16:09
# A$ U# S) }3 w1 x: Ccephadm bootstrap --mon-ip 10.12.1.2 --fsid ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9 --allow-overwrite
...

7 e* ]/ Z) {' b' z3 y* l; U( gGenerating a dashboard self-signed certificate...
Creating initial admin user...
5 j+ c% B4 t+ Z: RFetching dashboard port number...
6 L! B* ^; Q8 C2 `+ w% A8 R: S* b9 nCeph Dashboard is now available at:

             URL: https://gm268-1:8443/
            User: admin
        Password: j2ixndfanu

4 J$ q3 d1 L* I, Z# kYou can access the Ceph CLI with:

        sudo /usr/sbin/cephadm shell --fsid ce68aab8-8f46-11ed-88c0-ac1f6b3a30b9 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring

7 X7 L( ?6 m2 ]! pPlease consider enabling telemetry to help improve Ceph:

        ceph telemetry on

& b6 O4 S% L/ N/ \& R: b; iFor more information see:
7 \$ A: K* A, a# V' ?4 }' |" ^
        https://docs.ceph.com/docs/master/mgr/telemetry/
# R+ F2 U  h, `
Bootstrap complete.
7 q# f& V! I9 \0 ?& J1 W  B