ceph osd down message日志

admin

Jul 20 12:47:56 compute03 ceph-osd: 2021-07-20 12:47:56.933 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
Jul 20 12:47:57 compute03 ceph-osd: 2021-07-20 12:47:57.950 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
2 l( c4 q/ t8 h: s* BJul 20 12:47:58 compute03 ceph-osd: 2021-07-20 12:47:58.986 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
9 E: J9 Q; v2 q# NJul 20 12:47:59 compute03 ceph-osd: 2021-07-20 12:47:59.978 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
Jul 20 12:48:00 compute03 ceph-osd: 2021-07-20 12:48:00.955 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
Jul 20 12:48:01 compute03 ceph-osd: 2021-07-20 12:48:01.925 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
; o$ M3 ]9 k9 PJul 20 12:48:02 compute03 ceph-osd: 2021-07-20 12:48:02.949 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)
Jul 20 12:48:03 compute03 ceph-osd: 2021-07-20 12:48:03.971 7f87af881700 -1 osd.2 53 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 12:47:21.171167 (oldest deadline 2021-07-20 12:47:41.171167)

( P& o; {6 C# z) Z( C
通过stop NetworkMange服务，和firewalld服务：
[root@compute03 ceph]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
; ]3 n0 T: p3 Z7 T* `& u   Active: inactive (dead)
     Docs: man:firewalld(1)
, a# s- g' S8 K1 p5 r- L- j1 S0 O[root@compute03 ceph]# systemctl disable firewalld.service
[root@compute03 ceph]# systemctl status firewalld.service
% m) o0 ~' X  q0 x. y& s● firewalld.service - firewalld - dynamic firewall daemon
8 A+ \" [  Q1 c) Y   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
8 I1 F# \6 M; \   Active: inactive (dead)
0 L; l1 s9 e. g( g+ @  ~, V5 v6 H     Docs: man:firewalld(1)
4 F3 z  D: |4 N) G4 s' d% z[root@compute03 ceph]#
0 B6 U( I& S( A2 o, b: `" p
& s9 y. C$ q0 W( _/ k[root@compute02 ceph]# systemctl disable firewalld.service
% H  ^1 g+ U. Z  ][root@compute02 ceph]# systemctl stop firewalld.service
[root@compute02 ceph]# ceph osd tree
ID CLASS WEIGHT TYPE NAME    STATUS REWEIGHT PRI-AFF
; Y# e' i" v, G6 W: D-1            0 root default                        
) _' w# j5 |: O6 y. W( y 0   hdd      0 osd.0            up  1.00000 1.00000
1   hdd      0 osd.1            up  1.00000 1.00000
3 f. s4 q: }# h. B8 @# G5 { 2   hdd      0 osd.2            up  1.00000 1.00000
5 y1 v1 H6 q: o! V# F; w

admin

Jul 20 12:50:11 compute03 ceph-osd: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
! O% \' Z: _! G& S5 |. qJul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
* v( q/ r- j% X4 i. `6 LJul 20 12:50:12 compute03 ceph-osd: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
Jul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
Jul 20 12:50:13 compute03 ceph-osd: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
8 z6 ]& a3 u! \! l4 r& T% kJul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 since back 2021-07-20 12:49:20.232233 front 2021-07-20 12:50:10.936408 (oldest deadline 2021-07-20 12:49:44.931804)
! R, g' a/ c: f: t: _2 \Jul 20 12:50:14 compute03 ceph-osd: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 12:49:33.632551 (oldest deadline 2021-07-20 12:49:53.632551)
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***
Jul 20 12:50:15 compute03 ceph-osd: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***

admin

● ceph-osd@2.service - Ceph object storage daemon osd.2
: N& t5 x2 F9 B$ T; n   Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: disabled)
7 T& ~; j; m0 X: T   Active: inactive (dead) since Tue 2021-07-20 12:50:15 CST; 6min ago
  Process: 4680 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS)
Main PID: 4680 (code=exited, status=0/SUCCESS)

, P1 z7 V7 l/ r) \3 v* e  ]. u. i) ZJul 20 12:50:11 compute03 ceph-osd[4680]: 2021-07-20 12:50:11.806 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
+ d6 T9 l$ ?7 I3 r& }  D$ KJul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
Jul 20 12:50:12 compute03 ceph-osd[4680]: 2021-07-20 12:50:12.775 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
Jul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
9 z& O3 |- M3 HJul 20 12:50:13 compute03 ceph-osd[4680]: 2021-07-20 12:50:13.739 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
7 y: n; j( D8 R2 o+ `# RJul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.75:6802 osd.0 sinc...44.931804)
" [: Y% Q6 j7 cJul 20 12:50:14 compute03 ceph-osd[4680]: 2021-07-20 12:50:14.704 7f87af881700 -1 osd.2 60 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever...53.632551)
( d6 S' b$ k# M; QJul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthr...) ) UID: 0
Jul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Got signal Interrupt ***
$ v2 A+ x' _: q0 k7 n! G* k& ~1 j4 YJul 20 12:50:15 compute03 ceph-osd[4680]: 2021-07-20 12:50:15.650 7f87b1903700 -1 osd.2 61 *** Immediate shutdown (osd_fast_shutdown=true) ***
Hint: Some lines were ellipsized, use -l to show in full.

admin

Jul 20 13:03:22 compute03 ceph-osd: 2021-07-20 13:03:22.222 7f308763f700 -1 osd.2 79 heartbeat_check: no reply from 192.168.0.77:6804 osd.1 ever on either front or back, first ping sent 2021-07-20 13:02:43.327130 (oldest deadline 2021-07-20 13:03:03.327130)
8 q' W- F; L" K/ ]0 I! s5 JJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 received  signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
4 s) v! {1 R1 B( EJul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Got signal Interrupt ***
Jul 20 13:03:23 compute03 ceph-osd: 2021-07-20 13:03:23.143 7f30896c1700 -1 osd.2 80 *** Immediate shutdown (osd_fast_shutdown=true) ***

admin

back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)
Jul 20 13:35:13 compute03 ceph-osd: 2021-07-20 13:35:13.237 7f3703776700 -1 osd.2 87 heartbeat_check: no reply from 192.168.0.77:6802 osd.1 ever on either front or back, first ping sent 2021-07-20 13:34:07.693512 (oldest deadline 2021-07-20 13:34:27.693512)
^C
9 R: E  E/ v% J3 z7 I[root@compute03 ~]# service iptables stop
7 Z3 E6 p6 @8 Q3 y) E0 Z7 `7 i6 xRedirecting to /bin/systemctl stop iptables.service
1 @4 r! O8 o8 W4 N1 L" z) v3 UFailed to stop iptables.service: Unit iptables.service not loaded.
[root@compute03 ~]# getenforce
Permissive

admin

关闭selinux
永久性关闭（这样需要重启服务器后生效）
# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

5 `/ w7 Q! C" z/ @4 b
4 G9 g' H' D# w- H4 S/ c sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
4 A9 I; N- t# F( l

admin

最后发现是有张网卡的ip地址配置成一模一样的，造成ip地址冲突。排查网络ip问题，也是问题原因。