log_channel(cluster) log [ERR].. soid 1:0a2506d3:::rbd_data....head candidate ha

admin

ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error

blk_update_request: I/O error, dev sdi, sector 72908960
6 l2 u$ n- L2 [  I6 j

) I4 F0 g9 r6 o- vJul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 Sense Key : Medium Error [current]
( g2 `& f0 \  |" q, G+ \Jul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 Add. Sense: No additional sense information
( X* T9 x9 r, |% L+ OJul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 10:32:18 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
) H/ T8 u7 g6 i+ ^5 JJul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.121742 7f0b5a5ef700 -1 osd.67 790 shutdown
8 y9 x0 Z/ R  f7 P1 e0 k) _; [$ gJul 16 10:32:19 cn01 systemd-logind: New session 103926 of user root.
Jul 16 10:32:19 cn01 systemd: Started Session 103926 of user root.
& n. X5 T; g* kJul 16 10:32:19 cn01 systemd: Starting Session 103926 of user root.
Jul 16 10:32:22 cn01 systemd: Stopped Ceph object storage daemon.


+ Q9 g" C: k) M% G+ d
通过日志分析，应该是磁盘损坏了。
" q8 C, m3 _  f0 z更换磁盘。

admin

Jul 16 10:35:41 cn01 systemd: Starting Ceph object storage daemon...
, \6 ?; ?5 r3 u( b; K+ C6 |Jul 16 10:35:41 cn01 systemd: Started Ceph object storage daemon.
Jul 16 10:35:42 cn01 ceph-osd: starting osd.67 at :/0 osd_data /var/lib/ceph/osd/ceph-67 /var/lib/ceph/osd/ceph-67/journal
5 n1 Y/ c% q! G% P3 I  hJul 16 10:35:42 cn01 ceph-osd: 2023-07-16 10:35:42.085939 7fe6fa918f80 -1 leveldb: Compacting leveldb store...
% ~2 F* y" `# ~+ h1 |6 iJul 16 10:35:43 cn01 ceph-osd: 2023-07-16 10:35:43.212904 7fe6fa918f80 -1 leveldb: Finished compacting leveldb store
1 p  v/ n4 _/ f: u0 @- aJul 16 10:35:47 cn01 ceph-osd: 2023-07-16 10:35:47.177395 7fe6fa918f80 -1 osd.67 790 log_to_monitors {default=true}
: l0 n8 V7 m9 X: Z8 U1 [

admin

Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
: b7 S5 l% N/ J, C1 B. iJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Sense Key : Medium Error [current]
# K7 M1 X- u, d% m0 TJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Add. Sense: No additional sense information
7 v) D' [7 o; r2 b7 E8 fJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
0 P- |$ v) j$ t  lJul 16 11:16:55 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 11:16:56 cn01 ceph-osd: 2023-07-16 11:16:56.623469 7fe6a75bc700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
1 t; E7 ]% i( g, }5 J+ v/ z) A) c5 j3 VJul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12266 (742792048s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
; `  n( d% I2 [2 xJul 16 11:17:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
! k% M/ y1 q5 c: Z) [% iJul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12267 (742792048s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
7 N, q. v# ?" T& v* HJul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12270 (742792055s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12271 (742792055s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
( Q+ h, ~$ {- y! e9 P, kJul 16 11:18:02 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12274 (742792062s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12275 (742792062s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:09 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12278 (742792068s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12279 (742792069s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:16 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
# J) N, t9 v8 BJul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12282 (742792075s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
5 u4 w- w# n, h! n5 u; {/ T- nJul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12283 (742792075s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:22 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
. u% ^6 H, r% }* p1 s! DJul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12286 (742792082s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
' t* k5 D& I, H* `' U( x, `Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
0 ~* A$ k7 t8 ~. A7 c, uJul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Sense Key : Medium Error [current]
: `/ D) u7 x& Q6 ~! p9 fJul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Add. Sense: No additional sense information
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 11:18:29 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
6 a( Y1 a+ y/ m9 P/ @/ Z, v/ `Jul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12287 (742792082s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
4 ]6 A( E# z2 AJul 16 11:18:36 cn01 kernel: megaraid_sas 0000:18:00.0: 12290 (742792089s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:36 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
+ ]( c, [2 ]% B9 |/ ~" W