log_channel(cluster) log [ERR].. soid 1:0a2506d3:::rbd_data....head candidate ha

admin

ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
& J. [, R! w. V' p" H
0 z& K# w! S9 [4 M/ j* ^1 xblk_update_request: I/O error, dev sdi, sector 72908960
$ W# s* ]- ], S( |4 }- P2 K
/ X  Y3 F. C/ T4 V8 j* i5 `
. x* p1 }% M4 s% ~8 K7 LJul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 Sense Key : Medium Error [current]
Jul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 Add. Sense: No additional sense information
Jul 16 10:32:18 cn01 kernel: sd 0:2:8:0: [sdi] tag#1 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 10:32:18 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
3 T" M7 v) q  n. nJul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.121742 7f0b5a5ef700 -1 osd.67 790 shutdown
/ p# ^( }0 U( h5 o& {  rJul 16 10:32:19 cn01 systemd-logind: New session 103926 of user root.
Jul 16 10:32:19 cn01 systemd: Started Session 103926 of user root.
Jul 16 10:32:19 cn01 systemd: Starting Session 103926 of user root.
Jul 16 10:32:22 cn01 systemd: Stopped Ceph object storage daemon.

! y; Q- E( ~  ~

通过日志分析，应该是磁盘损坏了。
更换磁盘。
- U$ E+ j( ?8 X4 a

admin

Jul 16 10:35:41 cn01 systemd: Starting Ceph object storage daemon...
Jul 16 10:35:41 cn01 systemd: Started Ceph object storage daemon.
Jul 16 10:35:42 cn01 ceph-osd: starting osd.67 at :/0 osd_data /var/lib/ceph/osd/ceph-67 /var/lib/ceph/osd/ceph-67/journal
1 {3 f) g* a: {0 Q5 rJul 16 10:35:42 cn01 ceph-osd: 2023-07-16 10:35:42.085939 7fe6fa918f80 -1 leveldb: Compacting leveldb store...
Jul 16 10:35:43 cn01 ceph-osd: 2023-07-16 10:35:43.212904 7fe6fa918f80 -1 leveldb: Finished compacting leveldb store
- Z( _8 y8 |. |' K$ G- ^" oJul 16 10:35:47 cn01 ceph-osd: 2023-07-16 10:35:47.177395 7fe6fa918f80 -1 osd.67 790 log_to_monitors {default=true}

admin

Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
5 B8 j& s" \7 YJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Sense Key : Medium Error [current]
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Add. Sense: No additional sense information
) M, N7 j  Q% |" i% L% DJul 16 11:16:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 11:16:55 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 11:16:56 cn01 ceph-osd: 2023-07-16 11:16:56.623469 7fe6a75bc700 -1 log_channel(cluster) log [ERR] : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
Jul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12266 (742792048s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:17:55 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12267 (742792048s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12270 (742792055s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12271 (742792055s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
; m/ _9 T8 w+ d$ C# J. ~Jul 16 11:18:02 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
7 W: A0 M, l6 Q, `( b$ U) B( k( iJul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12274 (742792062s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
) E5 ?# r) A8 Y9 k0 D& FJul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12275 (742792062s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:09 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12278 (742792068s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
, ?9 X5 X+ }5 m& ]+ b* r- pJul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12279 (742792069s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:16 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12282 (742792075s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12283 (742792075s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:22 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
1 Q# y4 T" X0 s  D, F3 W) `! rJul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12286 (742792082s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
1 a8 b  E0 _" x- r8 V3 fJul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Sense Key : Medium Error [current]
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 Add. Sense: No additional sense information
. {4 a" U+ D7 d/ v; x5 a. \Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
( i6 k' r* z/ k! TJul 16 11:18:29 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
8 ?0 w; Y  v" B  a; p: [Jul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12287 (742792082s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
0 X! D% c7 ?5 g, g1 z* @Jul 16 11:18:36 cn01 kernel: megaraid_sas 0000:18:00.0: 12290 (742792089s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
+ I6 G( m: l( ]Jul 16 11:18:36 cn01 kernel: sd 0:2:8:0: [sdi] tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0