log_channel(cluster) log [ERR].. soid 1:0a2506d3:::rbd_data....head candidate ha
ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read errorblk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 10:32:18 cn01 kernel: sd 0:2:8:0: tag#1 Sense Key : Medium Error
Jul 16 10:32:18 cn01 kernel: sd 0:2:8:0: tag#1 Add. Sense: No additional sense information
Jul 16 10:32:18 cn01 kernel: sd 0:2:8:0: tag#1 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 10:32:18 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.107150 7f0b659fd700 -1 log_channel(cluster) log : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
Jul 16 10:32:19 cn01 ceph-osd: 2023-07-16 10:32:19.121742 7f0b5a5ef700 -1 osd.67 790 shutdown
Jul 16 10:32:19 cn01 systemd-logind: New session 103926 of user root.
Jul 16 10:32:19 cn01 systemd: Started Session 103926 of user root.
Jul 16 10:32:19 cn01 systemd: Starting Session 103926 of user root.
Jul 16 10:32:22 cn01 systemd: Stopped Ceph object storage daemon.
通过日志分析，应该是磁盘损坏了。
更换磁盘。
Jul 16 10:35:41 cn01 systemd: Starting Ceph object storage daemon...
Jul 16 10:35:41 cn01 systemd: Started Ceph object storage daemon.
Jul 16 10:35:42 cn01 ceph-osd: starting osd.67 at :/0 osd_data /var/lib/ceph/osd/ceph-67 /var/lib/ceph/osd/ceph-67/journal
Jul 16 10:35:42 cn01 ceph-osd: 2023-07-16 10:35:42.085939 7fe6fa918f80 -1 leveldb: Compacting leveldb store...
Jul 16 10:35:43 cn01 ceph-osd: 2023-07-16 10:35:43.212904 7fe6fa918f80 -1 leveldb: Finished compacting leveldb store
Jul 16 10:35:47 cn01 ceph-osd: 2023-07-16 10:35:47.177395 7fe6fa918f80 -1 osd.67 790 log_to_monitors {default=true}
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: tag#0 Sense Key : Medium Error
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: tag#0 Add. Sense: No additional sense information
Jul 16 11:16:55 cn01 kernel: sd 0:2:8:0: tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 11:16:55 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 11:16:56 cn01 ceph-osd: 2023-07-16 11:16:56.623469 7fe6a75bc700 -1 log_channel(cluster) log : 1.450 shard 67: soid 1:0a2506d3:::rbd_data.d0cee7f011e19.0000000000002c80:head candidate had a read error
Jul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12266 (742792048s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:17:55 cn01 kernel: sd 0:2:8:0: tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:17:55 cn01 kernel: megaraid_sas 0000:18:00.0: 12267 (742792048s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12270 (742792055s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:02 cn01 kernel: megaraid_sas 0000:18:00.0: 12271 (742792055s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:02 cn01 kernel: sd 0:2:8:0: tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12274 (742792062s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:09 cn01 kernel: megaraid_sas 0000:18:00.0: 12275 (742792062s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:09 cn01 kernel: sd 0:2:8:0: tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12278 (742792068s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:16 cn01 kernel: megaraid_sas 0000:18:00.0: 12279 (742792069s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:16 cn01 kernel: sd 0:2:8:0: tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12282 (742792075s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:22 cn01 kernel: megaraid_sas 0000:18:00.0: 12283 (742792075s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:22 cn01 kernel: sd 0:2:8:0: tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12286 (742792082s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: tag#0 Sense Key : Medium Error
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: tag#0 Add. Sense: No additional sense information
Jul 16 11:18:29 cn01 kernel: sd 0:2:8:0: tag#0 CDB: Read(16) 88 00 00 00 00 00 04 58 80 a0 00 00 00 08 00 00
Jul 16 11:18:29 cn01 kernel: blk_update_request: I/O error, dev sdi, sector 72908960
Jul 16 11:18:29 cn01 kernel: megaraid_sas 0000:18:00.0: 12287 (742792082s/0x0001/FATAL) - Bad block table on VD 08/8 is full; unable to log block 45880a7 (on PD 09(e0x20/s9) at 45880a7)
Jul 16 11:18:36 cn01 kernel: megaraid_sas 0000:18:00.0: 12290 (742792089s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 09(e0x20/s9) at 45880a7
Jul 16 11:18:36 cn01 kernel: sd 0:2:8:0: tag#0 BRCM Debug mfi stat 0x2d, data lenrequested/completed 0x1000/0x0
页:
[1]