ceph osd 报错 osd.6 179 unable to obtain rotating service keys; retrying
1 osd.6 179 unable to obtain rotating service keys; retryingsystemctl status ceph-osd@6.service
● ceph-osd@6.service - Ceph object storage daemon osd.6
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
Active: active (running) since Wed 2024-01-17 16:09:24 CST; 2min 7s ago
Process: 4244 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
Main PID: 4254 (ceph-osd)
Tasks: 62
Memory: 26.4M
CPU: 1.689s
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
└─4254 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
Jan 17 16:09:24 compute02 systemd: Starting Ceph object storage daemon osd.6...
Jan 17 16:09:24 compute02 systemd: Started Ceph object storage daemon osd.6.
Jan 17 16:09:28 compute02 ceph-osd: 2024-01-17T16:09:28.102+0800 7f5fedb67800 -1 osd.6 179 log_to_monitors true
Jan 17 16:09:58 compute02 ceph-osd: 2024-01-17T16:09:58.118+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:10:28 compute02 ceph-osd: 2024-01-17T16:10:28.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:10:58 compute02 ceph-osd: 2024-01-17T16:10:58.119+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:11:28 compute02 ceph-osd: 2024-01-17T16:11:28.116+0800 7f5fedb67800 -1 osd.6 179 unable to obtain rotating service keys; retrying
导致问题：ceph状态异常：
root@controller:~# ceph osd tree
IDCLASSWEIGHT TYPE NAME STATUSREWEIGHTPRI-AFF
-1 9.00000root default
-4 3.00000 host compute01
3 hdd1.00000 osd.3 up 1.000001.00000
4 hdd1.00000 osd.4 up 1.000001.00000
5 hdd1.00000 osd.5 up 1.000001.00000
-5 3.00000 host compute02
6 hdd1.00000 osd.6 down 01.00000
7 hdd1.00000 osd.7 down 01.00000
8 hdd1.00000 osd.8 up 1.000001.00000
-3 3.00000 host controller
0 hdd1.00000 osd.0 up 1.000001.00000
1 hdd1.00000 osd.1 up 1.000001.00000
2 hdd1.00000 osd.2 up 1.000001.00000
这种问题是因为时间不同步导致的：
调整时间配置chrony
一般是检查/etc/chrony/chrony.conf文件中同步的配置。
pool controller iburst
另一个是时间服务器的节点中controller节点的allow配置有没有放通网段的配置：
vim /etc/chrony/chrony.conf
allow 192.168.8.0/24
所有的都重启下chrony的服务。
root@controller:~# systemctl restart chrony.service
计算节点检查：
chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns
root@compute02:~# chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.8.65 0 8 0 - +0ns[ +0ns] +/- 0ns
root@compute02:~# systemctl enable chrony.service
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable chrony
计算节点重启chrony的服务：
root@compute02:~# systemctl restart chrony.service
检查状态：
root@compute02:~# chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.8.65 2 6 3 1+8087ms[+8087ms] +/- 13ms
root@compute02:~# systemctl status ceph-osd@6.service
● ceph-osd@6.service - Ceph object storage daemon osd.6
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
Active: active (running) since Wed 2024-01-17 16:15:08 CST; 10min ago
Main PID: 6049 (ceph-osd)
Tasks: 62
Memory: 84.6M
CPU: 15.598s
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
└─6049 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
Jan 17 16:15:08 compute02 systemd: Starting Ceph object storage daemon osd.6...
Jan 17 16:15:08 compute02 systemd: Started Ceph object storage daemon osd.6.
Jan 17 16:15:11 compute02 ceph-osd: 2024-01-17T16:15:11.868+0800 7fee327c7800 -1 osd.6 179 log_to_monitors true
Jan 17 16:15:41 compute02 ceph-osd: 2024-01-17T16:15:41.885+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:16:11 compute02 ceph-osd: 2024-01-17T16:16:11.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:16:41 compute02 ceph-osd: 2024-01-17T16:16:41.886+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:17:11 compute02 ceph-osd: 2024-01-17T16:17:11.887+0800 7fee327c7800 -1 osd.6 179 unable to obtain rotating service keys; retrying
Jan 17 16:17:14 compute02 ceph-osd: 2024-01-17T16:17:14.273+0800 7fee2855f640 -1 osd.6 179 set_numa_affinity unable to identify public interf>
root@compute02:~#
root@compute02:~# systemctl restart ceph-osd@6.service
root@compute02:~# systemctl status ceph-osd@6.service
● ceph-osd@6.service - Ceph object storage daemon osd.6
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled-runtime; vendor preset: enabled)
Active: active (running) since Wed 2024-01-17 16:25:44 CST; 1s ago
Process: 7784 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 6 (code=exited, status=0/SUCCESS)
Main PID: 7795 (ceph-osd)
Tasks: 14
Memory: 11.7M
CPU: 339ms
CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd@6.service
└─7795 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
Jan 17 16:25:44 compute02 systemd: Starting Ceph object storage daemon osd.6...
Jan 17 16:25:44 compute02 systemd: Started Ceph object storage daemon osd.6.
检查日志：
tail -f /var/log/ceph/ceph-osd.6.log
2024-01-17T16:25:50.363+0800 7f5916bfe6401 osd.6 pg_epoch: 224 pg r=2 lpr=224 pi= start_peering_interval up -> , acting -> , acting_primary 3 -> 3, up_primary 3 -> 3, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f59153fb6401 osd.6 pg_epoch: 224 pg r=2 lpr=224 pi= start_peering_interval up -> , acting -> , acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f5916bfe6401 osd.6 pg_epoch: 224 pg r=2 lpr=224 pi= state<Start>: transitioning to Stray
2024-01-17T16:25:50.363+0800 7f59153fb6401 osd.6 pg_epoch: 224 pg r=2 lpr=224 pi= state<Start>: transitioning to Stray
2024-01-17T16:25:50.363+0800 7f5916bfe6401 osd.6 pg_epoch: 224 pg r=2 lpr=224 pi= start_peering_interval up -> , acting -> , acting_primary 5 -> 5, up_primary 5 -> 5, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f5916bfe6401 osd.6 pg_epoch: 224 pg r=2 lpr=224 pi= state<Start>: transitioning to Stray
2024-01-17T16:25:50.363+0800 7f59153fb6401 osd.6 pg_epoch: 224 pg r=2 lpr=224 pi= start_peering_interval up -> , acting -> , acting_primary 2 -> 2, up_primary 2 -> 2, role -1 -> 2, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f59153fb6401 osd.6 pg_epoch: 224 pg r=2 lpr=224 pi= state<Start>: transitioning to Stray
2024-01-17T16:25:50.363+0800 7f5916bfe6401 osd.6 pg_epoch: 224 pg r=0 lpr=224 pi= start_peering_interval up -> , acting -> , acting_primary 4 -> 6, up_primary 4 -> 6, role -1 -> 0, features acting 4540138320759226367 upacting 4540138320759226367
2024-01-17T16:25:50.363+0800 7f5916bfe6401 osd.6 pg_epoch: 224 pg r=0 lpr=224 pi= state<Start>: transitioning to Primary
恢复正常。
页:
[1]