failed to probe daemons or devices 问题ceph -s出现的错误

admin

[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host private-registry.example.com `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -e NODE_NAME=private-registry.example.com -e CEPH_USE_RANDOM_NONCE=1 ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -c %u %g /var/lib/ceph
stat: stderr Trying to pull ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01...
2 m( b. x/ `4 a4 estat: stderr Error: initializing source docker://ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01: reading manifest sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 in ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8: unauthorized: authentication required
) J- E( a( \8 u+ yHow to eliminate this warning?
# Q, ?( D5 t+ j: p; O2 d1 vResolution
Login to the cephadm shell on the lead monitor node.
# K% U7 Y9 F( |4 B7 ^  j* f0 b
( r3 O4 ?4 P) }; f8 N( w% cRaw
# cephadm shell
8 b5 v4 f! S* ?" d4 S' f7 dLog in manually to the custom registry on all the new hosts simultaneously:

Raw
* ~2 q0 P3 Q9 C1 j) v# r% c4 F# ceph cephadm registry-login --registry-url <CUSTOM_REGISTRY_NAME>  --registry_username <REGISTRY_USERNAME> --registry_password <PASSWORD>
: m. @/ F# X- HExecuting the aforementioned command will produce a podman-auth.json file in the /etc/ceph directory that contains the custom registry's authentication details.
0 Z4 B) K- M6 x) \; u$ P9 D
+ G+ }, ]4 A/ ~0 oWait 3-5 minutes to see if the daemon starts up.
) e6 M+ _. u& ?4 M  C5 ^0 q) r
# R$ a8 k& W9 E6 U% O+ FRaw
; g4 Z+ B# R! g# watch ceph orch ls
3 k- P" M; p9 q3 ], x) \/ EOPTIONAL: Restart the daemon if it is still not in running state.

6 F. q5 b5 @# v6 U3 NRaw
# ceph orch restart <DAEMON_NAME>
: E- O: k' R9 y4 b, iSPECIAL CASE:
For daemons like node-exporter, prometheus, alertmanager, grafana:

% n: F) r' M0 G% A7 I; TAfter logging into the custom registry, use the ceph config command to configure the custom container images:
; f( v2 p: N8 G8 g, _5 p7 L  Z/ ]
8 |9 h4 r% i: e. s- F8 xRaw
# ceph config set mgr mgr/cephadm/OPTION_NAME CUSTOM_REGISTRY_NAME/CONTAINER_NAME
Use the following options for OPTION_NAME:
( Y. J% z% \. k' A$ n' v& Z- L* Y
Raw
. K) C4 J9 I: F* Z; d4 ]: [! Ncontainer_image_prometheus
' I$ C; F3 ^" C; q- e" W& u" w# f9 y5 g2 Jcontainer_image_grafana
container_image_alertmanager
container_image_node_exporter
Redeploy each daemon:

5 Y9 d' ~7 \8 C: D; ERaw
+ ]8 r3 A+ F5 C; o1 D& ^# ceph orch redeploy DAEMON_NAME
Root Cause
While adding ceph daemons on newly added hosts, cephadm is not able to determine the custom registry credentials under podman-auth.json file under the /etc/ceph directory or the file itself is missing.

  W5 ^; i! n3 ~, H+ @  |" e' z2 p; _Behind the scene, when we run this command it actually only logs in the host where the command is run. As a result, when cephadm attempts to pull the image on other hosts it still fails due to not logged in on the other hosts.
) l1 v3 h" D2 r. t. I7 J
For this, instead of running the command on the host itself, use the ceph cephadm registry-login command from inside the cephadm shell and it should log in all the hosts in the cluster.
4 N2 F4 T' |8 X( }& i! W
Diagnostic Steps
( b4 h% p$ O5 `! G; pCheck if the podman-auth.json file is present under /etc/ceph/ on each node.

# k9 ^3 E! e' ?Raw
. L1 O. g4 D) \# ls -l /etc/ceph/
Verify the daemon status:
9 f- k6 ?( z$ T2 m5 P! W
Raw
" h$ S) [3 R2 c+ F1 \# ceph orch ls
# ceph orch ps

6 C/ s4 _- C  [& c" `4 @9 M1 s