failed to probe daemons or devices 问题ceph -s出现的错误

admin

[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host private-registry.example.com `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -e NODE_NAME=private-registry.example.com -e CEPH_USE_RANDOM_NONCE=1 ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -c %u %g /var/lib/ceph
" P0 e  h3 B& g6 L! zstat: stderr Trying to pull ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01...
stat: stderr Error: initializing source docker://ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01: reading manifest sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 in ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8: unauthorized: authentication required
8 d: c! o' I+ C; {/ V% nHow to eliminate this warning?
Resolution
Login to the cephadm shell on the lead monitor node.

Raw
0 j7 H# ^1 i  ?' Y& t# cephadm shell
/ @5 c3 [: W& B- e6 I+ A! NLog in manually to the custom registry on all the new hosts simultaneously:

  i* T5 d$ j- ~" }Raw
( f& v& R, m2 @1 O# ceph cephadm registry-login --registry-url <CUSTOM_REGISTRY_NAME>  --registry_username <REGISTRY_USERNAME> --registry_password <PASSWORD>
Executing the aforementioned command will produce a podman-auth.json file in the /etc/ceph directory that contains the custom registry's authentication details.
& |+ y+ d. S% ~2 P$ d" k7 l
Wait 3-5 minutes to see if the daemon starts up.
) q0 ^9 E& O8 b9 G+ Q# Y0 \# [
$ a5 h& v2 Z0 I) X! e: s: VRaw
# watch ceph orch ls
  G: J5 ^9 b7 K* zOPTIONAL: Restart the daemon if it is still not in running state.

Raw
# ceph orch restart <DAEMON_NAME>
SPECIAL CASE:
For daemons like node-exporter, prometheus, alertmanager, grafana:

After logging into the custom registry, use the ceph config command to configure the custom container images:
/ ^& B% V( b& N/ T) W7 t
Raw
# ceph config set mgr mgr/cephadm/OPTION_NAME CUSTOM_REGISTRY_NAME/CONTAINER_NAME
- `& m# y" j2 PUse the following options for OPTION_NAME:

8 E0 F" a- U! ]3 s/ E+ |. ERaw
8 {: P) E/ V& |container_image_prometheus
- H* ^. `% `/ x3 gcontainer_image_grafana
container_image_alertmanager
' I8 p3 B5 I! `: T* pcontainer_image_node_exporter
Redeploy each daemon:

Raw
3 u, X6 ]: O1 k# ceph orch redeploy DAEMON_NAME
Root Cause
) F) f$ z: w# m  R' k! gWhile adding ceph daemons on newly added hosts, cephadm is not able to determine the custom registry credentials under podman-auth.json file under the /etc/ceph directory or the file itself is missing.
* H, f* K' @0 L" `
7 i1 Q4 s" \; @: V& J2 H$ x7 }- ]  [4 IBehind the scene, when we run this command it actually only logs in the host where the command is run. As a result, when cephadm attempts to pull the image on other hosts it still fails due to not logged in on the other hosts.
+ W5 x' v% }1 S$ |* }/ j
For this, instead of running the command on the host itself, use the ceph cephadm registry-login command from inside the cephadm shell and it should log in all the hosts in the cluster.

$ y  K! q; @# ^% }4 xDiagnostic Steps
Check if the podman-auth.json file is present under /etc/ceph/ on each node.
" u. ?7 N8 Z( Z, B3 E2 D$ Z
( D, z3 h9 Q( S3 e% ^Raw
# ls -l /etc/ceph/
) s, Q4 e0 _$ B. eVerify the daemon status:
: L+ `3 c  @/ y/ {. s$ }6 ?
Raw
# ceph orch ls
4 a1 U! E& ^( A- O# ceph orch ps
8 v, e9 B. }; I3 ~& c3 [7 z( H
+ N# |( J5 G2 I' z7 n: n& C  B; q