failed to probe daemons or devices 问题ceph -s出现的错误

admin

[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host private-registry.example.com `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -e NODE_NAME=private-registry.example.com -e CEPH_USE_RANDOM_NONCE=1 ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -c %u %g /var/lib/ceph
stat: stderr Trying to pull ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01...
1 r1 W! Y4 h/ b7 _stat: stderr Error: initializing source docker://ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01: reading manifest sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 in ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8: unauthorized: authentication required
! t4 a+ R8 c% ?1 s" xHow to eliminate this warning?
" n( i- T$ c. M4 _0 f4 o; XResolution
Login to the cephadm shell on the lead monitor node.
, V! l3 N8 R' S/ |& h! K2 v" Q
Raw
, `# v* a3 ?) p) _, X/ L! z9 q: Q# cephadm shell
9 Z. R/ A, c: W6 e, u5 H% QLog in manually to the custom registry on all the new hosts simultaneously:
. `6 U5 ?8 ]  H  I. s0 `5 i
Raw
9 \9 o& A% A% Q5 X+ f$ L, \# ceph cephadm registry-login --registry-url <CUSTOM_REGISTRY_NAME>  --registry_username <REGISTRY_USERNAME> --registry_password <PASSWORD>
& M. R+ P, J6 q3 x" BExecuting the aforementioned command will produce a podman-auth.json file in the /etc/ceph directory that contains the custom registry's authentication details.

Wait 3-5 minutes to see if the daemon starts up.

Raw
- `2 }3 w1 Q  f/ K3 C! O# watch ceph orch ls
OPTIONAL: Restart the daemon if it is still not in running state.

9 B  r5 v4 z5 m, a5 L  Q9 A. lRaw
# ceph orch restart <DAEMON_NAME>
SPECIAL CASE:
For daemons like node-exporter, prometheus, alertmanager, grafana:
2 A) b2 r. `6 q& _) A- X
  w- @: E$ D* v. O& I  X! T& zAfter logging into the custom registry, use the ceph config command to configure the custom container images:

. B$ C! P6 |, \; Y( [: lRaw
% L) x. V* Y" E7 C( F' s6 D1 L) {# ceph config set mgr mgr/cephadm/OPTION_NAME CUSTOM_REGISTRY_NAME/CONTAINER_NAME
Use the following options for OPTION_NAME:
) B4 N2 H$ r* u9 u$ C
Raw
container_image_prometheus
container_image_grafana
container_image_alertmanager
! M3 e) q' O- q- acontainer_image_node_exporter
Redeploy each daemon:
/ K4 ^$ V0 L' g% F
Raw
# ceph orch redeploy DAEMON_NAME
6 L* g/ J% q* |3 hRoot Cause
, e) F0 E! n  D5 L' DWhile adding ceph daemons on newly added hosts, cephadm is not able to determine the custom registry credentials under podman-auth.json file under the /etc/ceph directory or the file itself is missing.

Behind the scene, when we run this command it actually only logs in the host where the command is run. As a result, when cephadm attempts to pull the image on other hosts it still fails due to not logged in on the other hosts.
: o1 S; W. @, [: V7 l) L( _& R8 I
+ m; m+ i- ^# D4 n$ C- t( @For this, instead of running the command on the host itself, use the ceph cephadm registry-login command from inside the cephadm shell and it should log in all the hosts in the cluster.

! ^* Z: ^2 g/ P* ^! r+ r7 u. p) ]Diagnostic Steps
Check if the podman-auth.json file is present under /etc/ceph/ on each node.

$ f! |* O# D/ Y* O$ a! kRaw
# ls -l /etc/ceph/
2 t  q$ l+ c8 T9 s0 EVerify the daemon status:

Raw
! g+ U  F# P; _4 C# ceph orch ls
# ceph orch ps
( h6 i, ^& V/ P/ |6 x+ t5 v