failed to probe daemons or devices 问题ceph -s出现的错误

admin

[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host private-registry.example.com `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -e NODE_NAME=private-registry.example.com -e CEPH_USE_RANDOM_NONCE=1 ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -c %u %g /var/lib/ceph
/ H, K/ M; V0 @% ostat: stderr Trying to pull ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01...
6 a( _/ p5 P- astat: stderr Error: initializing source docker://ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01: reading manifest sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 in ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8: unauthorized: authentication required
2 r0 P) U1 l+ \How to eliminate this warning?
Resolution
Login to the cephadm shell on the lead monitor node.
; R2 H" U6 f- f  e) [
. b$ d- n  e+ j/ y4 D6 @Raw
2 K3 \1 k; X" t9 S. F# cephadm shell
5 C) A, l, x) [. w9 l: c* ALog in manually to the custom registry on all the new hosts simultaneously:

Raw
5 d7 a! A" i, D; q1 i# ceph cephadm registry-login --registry-url <CUSTOM_REGISTRY_NAME>  --registry_username <REGISTRY_USERNAME> --registry_password <PASSWORD>
Executing the aforementioned command will produce a podman-auth.json file in the /etc/ceph directory that contains the custom registry's authentication details.
5 z" B; q- s/ e: E- A$ n( `# K6 i
4 ?8 Z! X$ w% ?9 J! sWait 3-5 minutes to see if the daemon starts up.

Raw
& C$ G4 w/ T5 K+ N1 _# watch ceph orch ls
, x) t3 T6 B' ]" dOPTIONAL: Restart the daemon if it is still not in running state.
9 m$ I7 b4 J% W, V5 v
Raw
# ceph orch restart <DAEMON_NAME>
: P9 c3 u) x& i4 L. l4 l; `4 ?SPECIAL CASE:
7 c8 m7 y3 J  p1 B# DFor daemons like node-exporter, prometheus, alertmanager, grafana:

After logging into the custom registry, use the ceph config command to configure the custom container images:
% w* q% N. e  U; p; Q1 |
' q0 O5 P9 M+ n. L2 J. m2 d) tRaw
' }5 Q! ^+ ?/ T2 U. O1 t# ceph config set mgr mgr/cephadm/OPTION_NAME CUSTOM_REGISTRY_NAME/CONTAINER_NAME
Use the following options for OPTION_NAME:
8 f8 x# e( ]4 a% Q
Raw
$ Q8 G: O$ @8 `" J0 H) V6 Vcontainer_image_prometheus
container_image_grafana
container_image_alertmanager
container_image_node_exporter
  a& `# ?8 @' d+ N) Z/ p. \Redeploy each daemon:

$ k* l; d* o0 y" H' h. PRaw
# ceph orch redeploy DAEMON_NAME
Root Cause
$ j$ i/ y9 d3 V6 v- [- `6 g6 f4 Z4 zWhile adding ceph daemons on newly added hosts, cephadm is not able to determine the custom registry credentials under podman-auth.json file under the /etc/ceph directory or the file itself is missing.
" [+ \% D. s: D/ v
Behind the scene, when we run this command it actually only logs in the host where the command is run. As a result, when cephadm attempts to pull the image on other hosts it still fails due to not logged in on the other hosts.

For this, instead of running the command on the host itself, use the ceph cephadm registry-login command from inside the cephadm shell and it should log in all the hosts in the cluster.
$ C4 @3 z* ?% R# x/ E" m  c# t
# o3 c  L4 M' i7 t  k" L, g; zDiagnostic Steps
Check if the podman-auth.json file is present under /etc/ceph/ on each node.
% ?/ j# _  ^3 s% l6 c, p
' y( z% h) T" g% GRaw
8 ]# D3 b9 w6 V, q& k; ?' d- N! K# ls -l /etc/ceph/
Verify the daemon status:
& D' v  B* L+ o, J' [" ^) R
  m2 E7 A: d) e  _Raw
# ceph orch ls
1 `$ c3 I3 ~8 y' v# ceph orch ps