failed to probe daemons or devices 问题ceph -s出现的错误

admin

[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host private-registry.example.com `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -e NODE_NAME=private-registry.example.com -e CEPH_USE_RANDOM_NONCE=1 ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 -c %u %g /var/lib/ceph
stat: stderr Trying to pull ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01...
* k9 L: ~1 o4 D5 Zstat: stderr Error: initializing source docker://ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8@sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01: reading manifest sha256:d42c0d99ddeaa001570dce4eb90b71699e0401fe449966b935f669ffad22bd01 in ceph5-2-1-registry.example.com:5000/rhceph/rhceph-5-rhel8: unauthorized: authentication required
2 y) X6 D1 ?2 A$ d$ yHow to eliminate this warning?
2 Q9 P" S/ {# z. V: b6 Y+ yResolution
Login to the cephadm shell on the lead monitor node.

Raw
# cephadm shell
Log in manually to the custom registry on all the new hosts simultaneously:

! V( H) \) q* T' s" h# q" _) g/ YRaw
# ceph cephadm registry-login --registry-url <CUSTOM_REGISTRY_NAME>  --registry_username <REGISTRY_USERNAME> --registry_password <PASSWORD>
! K$ i* U& b1 a1 r5 o8 p! fExecuting the aforementioned command will produce a podman-auth.json file in the /etc/ceph directory that contains the custom registry's authentication details.
6 z! A2 W4 S. q
Wait 3-5 minutes to see if the daemon starts up.

) K3 _4 \8 P% {0 {& [5 DRaw
' l5 ^7 L0 C5 }# watch ceph orch ls
, g9 @4 E  T; D; Z5 W- uOPTIONAL: Restart the daemon if it is still not in running state.
# I2 T; e2 |8 X0 a
Raw
0 u& M$ Z/ p  x& T4 P. k4 ^# ceph orch restart <DAEMON_NAME>
8 [7 N  J" A6 u  }3 X+ M  f* j  ]SPECIAL CASE:
. a7 }5 o3 o; j* S& I9 X7 bFor daemons like node-exporter, prometheus, alertmanager, grafana:

9 U$ f2 I  Q- b& ~) LAfter logging into the custom registry, use the ceph config command to configure the custom container images:

Raw
: w. j6 K9 u, G# ceph config set mgr mgr/cephadm/OPTION_NAME CUSTOM_REGISTRY_NAME/CONTAINER_NAME
Use the following options for OPTION_NAME:
1 M8 Q5 d8 G9 Q1 U* C# G& m# n/ y* b
Raw
" `( o8 r, b' I/ hcontainer_image_prometheus
container_image_grafana
" T* g& i5 n8 V& r# Bcontainer_image_alertmanager
1 ~( G( u" P7 `5 }2 bcontainer_image_node_exporter
Redeploy each daemon:
3 S1 ?- j6 `5 f; @% ]( j6 }+ ~* o
Raw
# ceph orch redeploy DAEMON_NAME
# S( S) g; p2 mRoot Cause
While adding ceph daemons on newly added hosts, cephadm is not able to determine the custom registry credentials under podman-auth.json file under the /etc/ceph directory or the file itself is missing.

Behind the scene, when we run this command it actually only logs in the host where the command is run. As a result, when cephadm attempts to pull the image on other hosts it still fails due to not logged in on the other hosts.

  p6 P. u3 |: Q& S! qFor this, instead of running the command on the host itself, use the ceph cephadm registry-login command from inside the cephadm shell and it should log in all the hosts in the cluster.
% e  R0 _$ @' h  n- P
( }) ^3 t* o- R, [, W# |* p1 \/ ZDiagnostic Steps
- f1 y# N" F( L. y1 Z6 [, ZCheck if the podman-auth.json file is present under /etc/ceph/ on each node.

Raw
# ls -l /etc/ceph/
Verify the daemon status:
3 |9 G; I9 ?" j0 s. ^/ D( D* D
Raw
# ceph orch ls
% e) B0 [7 w% c( U4 E# ceph orch ps
9 ^  G4 Z1 w7 K+ }  X2 q5 b
! r" N7 c( P# ^' n