HEALTH_WARN failed to probe daemons or devices

admin

问题

• After upgrading Red Hat Ceph Storage ( RHCS ) from version 5.0x to version 5.1, the following warning shows in the ceph health detail output:
  [size=0.75]Raw
  . P+ N( y+ v( m. F/ m5 B[root@node01 ~]# ceph health detailHEALTH_WARN failed to probe daemons or devices[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices    host node01 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/<fsid>/mon.node01/configERROR: [Errno 2] No such file or directory: '/var/lib/ceph/<fsid>/mon.node01/config'
  3 u% }% E9 I8 r! T
  The above error points to a non existing directory /var/lib/ceph/<fsid>/mon.node01/config. This directory holds the configuration files for a Ceph mon service, but oddly, the node node01 is not a monitor.
• Why is this message showing in a node that is not a monitor?
• How to eliminate this warning?
  
  5 X/ I, x; k( `5 I6 b

决议

• This issue has been reported in Red Hat internal BZ #2080242. It'll be fixed in RHCS 5.2. At the time of writing, this version is still not released.
• As a workaround, remove any mon folders named /var/lib/ceph/mon in the affected node. The mentioned warning in the issue description will disappear after that.
  CAUTION
  Please, be aware that the folder /var/lib/ceph/<fsid> contains critical cluster information. Please, do not remove any content from this directory, or the directory itself. This might cause permanent damage to the Ceph cluster.
  
  

根源

This issue is observed in deployments upgraded from RHCS 4.x to RHCS 5.0. The mentioned directory /var/lib/ceph/mon is a leftover from the ceph-ansible deployment that wasn't cleared as part of the upgrade process.

诊断步骤

• Review the output of the cephadm ls command. It'll show a legacy mon service that shouldn't be present in a Ceph node that is not a monitor:
  [size=0.75]Raw
      {            "style": "legacy",            "name": "mon.node01",            "fsid": "<fsid>",            "systemd_unit": "ceph-mon@node1",            "enabled": false,            "state": "stopped",            "host_version": "16.2.0-152.el8cp"        }
  
• The cephadm tool pulls the information about legacy services from /var/lib/ceph/<daemon-name> so in this case, it would be /var/lib/ceph/mon. Review the contents of this folder to confirm a mon directory is listed.
  
  ( U' S6 Z1 p: m" {/ i