HEALTH_WARN failed to probe daemons or devices

admin

问题

• After upgrading Red Hat Ceph Storage ( RHCS ) from version 5.0x to version 5.1, the following warning shows in the ceph health detail output:
  [size=0.75]Raw
  [root@node01 ~]# ceph health detailHEALTH_WARN failed to probe daemons or devices[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices    host node01 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/<fsid>/mon.node01/configERROR: [Errno 2] No such file or directory: '/var/lib/ceph/<fsid>/mon.node01/config'
  
  The above error points to a non existing directory /var/lib/ceph/<fsid>/mon.node01/config. This directory holds the configuration files for a Ceph mon service, but oddly, the node node01 is not a monitor.
• Why is this message showing in a node that is not a monitor?
• How to eliminate this warning?
  
  & X' g5 @6 ~* R; `

决议

• This issue has been reported in Red Hat internal BZ #2080242. It'll be fixed in RHCS 5.2. At the time of writing, this version is still not released.
• As a workaround, remove any mon folders named /var/lib/ceph/mon in the affected node. The mentioned warning in the issue description will disappear after that.
  CAUTION
  Please, be aware that the folder /var/lib/ceph/<fsid> contains critical cluster information. Please, do not remove any content from this directory, or the directory itself. This might cause permanent damage to the Ceph cluster.
  
  , V# h, c. r; M* [* f6 ~

根源

This issue is observed in deployments upgraded from RHCS 4.x to RHCS 5.0. The mentioned directory /var/lib/ceph/mon is a leftover from the ceph-ansible deployment that wasn't cleared as part of the upgrade process.

诊断步骤

• Review the output of the cephadm ls command. It'll show a legacy mon service that shouldn't be present in a Ceph node that is not a monitor:
  [size=0.75]Raw
  / n; d6 w! n0 ?8 n    {            "style": "legacy",            "name": "mon.node01",            "fsid": "<fsid>",            "systemd_unit": "ceph-mon@node1",            "enabled": false,            "state": "stopped",            "host_version": "16.2.0-152.el8cp"        }
  
• The cephadm tool pulls the information about legacy services from /var/lib/ceph/<daemon-name> so in this case, it would be /var/lib/ceph/mon. Review the contents of this folder to confirm a mon directory is listed.
  
  7 ^& K- C6 n* x