ceph fs status报错解决 AttributeError: ‘NoneType’ object has no attribute ‘get’

admin · 发表于 2022-8-2 13:27:42

问题描述：
[root@mon1 ~]# ceph fs status
Error EINVAL: Traceback (most recent call last):
File “/usr/lib64/ceph/mgr/status/module.py”, line 310, in handle_command
return self.handle_fs_status(cmd)
File “/usr/lib64/ceph/mgr/status/module.py”, line 176, in handle_fs_status
mds_versions[metadata.get(‘ceph_version’, “unknown”)].append(info[‘name’])
AttributeError: ‘NoneType’ object has no attribute ‘get’
解决：
报错的原因可能176行是获取的 metadata 为空；
[root@mon1 ~]# ceph mds metadata
[
{
“name”: “mon1”,
“addr”: “12.10.1.24:6800/583896531”,
“arch”: “x86_64”,
“ceph_version”: “ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)”,
“cpu”: “Intel® Xeon® CPU E5-2670 v2 @ 2.50GHz”,
“distro”: “centos”,
“distro_description”: “Redhat Linux 7 (Core)”,
“distro_version”: “7”,
“hostname”: “mon1”,
“kernel_description”: “#1 SMP Thu Nov 30 09:35:20 EST 2017”,
“kernel_version”: “4.14.3-1.el7.elrepo.x86_64”,
“mem_swap_kb”: “20971516”,
“mem_total_kb”: “65923428”,
“os”: “Linux”
},
{
"name": "mon2"
},
{
“name”: “mon3”,
“addr”: “12.10.1.25:6800/3412998711”,
“arch”: “x86_64”,
“ceph_version”: “ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)”,
“cpu”: “Intel® Xeon® CPU E5-2697 v4 @ 2.30GHz”,
“distro”: “centos”,
“distro_description”: “Redhat Linux 7 (Core)”,
“distro_version”: “7”,
“hostname”: “mon3”,
“kernel_description”: “#1 SMP Wed May 9 18:05:47 UTC 2018”,
“kernel_version”: “3.10.0-862.2.3.el7.x86_64”,
“mem_swap_kb”: “20479996”,
“mem_total_kb”: “197975904”,
“os”: “Linux”
}
]

查询发现有的mds只有name，

这个时候只要把有问题的mds服务重启下即可
systemctl restart ceph-mds@mon2.service

，问题解决。

admin · 发表于 2022-8-3 10:12:55

[root@ceph-2 ~]# ceph mds metadata
[
{
      "name": "ceph-3",
      "addr": "192.168.120.31:6800/2291079756",
      "arch": "x86_64",
      "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",
      "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",
      "distro": "rhel",
      "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
      "distro_version": "7.9",
      "hostname": "ceph-3.novalocal",
      "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
      "kernel_version": "3.10.0-1160.el7.x86_64",
      "mem_swap_kb": "0",
      "mem_total_kb": "3880160",
      "os": "Linux"
},
{
      "name": "ceph-2",
      "addr": "192.168.120.16:6800/463439748",
      "arch": "x86_64",
      "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",
      "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",
      "distro": "rhel",
      "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
      "distro_version": "7.9",
      "hostname": "ceph-2.novalocal",
      "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
      "kernel_version": "3.10.0-1160.el7.x86_64",
      "mem_swap_kb": "0",
      "mem_total_kb": "3880160",
      "os": "Linux"
},
{
      "name": "ceph-1",
      "addr": "192.168.120.23:6800/4181220800",
      "arch": "x86_64",
      "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",
      "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",
      "distro": "rhel",
      "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
      "distro_version": "7.9",
      "hostname": "ceph-1.novalocal",
      "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
      "kernel_version": "3.10.0-1160.el7.x86_64",
      "mem_swap_kb": "0",
      "mem_total_kb": "3880168",
      "os": "Linux"
}
]

admin · 发表于 2022-8-3 10:14:30

ceph集群提供元数据服务，则客户端可以提挂载ceph文件系统。客户端访问数据时，则在客户端和元数据服务器中都缓存相应的数据。元数据服务器会和客户端inode占用情况来消减缓存。当客户端响应太慢，则会报错“failing to respond to cache pressure” or MDS_HEALTH_CLIENT_RECALL。若确实是客户端负荷较大，是正常读写操作，可以考虑增大mds_recall_warning_decay_rate参数的值（默认为60s），从而消除警告。

可以查询ceph客户端的ID号及其使用inode数（num_caps的值）。

ceph tell mds.0 session ls
谨慎使用如下命令踢出目标客户端或全部客户端。

ceph tell mds.0 session evict id=11134635
ceph tell mds.0 session evict
踢出客户端是将客户端加入了黑名单，可以使用如下命令查看黑名单信息或移出黑名单。虽然移出黑名单，可能还不能让客户端正常挂载ceph文件系统，因此需要谨慎处理。

ceph osd blacklist ls
ceph osd blacklist rm 192.168.20.1:0/1498586492
ceph osd blacklist clear
6. Reduced data availability: 4 pgs inactive, 4 pgs incomplete
当有pgs出现incomplete时，表明pgs对应的OSDs存活数量少于最小副本数。因此，其对应的数据无法读写，处于reduced状态，会导致MDS服务出问题，提示如下报错信息，示例：

3 MDSs report slow metadata IOs
2 MDSs report slow requests
2 MDSs behind on trimming
Reduced data availability: 4 pgs inactive, 4 pgs incomplete

pg 5.6de is incomplete, acting [254,356,222,352,111,247,100,133,351,206] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
pg 5.6e9 is incomplete, acting [276,244,357,358,221,321,311,229,314,351] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
pg 5.73b is incomplete, acting [186,279,351,247,293,354,359,220,181,283] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
pg 5.eda is incomplete, acting [164,157,120,227,353,351,295,269,95,354] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
此时，需要修复pgs。

# 查询pg信息（pg id 为 5.6de）
ceph pg 5.6de query

# 强行重建pg
ceph osd force-create-pg 5.6de --yes-i-really-mean-it

		自动登录	找回密码
密码			注册