ceph fs status报错解决 AttributeError: ‘NoneType’ object has no attribute ‘get’

admin

问题描述：
[root@mon1 ~]# ceph fs status
Error EINVAL: Traceback (most recent call last):
) [1 Z. N8 q3 xFile “/usr/lib64/ceph/mgr/status/module.py”, line 310, in handle_command
return self.handle_fs_status(cmd)
, R9 R; B5 i  i$ H" i) IFile “/usr/lib64/ceph/mgr/status/module.py”, line 176, in handle_fs_status
4 L- l  e! \8 r, {+ O7 B5 s6 x" m  Jmds_versions[metadata.get(‘ceph_version’, “unknown”)].append(info[‘name’])
: w' y& K4 K" S1 H/ D( w+ q1 wAttributeError: ‘NoneType’ object has no attribute ‘get’
" w" K3 W* {6 I  x解决：
) `* `% @4 B3 {# J9 |报错的原因可能176行是获取的 metadata 为空；
[root@mon1 ~]# ceph mds metadata
[
! g8 `  V3 c+ h( y! s" u8 M5 x{
6 I6 H' A2 }3 ~“name”: “mon1”,
  P1 H8 D6 e* I* x1 J( S' w* b“addr”: “12.10.1.24:6800/583896531”,
“arch”: “x86_64”,
5 k) B& j9 ?- i, v% W9 p“ceph_version”: “ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)”,
“cpu”: “Intel® Xeon® CPU E5-2670 v2 @ 2.50GHz”,
“distro”: “centos”,
“distro_description”: “Redhat Linux 7 (Core)”,
) Y- |- T5 s; r" o/ s5 J“distro_version”: “7”,
' H! z7 e8 C% H# I, S, W" ?* l“hostname”: “mon1”,
“kernel_description”: “#1 SMP Thu Nov 30 09:35:20 EST 2017”,
' E  G/ N9 L! K4 W8 I$ I“kernel_version”: “4.14.3-1.el7.elrepo.x86_64”,
“mem_swap_kb”: “20971516”,
! a3 y. ]. ~) L“mem_total_kb”: “65923428”,
; r" V+ H, O0 t3 `% @# Y+ E1 e7 K“os”: “Linux”
},
1 c: e* ]+ P+ U{
0 B2 x  O8 d  M- \7 _+ _0 e"name": "mon2"
7 G) V% S' ~4 r* w( K7 Q},
{
“name”: “mon3”,
“addr”: “12.10.1.25:6800/3412998711”,
“arch”: “x86_64”,
; z# m. v5 u9 {“ceph_version”: “ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)”,
/ O$ e+ ?- z5 h; T/ B  O/ ]“cpu”: “Intel® Xeon® CPU E5-2697 v4 @ 2.30GHz”,
“distro”: “centos”,
“distro_description”: “Redhat Linux 7 (Core)”,
“distro_version”: “7”,
“hostname”: “mon3”,
“kernel_description”: “#1 SMP Wed May 9 18:05:47 UTC 2018”,
" P( D2 t; ]# H- c7 A% q“kernel_version”: “3.10.0-862.2.3.el7.x86_64”,
; r6 u+ V. G; w' p3 Z8 O“mem_swap_kb”: “20479996”,
$ D* a/ h% ?2 B) Q+ v0 y) S) c! N“mem_total_kb”: “197975904”,
“os”: “Linux”
}
]
( ^# R! r3 t) a
查询发现有的mds只有name，

这个时候只要把有问题的mds服务重启下即可
systemctl restart ceph-mds@mon2.service

，问题解决。
) _+ `* c0 h1 }" \

  ]) t6 P- P# `2 `+ D: d

admin

[root@ceph-2 ~]# ceph mds metadata
) S: Q: G& E6 ^) c" @, g[
    {
6 R: J, r  }0 ]& E        "name": "ceph-3",
        "addr": "192.168.120.31:6800/2291079756",
        "arch": "x86_64",
/ q) j1 P  P: Y        "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",
8 d) c3 `0 y* [2 \1 z( D, w        "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",
        "distro": "rhel",
        "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
        "distro_version": "7.9",
        "hostname": "ceph-3.novalocal",
$ i& n" G8 x( w        "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
$ h) p, I  p6 R1 s, Q6 Z7 I        "kernel_version": "3.10.0-1160.el7.x86_64",
( V0 t( I9 E. U8 V% n        "mem_swap_kb": "0",
* W; i% \: |4 D  _' ~1 c: |/ l        "mem_total_kb": "3880160",
4 ^( y. E2 S: H2 y$ l, s* ]        "os": "Linux"
    },
/ \) m+ \) |0 x" G8 e    {
% Z7 I" _$ R& [7 \& ]- T) F        "name": "ceph-2",
        "addr": "192.168.120.16:6800/463439748",
        "arch": "x86_64",
        "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",
        "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",
# L+ O/ f$ f% }8 b% e# q1 S        "distro": "rhel",
  E  J6 m' `$ q; q0 w        "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
6 r  u% X: U+ ]4 m% p$ C; P* P7 M! v        "distro_version": "7.9",
/ Q6 S; I* J" X        "hostname": "ceph-2.novalocal",
$ d# p5 E0 E3 o7 _! |: Q        "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
/ T2 N. f! ]/ O( c        "kernel_version": "3.10.0-1160.el7.x86_64",
3 i% ~5 t7 N: v+ ?4 o9 A' c6 w        "mem_swap_kb": "0",
        "mem_total_kb": "3880160",
; B3 e; }/ l. W( \/ @, K! A        "os": "Linux"
$ ?0 c( g  z# t8 z# w& l    },
. D# r" m0 p9 d; }    {
( E! S7 J0 `+ @; S) o        "name": "ceph-1",
        "addr": "192.168.120.23:6800/4181220800",
! Q  ]% @3 @0 W* ]( C7 A        "arch": "x86_64",
        "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",
        "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",
        "distro": "rhel",
  L5 b9 c7 @6 |/ @        "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
- m! ?" b( o6 }: e6 n1 ~1 k        "distro_version": "7.9",
2 N' S+ m6 o# d' m        "hostname": "ceph-1.novalocal",
        "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
        "kernel_version": "3.10.0-1160.el7.x86_64",
+ \- Q" b! |/ H+ Q/ ?0 W; `        "mem_swap_kb": "0",
: ^6 c% A2 ?2 J4 c  M" R0 |        "mem_total_kb": "3880168",
7 |% h+ F( A6 u3 Z7 ?        "os": "Linux"
    }
]

admin

ceph集群提供元数据服务，则客户端可以提挂载ceph文件系统。客户端访问数据时，则在客户端和元数据服务器中都缓存相应的数据。元数据服务器会和客户端inode占用情况来消减缓存。当客户端响应太慢，则会报错“failing to respond to cache pressure” or MDS_HEALTH_CLIENT_RECALL。若确实是客户端负荷较大，是正常读写操作，可以考虑增大mds_recall_warning_decay_rate参数的值（默认为60s），从而消除警告。

可以查询ceph客户端的ID号及其使用inode数（num_caps的值）。
8 p8 O* E3 t; F% ~1 {& c6 e
7 u# {: z( M3 n4 ^) e# Pceph tell mds.0 session ls
8 T  N6 S; u3 d5 T4 V: T" [) z谨慎使用如下命令踢出目标客户端或全部客户端。

ceph tell mds.0 session evict id=11134635
& U- m- j/ S' t% o1 Dceph tell mds.0 session evict
踢出客户端是将客户端加入了黑名单，可以使用如下命令查看黑名单信息或移出黑名单。虽然移出黑名单，可能还不能让客户端正常挂载ceph文件系统，因此需要谨慎处理。

ceph osd blacklist ls
& F' ^8 y/ |0 E! Tceph osd blacklist rm 192.168.20.1:0/1498586492
ceph osd blacklist clear
6. Reduced data availability: 4 pgs inactive, 4 pgs incomplete
( `9 a  t) t: p6 P/ ?- C2 g& _当有pgs出现incomplete时，表明pgs对应的OSDs存活数量少于最小副本数。因此，其对应的数据无法读写，处于reduced状态，会导致MDS服务出问题，提示如下报错信息，示例：

3 MDSs report slow metadata IOs
1 d  X$ u$ R* n: R" Z7 L2 MDSs report slow requests
$ F5 @0 r0 d9 D1 V2 MDSs behind on trimming
Reduced data availability: 4 pgs inactive, 4 pgs incomplete
, A! ~$ j2 J  p7 W
/ |) R8 ~3 c1 ?% k2 Hpg 5.6de is incomplete, acting [254,356,222,352,111,247,100,133,351,206] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
pg 5.6e9 is incomplete, acting [276,244,357,358,221,321,311,229,314,351] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
8 h5 _9 F# a* t$ e: P' Cpg 5.73b is incomplete, acting [186,279,351,247,293,354,359,220,181,283] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
9 B$ \, Y+ T0 A7 Gpg 5.eda is incomplete, acting [164,157,120,227,353,351,295,269,95,354] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
6 _5 H3 x1 B" _" p# |) h( X; N0 e0 a此时，需要修复pgs。

% e# U! }) C& u; g0 T. Q: A4 I# Y& M# 查询pg信息（pg id 为 5.6de）
ceph pg 5.6de query
$ H, L9 N% [; Z2 {9 G6 ?; f
# 强行重建pg
7 o* |5 A# _* f: G0 P: y6 fceph osd force-create-pg 5.6de --yes-i-really-mean-it
" h1 `+ S8 e( y! o; ~0 X$ R4 w/ D