找回密码
 注册
查看: 636|回复: 2

ceph fs status报错解决 AttributeError: ‘NoneType’ object has no attribute ‘get’

[复制链接]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2022-8-2 13:27:42 | 显示全部楼层 |阅读模式
问题描述:( t, \+ K! u/ E) R- ?* w: c
[root@mon1 ~]# ceph fs status
" P' P& q9 Z* ]6 m$ y: jError EINVAL: Traceback (most recent call last):
8 A  k0 ]7 X0 r( ~3 m% |) [0 KFile “/usr/lib64/ceph/mgr/status/module.py”, line 310, in handle_command; j, P& z# A( R4 `
return self.handle_fs_status(cmd)
* o; U. b' B' V- KFile “/usr/lib64/ceph/mgr/status/module.py”, line 176, in handle_fs_status$ K  L2 h& ~, I* z
mds_versions[metadata.get(‘ceph_version’, “unknown”)].append(info[‘name’])
/ s  D8 x( c  |  w, F# x" Y( \/ lAttributeError: ‘NoneType’ object has no attribute ‘get’% n( u) t- m3 h$ H3 ?
解决:5 `! h; N; f9 w- I, W2 v
报错的原因可能176行是获取的 metadata 为空;4 I/ }  R; {% d$ P5 T* }4 Y! E
[root@mon1 ~]# ceph mds metadata( x  f$ Q/ @2 R2 Q  U
[
4 f* W  d2 ~5 h{
6 P5 K8 T  T5 A“name”: “mon1”,+ }% a  W$ c1 ^( Y3 q
“addr”: “12.10.1.24:6800/583896531”,; b& X# D- t; n/ B) G* ?5 i
“arch”: “x86_64”,; a7 }/ B+ o( \" H. l( ?! S
“ceph_version”: “ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)”,' `2 @, y, ~2 W7 v2 y
“cpu”: “Intel® Xeon® CPU E5-2670 v2 @ 2.50GHz”,
2 X+ ]# q! {+ x“distro”: “centos”,
. c' Y- J- k+ p3 i3 Y0 u3 Y% [; r“distro_description”: “Redhat Linux 7 (Core)”,& Q8 k' @' ]3 J* {% j
“distro_version”: “7”,
4 |( S! D" y( {/ w% E# `6 S“hostname”: “mon1”,- z/ r1 ^: y' Z8 f( s1 i
“kernel_description”: “#1 SMP Thu Nov 30 09:35:20 EST 2017”,: K" u2 M1 n" r
“kernel_version”: “4.14.3-1.el7.elrepo.x86_64”,; T( D% c1 i" R) s
“mem_swap_kb”: “20971516”,% ~4 Y$ P; f. P
“mem_total_kb”: “65923428”,: [- y: M/ W5 b
“os”: “Linux”, t. F2 b8 l; H+ Q
},7 @! f7 n3 d/ H4 g5 n" E1 k; @
{
6 {( ~: `, Y7 N5 X0 s"name": "mon2"
1 }: t* y0 m2 _},
+ P2 U7 ]2 ~" ]{
7 y5 {, X$ y3 X4 E7 R9 x5 I“name”: “mon3”,
# P7 `5 {, R1 b# x“addr”: “12.10.1.25:6800/3412998711”,
( R2 C2 Q5 S2 m7 a" t% K3 o“arch”: “x86_64”,* Q* i; A' t& g$ ?( b+ m) A& h* A
“ceph_version”: “ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)”,/ K, x2 q1 C# V! A: ~6 v7 q0 P
“cpu”: “Intel® Xeon® CPU E5-2697 v4 @ 2.30GHz”,. k% s% v2 P, n1 e$ V3 @
“distro”: “centos”,
5 u& Q2 m# V) k$ b  O! r“distro_description”: “Redhat Linux 7 (Core)”,
* S# u* ?& |1 t3 ~0 K* F' i$ }“distro_version”: “7”,; O4 i+ r' T' W# i4 Z, v8 o) A, n
“hostname”: “mon3”,# b7 n8 |; j3 C% H) ~( q3 M
“kernel_description”: “#1 SMP Wed May 9 18:05:47 UTC 2018”,
! `% O6 v% c- H5 T“kernel_version”: “3.10.0-862.2.3.el7.x86_64”,% Q4 q, w6 R9 d* u$ P" ~$ B
“mem_swap_kb”: “20479996”,
8 M# p. C  R6 D: A" Z: K“mem_total_kb”: “197975904”,& ]" p3 p4 S9 m
“os”: “Linux”, [% G. Y5 ^( s) W1 a" [% ^: x
}
' D$ L: Y% I' z2 N2 _6 c]
0 F, Z4 K- o2 q) Y' q  ]( V+ N& h; n' I5 U& n6 n/ [* H1 s7 z& E
查询发现有的mds只有name,
$ L0 U  P- u. Q, d6 m# v8 p7 v/ p# v% k& B# f5 b6 E3 V! ?: L. v
这个时候只要把有问题的mds服务重启下即可
8 n; [: P3 `( |) @$ W6 w/ zsystemctl restart ceph-mds@mon2.service
! [6 `$ w& _6 T( C) H5 C8 I' R7 y
2 G; g. B7 J! L! m) k3 t,问题解决。, g: x6 i9 w# P( L, r
, V- j# g5 a4 z, Y
, E" K, w1 K2 k$ B; y( ?

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2022-8-3 10:12:55 | 显示全部楼层
[root@ceph-2 ~]# ceph mds metadata
4 m) x% N" ], |[
: c$ j: ~7 e# q7 T    {
. V4 q6 m* i, D& u% w/ P        "name": "ceph-3",
' x9 i* b$ s. P+ Q. A        "addr": "192.168.120.31:6800/2291079756",
* C3 G+ {1 E2 C( B        "arch": "x86_64",
' f2 W7 G2 u4 h( h" v; R        "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",  V( Q2 I3 l& f9 e5 s  q
        "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",
/ k9 H2 H4 \, G. L' Q3 {        "distro": "rhel",$ [# w; D2 V* Z/ C" O! l- {  K0 L6 F
        "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",% h  M) ?0 O  y; ], l/ r
        "distro_version": "7.9",
2 g+ M/ Y) U1 r" A2 n        "hostname": "ceph-3.novalocal",
& k6 F/ C5 b1 I9 I' m( E7 k+ b: W        "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
& L+ G8 i. N/ b: C! C4 a4 T7 q        "kernel_version": "3.10.0-1160.el7.x86_64",/ x$ ~9 f5 _) R" w
        "mem_swap_kb": "0",
5 G0 S$ ?( B1 ^; Z# |, [        "mem_total_kb": "3880160"," p+ B1 v0 q) Z
        "os": "Linux"  o+ q8 h& m! ^* o' H, _
    },
# K! \: T1 }4 v0 S. p    {
# C: ~4 }  a. x5 P7 H$ V8 d# M        "name": "ceph-2",9 `+ f9 Z& {" R8 ]
        "addr": "192.168.120.16:6800/463439748",
9 R  C% q' Y9 i# \        "arch": "x86_64",) d: ?' F$ w1 e8 i; X
        "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",
# E0 }. z# }6 x9 U/ L        "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",# u2 s- W. F2 q% [
        "distro": "rhel",
4 R  n4 i% `, ]6 k; \0 h. w( n% t        "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
3 X# Q# F  [$ |6 F1 N        "distro_version": "7.9",
% J: g8 m* v* s) L% g. t0 h( f        "hostname": "ceph-2.novalocal",* ~, b1 b% i' z( a* M. O
        "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
) m6 ?9 ^% e  c8 m) R, w4 l' V( X        "kernel_version": "3.10.0-1160.el7.x86_64",9 _. R& v. P+ Y* ^/ U8 e7 T
        "mem_swap_kb": "0",
7 e) ^) F1 @: R        "mem_total_kb": "3880160",
  ]# {/ Z" ~3 A2 @        "os": "Linux"$ ]& m8 d9 B+ z% ]- B9 I3 _
    },2 u0 C9 J1 C( e$ D# p
    {
, c- W+ _% x1 q' Y        "name": "ceph-1",; E0 r7 q* T' a" @, p( Y9 K3 W
        "addr": "192.168.120.23:6800/4181220800",
/ x6 Z4 [1 G. b) r' h6 E7 |        "arch": "x86_64",
  E9 `8 ]& y0 V* ]        "ceph_version": "ceph version 12.2.8-52.el7 (3af3ca15b68572a357593c261f95038d02f46201) luminous (stable)",  k! X0 g- q1 y/ m6 L
        "cpu": "Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)",; V1 w$ P* v# X) e- b' w% I
        "distro": "rhel",  @" |! F1 f1 J0 K
        "distro_description": "Red Hat Enterprise Linux Server 7.9 (Maipo)",
! o  m- _7 Q! @0 n% \6 b        "distro_version": "7.9",
# _4 t1 `. E9 p        "hostname": "ceph-1.novalocal",
5 }  m# i) H' ^2 a        "kernel_description": "#1 SMP Tue Aug 18 14:50:17 EDT 2020",
) a+ k( W2 P5 `) h$ w        "kernel_version": "3.10.0-1160.el7.x86_64",
6 J  u6 b$ ]4 y5 S8 n4 t$ D. @        "mem_swap_kb": "0",
7 H" z* L; i1 F* e3 g/ Y        "mem_total_kb": "3880168",9 [) C+ ^" c8 Q, L7 P/ l( h; n
        "os": "Linux"
3 u- z$ I8 y! n" d! @; `/ j    }
9 F' T/ ^. m* q* q' ?/ L: p; z) M$ T. y]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2022-8-3 10:14:30 | 显示全部楼层
ceph集群提供元数据服务,则客户端可以提挂载ceph文件系统。客户端访问数据时,则在客户端和元数据服务器中都缓存相应的数据。元数据服务器会和客户端inode占用情况来消减缓存。当客户端响应太慢,则会报错“failing to respond to cache pressure” or MDS_HEALTH_CLIENT_RECALL。若确实是客户端负荷较大,是正常读写操作,可以考虑增大mds_recall_warning_decay_rate参数的值(默认为60s),从而消除警告。; [" ^! l! ~* _2 w
# g, D: a7 m# b$ L8 r  F4 t: M! [
可以查询ceph客户端的ID号及其使用inode数(num_caps的值)。
! l* x3 M' G& v
& Z, |8 y8 S# S  r: ^) t# _) oceph tell mds.0 session ls! p- x. H2 u, K0 I) A" x, Q
谨慎使用如下命令踢出目标客户端或全部客户端。
( Y1 t: y& z" `6 l; T
6 u7 {( {  W3 v; k& V# N* hceph tell mds.0 session evict id=11134635
# h: N, Q, c/ B5 u& f" a. [ceph tell mds.0 session evict4 U5 ~) e( `! E' `2 Q
踢出客户端是将客户端加入了黑名单,可以使用如下命令查看黑名单信息或移出黑名单。虽然移出黑名单,可能还不能让客户端正常挂载ceph文件系统,因此需要谨慎处理。
; L) `( {- b& g9 I0 ?1 S( E
0 z5 C4 r+ b& [1 Wceph osd blacklist ls( a% r' Z- Y% g" O9 T: x1 \
ceph osd blacklist rm 192.168.20.1:0/1498586492
* M4 C* z6 I$ ~ceph osd blacklist clear% |% Y0 x8 O* P! v9 q: c9 Q
6. Reduced data availability: 4 pgs inactive, 4 pgs incomplete
" q" g' o" _% ~( X0 s当有pgs出现incomplete时,表明pgs对应的OSDs存活数量少于最小副本数。因此,其对应的数据无法读写,处于reduced状态,会导致MDS服务出问题,提示如下报错信息,示例:' r  e( o* Q8 _: j; ]

4 R# `$ n+ }. w3 MDSs report slow metadata IOs: C* p. f8 [7 g* i) _
2 MDSs report slow requests+ S' E4 ^# J' J
2 MDSs behind on trimming
2 r* Y  D* t: k2 K- a3 R! uReduced data availability: 4 pgs inactive, 4 pgs incomplete
. h# A6 J# M; [; C
$ @8 x! S1 A+ ppg 5.6de is incomplete, acting [254,356,222,352,111,247,100,133,351,206] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
0 j! V) M9 |" wpg 5.6e9 is incomplete, acting [276,244,357,358,221,321,311,229,314,351] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')* j4 M2 [8 @$ z' t
pg 5.73b is incomplete, acting [186,279,351,247,293,354,359,220,181,283] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')7 u% ]# ?& L4 X0 O% w1 L2 F. q
pg 5.eda is incomplete, acting [164,157,120,227,353,351,295,269,95,354] (reducing pool cephfs_data min_size from 8 may help; search ceph.com/docs for 'incomplete')
* G+ z7 V) R8 q& B" n此时,需要修复pgs。
' g: V, W1 [* b( N" t3 Y/ h( P4 i2 f: r! O0 _* {, M7 ^
# 查询pg信息(pg id 为 5.6de)
5 r, I# }- d, u7 o1 P+ g$ zceph pg 5.6de query) w. B& l1 k2 J( w* A* ]) H
, B+ A. D- {8 T# P) R& \. }
# 强行重建pg
6 ?. o$ U  ~  |ceph osd force-create-pg 5.6de --yes-i-really-mean-it
6 O* S7 g3 n' T4 D" V5 F4 S
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-12 00:12 , Processed in 0.035886 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表