ceph health detail 显示pgs not active

admin

单节点安装完ceph后，只创建了两个OSD，导致状态是34.000% pgs not active，用python rados往pool存数据一直卡着，
4 ^  }/ [9 E! B6 ^
0 R8 {3 c! L  b  T4 M% W9 s9 x
将crush map导出
ceph osd getcrushmap -o /usr/src/crushmap
0 B4 W; i* v' V0 a输出，这一步的输出和osd数量有关系
将crush map反编译为可读模式
crushtool -d /usr/src/crushmap -o /usr/src/crushmap.txt
, J, E3 }: K2 T0 X# X4 @修改curshmap，主要是修改OSD级别
$ H$ h7 r& S8 r

. V# O4 G4 X8 N+ L
5 u; T- i7 W% v5 w; @) Bvim   /usr/src/crushmap.txt
# e: X: f) X; o- ?+ ?' W) S找到step chooseleaf firstn 0 type host
4 W% O* h* h" s8 `" x; ?! Z+ c改成step chooseleaf firstn 0 type osd

0 {: J. \1 n7 q3 s; g保存退出：
$ x5 o& F; e6 e" z# D0 b或者通过sed 修改：
sed -i ‘s/step chooseleaf firstn 0 type host/step chooseleaf firstn 0 type osd/’ /usr/src/crushmap.txt

3 c  q& L' _; o$ P- C5 {
& {3 i5 y/ F; p8 C) G  p, e; W5 F
grep ‘step chooseleaf’ /usr/src/crushmap.txt
6 }  S* s0 s4 H2 _输出step chooseleaf firstn 0 type osd
crushtool -c /usr/src/crushmap.txt -o /usr/src/crushmap-new
将修改过的crush map导入集群
ceph osd setcrushmap -i /usr/src/crushmap-new
( e) h) D  U5 c  Y; T& M0 M输出数值比导出数值大就对了。


9 f5 D4 v8 C- l- A8 e
现在再看ceph -s，pg数状态

& I: S4 X8 N* I, ~: P! T9 w

admin

我现在再运行代码，没问题，数据存进去了，对象名是/root/ceph-test/abc.txt
4 D% J" H& S0 f+ k5 D4 x
[root@controller ceph-test]# python pooltest.py
Object contents = /root/ceph-test/abc.txt
% r% m, p! `4 M: Q1 y[root@controller ceph-test]# rados -p data2 ls
) y, F' x7 o! V2 }/ g2 R/root/ceph-test/abc.txt
% }$ _* c4 G( h  \

admin

1、没有创建pool之前

6 }8 c2 a2 H3 o7 A. v, z复制代码
[root@cluster9 ceph-cluster]# ceph -s
) H$ f5 r. N. x$ H! f  cluster:
    id:     d81b3ce4-bcbc-4b43-870e-430950652315
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum cluster9
    mgr: cluster9(active)
    osd: 3 osds: 3 up, 3 in

9 D; J2 G( H' [3 N+ P4 o  G  ?$ W& y  data:
8 F1 e5 I, g: J) x5 Y  M    pools:   0 pools, 0 pgs
# Q& H0 a& [6 M' ]    objects: 0 objects, 0B
    usage:   3.06GiB used, 10.9TiB / 10.9TiB avail
6 c2 ?/ \- d4 Z+ \, i3 Q, \    pgs:     
复制代码
% h1 D' ?% M" Q0 B& Z2、创建pool后
: I) ?7 J; b( R! u, N% S
复制代码
7 D+ Z0 b4 [9 `4 l[root@cluster9 ceph-cluster]# ceph -s
  cluster:
8 J6 W4 O" P, _3 ~& o    id:     d81b3ce4-bcbc-4b43-870e-430950652315
    health: HEALTH_OK

  services:
9 ?5 r2 b% w' K) e$ j& P1 Y    mon: 1 daemons, quorum cluster9
. ]% ^' ]( _( f    mgr: cluster9(active)
' W: N  u6 h1 w6 h    osd: 3 osds: 3 up, 3 in
0 n  V) ?. z/ y/ k, ~) X3 ^/ [2 w
# @/ u6 k3 l( x, F  Z, f# D  data:
! q9 _3 X0 p, \    pools:   1 pools, 128 pgs
    objects: 0 objects, 0B
    usage:   3.06GiB used, 10.9TiB / 10.9TiB avail
; V- a* ?6 P, e    pgs:     100.000% pgs not active
             128 undersized+peered
# F( f0 L0 o9 \( {6 [复制代码
3、修改osd级别

复制代码
[root@cluster9 ceph-cluster]# cd /etc/ceph/
$ L3 v8 f! D- _3 [8 ][root@cluster9 ceph]# ceph osd getcrushmap -o /etc/ceph/crushmap
18
[root@cluster9 ceph]# crushtool -d /etc/ceph/crushmap -o /etc/ceph/crushmap.txt
$ ~9 i6 }. H9 [, z. }[root@cluster9 ceph]# sed -i 's/step chooseleaf firstn 0 type host/step chooseleaf firstn 0 type osd/' /etc/ceph/crushmap.txt
[root@cluster9 ceph]# grep 'step chooseleaf' /etc/ceph/crushmap.txt
    step chooseleaf firstn 0 type osd
[root@cluster9 ceph]# crushtool -c /etc/ceph/crushmap.txt -o /etc/ceph/crushmap-new
[root@cluster9 ceph]# ceph osd setcrushmap -i /etc/ceph/crushmap-new
+ G: k8 ?% Y  D5 c' w$ g; x19
复制代码
- ~+ h" O; _( W' |% t' {: f4、再次查看ceph状态

! y; O' _+ t" Q! u; t0 z; u复制代码
7 e/ Z. @+ u+ W3 J/ @* b[root@cluster9 ceph]# ceph -s
& d+ G4 W+ E+ b9 i8 p  cluster:
    id:     d81b3ce4-bcbc-4b43-870e-430950652315
3 ~4 y$ Z9 l7 Z    health: HEALTH_OK

  services:
* s+ Z, [) p: Z# s$ U* h$ ^! s# G    mon: 1 daemons, quorum cluster9
, r8 X( e6 e! Z; @; _    mgr: cluster9(active)
  H$ W& L) D. x    osd: 3 osds: 3 up, 3 in

  data:
" U2 J0 K( s: i; L& l) C1 \    pools:   1 pools, 128 pgs
9 k. f8 p5 ?# M    objects: 0 objects, 0B
    usage:   3.06GiB used, 10.9TiB / 10.9TiB avail
5 P) v2 f+ B7 c* x3 I    pgs:     128 active+clean
复制代码