ceph 所有问题总结 application not enabled on 1 pool(s)

admin

1.问题:

# ceph health

HEALTH_WARN      application not enabled on 1 pool(s)

解决:

( E9 A- J) {0 b9 _3 i* e

# ceph health detail

. R/ [) ^9 J# M8 @: G

HEALTH_WARN application not enabled on 1 pool(s)

1 w6 r3 C2 I( D4 p3 S  H/ T

POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)

$ c* I& K, c7 b4 A/ y2 g( a

    application not enabled on pool 'kube'

    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

# ceph osd pool application enable kube rbd

  enabled application 'rbd' on pool 'kube'

9 B6 s" ^* N$ Y$ e

# ceph health

HEALTH_OK

0 h' Q9 `/ M( }3 e

% ]2 s! E' G$ }3 p+ y' G6 R

2.问题:

# ceph -s
  cluster:
# e6 h& ?' w0 S4 O% b! F# l6 x, p/ y    id:     e781a2e4-097d-4867-858d-bdbd3a264435
# h" K( t" w( m3 @4 G    health: HEALTH_WARN
            clock skew detected on mon.ceph02, mon.ceph03

解决:

5 L' O& g1 M8 Y) E' t! f, Q####确认NTP服务是否正常工作
; @# i9 |$ F1 E9 ]2 s# u, C# systemctl status ntpd
9 d* B/ K2 d( A* B/ i& p####修改ceph配置中的时间偏差阈值
3 w/ ~5 Z0 u8 y/ [6 B/ t5 J( j+ Z# vim /etc/ceph/ceph.conf
8 l0 B% e# S0 x) H% j###在global字段下添加:
# ]: k- w7 u/ Q: B( G" g0 kmon clock drift allowed = 2
9 [5 ?8 a4 S  z! Q) U2 N, D  }5 zmon clock drift warn backoff = 30   
####向需要同步的mon节点推送配置文件
  D6 {- T. \( |) ~# q# cd /etc/ceph/
# ceph-deploy --overwrite-conf config  push ceph{01..03}
) C4 R% @8 R9 }% J4 w* p& I####重启mon服务并验证
# systemctl restart ceph-mon.target
# ceph -s
  cluster:
- W$ z+ J8 c3 e' C    id:     e781a2e4-097d-4867-858d-bdbd3a264435
    health: HEALTH_OK

1 l5 t2 W2 o9 z& Q; t5 {" g3.问题:
# z2 Z/ n* @$ ^" t, S2 S# rbd map abc/zhijian --id admin
* ?. ~! I2 l) q. h6 t. rrbd: sysfs write failed
+ |6 ^  U5 L  a# F& \RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
* Z0 P! m2 u9 PIn some cases useful info is found in syslog - try "dmesg | tail".
+ d0 Z. o, Y) T3 n) U$ t' T0 nrbd: map failed: (6) No such device or address
1 C+ P% v. u% Q! K+ Y; {# w# K
解决:
$ |2 v' j+ G. b8 \由于kernel不支持块设备镜像的一些特性，所以映射失败
; [2 v2 \+ R- ?6 N8 y( \8 {3 q) q# rbd feature disable abc/zhijian exclusive-lock, object-map, fast-diff, deep-flatten
# rbd info abc/zhijian
rbd image 'zhijian':
. t0 f# J0 W. F& n! ^4 Esize 1024 MB in 256 objects
+ v  U/ x# h/ S& r. M. _. worder 22 (4096 kB objects)
" @9 C0 I; @+ S; O# @block_name_prefix: rbd_data.1011074b0dc51
' e0 _$ N* |) M8 M5 iformat: 2
0 S" s/ e( h8 O* k! b9 cfeatures: layering
flags:
create_timestamp: Sun May  6 13:35:21 2018
- ?0 l9 H- o$ w: s, w! H# H* k! c# rbd map abc/zhijian --id admin
" t' ^% k: X, ]6 c/dev/rbd0
$ B! y7 Q) s( y6 l/ N% E+ H" I+ a$ t
- w+ e# b+ H0 C' Q/ x. B4.问题:
# ceph osd pool delete cephfs_data
$ n) ?; w9 k0 {Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool cephfs_data.  If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it.
# ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool

解决:
" m) ]' Q/ S' S3 |( |# tail -n 2 /etc/ceph/ceph.conf
9 N6 N  U4 L6 f2 H* h7 B[mon]
mon allow pool delete = true
3 \2 B1 h" L8 i" F  z% `
向需要同步的mon节点推送配置文件:
: ^" Z7 T! Z7 T* N. }7 B# cd /etc/ceph/
% l" S; b! g/ R# ceph-deploy --overwrite-conf config  push ceph{01..03}

重启mon服务并验证:
# systemctl restart ceph-mon.target
# ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it
pool 'cephfs_data' removed
  m& }" S* e8 m+ J/ c
$ [5 S: a! r3 W! c6 Y  o" i/ ?  G! c5.问题:
" {1 J; b5 X/ m# ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it
Error EBUSY: pool 'cephfs_data' is in use by CephFS
2 T" n- R3 a1 ]( v/ R, R0 X
解决:
# ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
, D% }+ B3 J* F#  ceph fs rm cephfs --yes-i-really-mean-it
Error EINVAL: all MDS daemons must be inactive before removing filesystem
" K1 Y( d9 e; X( }7 L- j2 }  S0 f# systemctl stop ceph-mds.target
# ceph fs rm cephfs
Error EPERM: this is a DESTRUCTIVE operation and will make data in your filesystem permanently inaccessible.  Add --yes-i-really-mean-it if you are sure you wish to continue.
# ceph fs rm cephfs --yes-i-really-mean-it
# ceph fs ls
6 B( z* S" n: V% U; u$ j9 y& lNo filesystems enabled
8 f- v/ x. Y4 U) H* |7 D
8 {7 E# J* A/ P6.问题:
使用静态PV创建pod，pod一直处于ContainerCreating状态:
8 O' s( h! j0 O% {7 Y5 X; P# kubectl get pod ceph-pod1
! c5 i8 R3 Y# H9 q: ]3 UNAME        READY     STATUS              RESTARTS   AGE
6 B; u" ^1 U) H, j6 t- m. g, Zceph-pod1   0/1       ContainerCreating   0          10s
......
# kubectl describe pod ceph-pod1
+ h& ?: m7 ?: `& ]Warning  FailedMount             41s (x8 over 1m)  kubelet, node01            MountVolume.WaitForAttach failed for volume "ceph-pv" : fail to check rbd image status with: (executable file not found in $PATH), rbd output: ()
* ^* U2 E& K+ T7 L) P: [% bWarning  FailedMount             0s                kubelet, node01            Unable to mount volumes for pod "ceph-pod1_default(14e3a07d-93a8-11e8-95f6-000c29b1ec26)": timeout expired waiting for volumes to attach or mount for pod "default"/"ceph-pod1". list of unmounted volumes=[ceph-vol1]. list of unattached volumes=[ceph-vol1 default-token-v9flt]
解决:node节点安装最新版的ceph-common解决该问题，ceph集群使用的是最新的mimic版本，而base源的版本太陈旧，故出现该问题

7.问题:
创建动态PV，PVC一直处于pending状态:
' J5 z) b- o. X5 F# g( P7 G# kubectl get pvc -n ceph
7 a* `) A* c; y  C0 q3 Z/ X1 _3 Z; ~$ ZNAME       STATUS   VOLUME  CAPACITY  ACCESS MODES  STORAGECLASS  AGE
9 ^8 k2 y* ]+ {1 a2 n% }/ Yceph-pvc   Pending                                     ceph-rbd    2m
# kubectl describe pvc -n ceph
......
5 \0 [; ]) ~+ z+ `. EWarning  ProvisioningFailed  27s   persistentvolume-controller  Failed to provision volume with StorageClass "ceph-rbd": failed to create rbd image: exit status 1, command output: 2018-07-31 11:10:33.395991 7faa3558b7c0 -1 did not load config file, using default settings.
rbd: extraneous parameter --image-feature
) z: u3 Q. j( ~& i" ?% S  n解决:
) c) F- ?+ p, V, L& n) N0 ~3 gpersistentvolume-controller 服务运行在master节点，受kube-controller-manager 控制，故master节点也需要安装ceph-common包

$ ?3 n2 [' Q% |! _, ]6 T  X

- l1 ^2 z9 F0 B- N

admin

[root@controller ~]# ceph health detail
! [3 ]: v3 z4 m. J7 _HEALTH_WARN application not enabled on 1 pool(s)
POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)
+ ~9 \+ w+ W* M3 O% z    application not enabled on pool 'nova-vms'
    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.
$ u/ E! M9 O4 B: x! F
# ceph osd pool application enable nova-vms rbd
enabled application 'rbd' on pool 'nova-vms'
6 `3 ]' |# i. ^; j% y. d$ N% t
* C& a4 t* I5 H- ]4 Y: o# ceph health
HEALTH_OK