ceph 所有问题总结 application not enabled on 1 pool(s)

admin

1.问题:

# ceph health

HEALTH_WARN      application not enabled on 1 pool(s)

解决:

. F; U$ _# B  s. U1 Z

# ceph health detail

8 j  o& m. B$ X

HEALTH_WARN application not enabled on 1 pool(s)

2 V" P: }. P% @/ \+ q/ n% ]& C

POOL_APP_NOT_ENABLED application not enabled on 1 pool(s)

    application not enabled on pool 'kube'

# J7 u. W' s7 o* p

    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

# ceph osd pool application enable kube rbd

  enabled application 'rbd' on pool 'kube'

# ceph health

HEALTH_OK

! |. B, M4 V2 F; o( o5 u: u1 i

2.问题:

# ceph -s
# `8 Z6 n7 _0 {% I  cluster:
8 Q$ `$ Y. p: q" L% d+ [' e& ]    id:     e781a2e4-097d-4867-858d-bdbd3a264435
5 ~) F: R5 a; T. l& j    health: HEALTH_WARN
8 L6 G1 H9 c5 l: z% Q            clock skew detected on mon.ceph02, mon.ceph03

. d: }% o$ u5 @2 q: a4 q解决:

8 _# H  e( ~/ G) ]& }7 `1 h1 F####确认NTP服务是否正常工作
: p8 H/ f, E, }# systemctl status ntpd
####修改ceph配置中的时间偏差阈值
4 H; Y) ]/ z( P- z; U2 F# vim /etc/ceph/ceph.conf
  l) d0 C( a& Y- \###在global字段下添加:
1 |, y# \/ N2 J# g  rmon clock drift allowed = 2
! D+ K" m* R" w# X3 z  f+ ?mon clock drift warn backoff = 30   
####向需要同步的mon节点推送配置文件
- O+ U; g% M3 z1 b# cd /etc/ceph/
# ceph-deploy --overwrite-conf config  push ceph{01..03}
# G: n% @- E! M% c9 Z2 ~####重启mon服务并验证
# systemctl restart ceph-mon.target
" _9 W7 p; a- s# ceph -s
: I* e" a. _+ m. \) d! R  cluster:
    id:     e781a2e4-097d-4867-858d-bdbd3a264435
    health: HEALTH_OK
, x+ A0 \1 t, ?( m2 p) y5 |
: T6 P/ [/ D6 x2 @: @/ ^  t3.问题:
# rbd map abc/zhijian --id admin
rbd: sysfs write failed
* Z% O. S5 c# ^) q7 HRBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address

6 O. ^* o  i' b: {5 d" k解决:
由于kernel不支持块设备镜像的一些特性，所以映射失败
& z2 z! T9 e( z1 _5 a5 h, D# rbd feature disable abc/zhijian exclusive-lock, object-map, fast-diff, deep-flatten
( H4 h3 ?7 v2 C' t) w8 J' X# rbd info abc/zhijian
% B1 M# ]$ m& P4 K4 |0 B7 ]/ ?6 ~rbd image 'zhijian':
6 z$ s" z0 Q, N8 Y& ssize 1024 MB in 256 objects
order 22 (4096 kB objects)
  K9 T  G; u3 U) c$ C) `block_name_prefix: rbd_data.1011074b0dc51
7 `4 K2 k5 Y4 D/ q" R) M5 Lformat: 2
features: layering
1 b7 k) E& l8 \& W; yflags:
, K: G! u0 y1 j: i9 @- Mcreate_timestamp: Sun May  6 13:35:21 2018
# rbd map abc/zhijian --id admin
/dev/rbd0

9 E; C( H# S5 ]% @1 U3 q6 d4.问题:
# ceph osd pool delete cephfs_data
Error EPERM: WARNING: this will *PERMANENTLY DESTROY* all data stored in pool cephfs_data.  If you are *ABSOLUTELY CERTAIN* that is what you want, pass the pool name *twice*, followed by --yes-i-really-really-mean-it.
# ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it
! h+ W8 {) x# I# xError EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool

解决:
# tail -n 2 /etc/ceph/ceph.conf
. P; k- x: J. o2 L. r# O1 a[mon]
% g* c& T) p& nmon allow pool delete = true

4 ?6 S8 n( l# ^; B8 C6 |8 G; t/ A. g向需要同步的mon节点推送配置文件:
' T1 C2 y" {& |/ l! u4 o5 c# cd /etc/ceph/
# ceph-deploy --overwrite-conf config  push ceph{01..03}
" Q$ k( z, X8 o( k
重启mon服务并验证:
  K; ~: C9 v/ G& b8 r) @# systemctl restart ceph-mon.target
4 w: T9 z* p; \/ ?: S5 a# ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it
9 G0 b/ p4 p9 ]9 o' J' D" H" M* k0 @pool 'cephfs_data' removed
8 w8 ~2 \8 W' B% p4 D! m
8 x  i+ {8 \! U* k1 P5.问题:
# ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it
Error EBUSY: pool 'cephfs_data' is in use by CephFS
& D9 @& H6 X6 X. p8 ^* [
解决:
# ceph fs ls
  F% V3 m+ q( F% jname: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
#  ceph fs rm cephfs --yes-i-really-mean-it
Error EINVAL: all MDS daemons must be inactive before removing filesystem
: v9 Z6 I& q% L" C% f8 u1 b. W# systemctl stop ceph-mds.target
. n9 }$ F; \  B: [6 y+ U9 o, Q9 K# ceph fs rm cephfs
& K% _! K- q* o/ e* V' LError EPERM: this is a DESTRUCTIVE operation and will make data in your filesystem permanently inaccessible.  Add --yes-i-really-mean-it if you are sure you wish to continue.
3 p3 A% e* m  U/ l# ceph fs rm cephfs --yes-i-really-mean-it
# ceph fs ls
1 X" Q2 O9 V5 g2 |; A2 h& n  @No filesystems enabled

6.问题:
! Z! X1 z+ c6 s! ~0 i使用静态PV创建pod，pod一直处于ContainerCreating状态:
# kubectl get pod ceph-pod1
NAME        READY     STATUS              RESTARTS   AGE
ceph-pod1   0/1       ContainerCreating   0          10s
2 V: Z8 n! V, S8 e/ }' |......
) u# h8 m& X# ]" M) K3 v7 ^% E) k4 h# kubectl describe pod ceph-pod1
  C+ @$ R! S$ a. P2 o' w" \: K) oWarning  FailedMount             41s (x8 over 1m)  kubelet, node01            MountVolume.WaitForAttach failed for volume "ceph-pv" : fail to check rbd image status with: (executable file not found in $PATH), rbd output: ()
0 A, m  H0 ]- @+ t; l; g4 v, zWarning  FailedMount             0s                kubelet, node01            Unable to mount volumes for pod "ceph-pod1_default(14e3a07d-93a8-11e8-95f6-000c29b1ec26)": timeout expired waiting for volumes to attach or mount for pod "default"/"ceph-pod1". list of unmounted volumes=[ceph-vol1]. list of unattached volumes=[ceph-vol1 default-token-v9flt]
解决:node节点安装最新版的ceph-common解决该问题，ceph集群使用的是最新的mimic版本，而base源的版本太陈旧，故出现该问题

$ x3 @. Q+ l, H. H2 V* F7.问题:
7 ]! p9 C' ~9 H; [; ]# p创建动态PV，PVC一直处于pending状态:
# kubectl get pvc -n ceph
" I  A. I9 \* k& c- f$ ONAME       STATUS   VOLUME  CAPACITY  ACCESS MODES  STORAGECLASS  AGE
, \& M4 i: o: g5 m8 }( E* l. F1 g- C# sceph-pvc   Pending                                     ceph-rbd    2m
# kubectl describe pvc -n ceph
8 w  z% f5 F! j% |3 w......
Warning  ProvisioningFailed  27s   persistentvolume-controller  Failed to provision volume with StorageClass "ceph-rbd": failed to create rbd image: exit status 1, command output: 2018-07-31 11:10:33.395991 7faa3558b7c0 -1 did not load config file, using default settings.
rbd: extraneous parameter --image-feature
" {. {" \) X2 i/ n& O+ g解决:
& q" j( Y  ^7 S4 C$ p0 Mpersistentvolume-controller 服务运行在master节点，受kube-controller-manager 控制，故master节点也需要安装ceph-common包

4 h6 Q$ |' p+ _1 z0 O$ r: F( S

% z# x  g4 b! N6 `7 n1 k  ~

admin

[root@controller ~]# ceph health detail
6 _7 j- v- j! [+ EHEALTH_WARN application not enabled on 1 pool(s)
& K+ o$ V$ L, V. F0 k; j; ePOOL_APP_NOT_ENABLED application not enabled on 1 pool(s)
    application not enabled on pool 'nova-vms'
    use 'ceph osd pool application enable <pool-name> <app-name>', where <app-name> is 'cephfs', 'rbd', 'rgw', or freeform for custom applications.

# ceph osd pool application enable nova-vms rbd
enabled application 'rbd' on pool 'nova-vms'
- Z+ s1 U, n) u, a
# ceph health
/ a9 O( J7 a" C6 B. o6 MHEALTH_OK