Volume f6c2389d-44ca-4dce-9d71-361bbeb112cd did not finish being created even af

admin

Volume f6c2389d-44ca-4dce-9d71-361bbeb112cd did not finish being created even after we waited 865 seconds or 121 attempts. And its status is creating.


, n0 Z4 ?! b5 @9 h: A. _# _% e
/ Q: J, D9 G; h7 _) O) ^
修改nova-compute.conf文件


On each compute node, edit the Nova configuration file:
Raw
# vi /var/lib/config-data/nova_libvirt/etc/nova/nova.conf
Find the following two parameters and increase them:
Raw
  ], `( `% k* Y+ Xblock_device_allocate_retries=120  <---- Default is 60
block_device_allocate_retries_interval=6  <---- Default is 3
Restart the nova_compute container on each compute node:
2 c. K+ J- D' t! d& E7 nRaw
# docker restart nova_compute
Once all of the nova_compute containers have fully restarted, retry the instance creation.
NOTE: If editing the Nova configuration files is not feasible, then it is also possible to create the Cinder volume first, and then use it to boot the Nova instance. The following is an example:
Raw
# R+ Y) v# \4 a/ J& b% M# openstack volume create --size 60 --image big_image boot_volume
# openstack server create --flavor 2 --volume boot_volume big_server
& f4 C0 w" Y) k2 e0 Z: p! MApplying the changes through the templates
- In order to apply this through the templates, add the following to an environment file such as network-environment.yaml:
) Z$ g- I/ |7 `Raw
  ComputeParameters:
    nova::block_device_allocate_retries: 120
    nova::block_device_allocate_retries_interval: 6
, v0 ?  L9 I8 J3 r# O5 p  ControllerParameters:
    cinder::rpc_response_timeout: 1800

admin

block_device_allocate_retries = 180
9 `) V# I, t1 C; |4 f0 wblock_device_allocate_retries_interval = 12

admin

[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
. h+ k5 U' o- }0 H! I0 S$ B    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
* Q6 {* u) k, T) }- W9 V& c9 A    mgr: compute03(active, since 24h), standbys: compute02, compute01
  g6 W# W3 ~2 q+ R    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

. j' w/ v$ x! M& ]; o  M  data:
    pools:   7 pools, 1552 pgs
    objects: 8.26k objects, 42 GiB
    usage:   99 GiB used, 12 TiB / 13 TiB avail
$ ?1 K8 R* W0 l6 _, d: q8 U% h0 T    pgs:     1552 active+clean

; P0 `4 G' Z: k- ~* Y4 h  io:
3 L" v8 N# h# J& K" }1 ?0 Q6 x  N    client:   11 MiB/s rd, 1 op/s rd, 0 op/s wr

" |" _0 i0 n# G[root@compute01 ~]# ceph -s
1 h% I' F1 k7 r6 H- ]1 l  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

  services:
, ?! W" t0 u) U- u$ }    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
0 S7 O2 U. ?" R) A
  data:
3 f, C: k" ~$ X0 W    pools:   7 pools, 1552 pgs
    objects: 8.26k objects, 42 GiB
; L! d9 d' e/ i! x' H/ A    usage:   99 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
0 k" @1 ~1 L& ~- m+ ?0 T6 Z
  io:
: {9 s' A: V( h    client:   11 MiB/s rd, 1 op/s rd, 0 op/s wr
0 ?  r8 E5 b  |1 K' _  r  k

8 P7 Q9 P# A8 |$ E& {
主要原因是ceph的写入慢导致的问题，现在通过调整ceph cinder用户添加一个rx的权限，数据上来了。
9 n% T4 u+ N  R$ M: U, Z! n" ]# tallow rx pool=images

4 m$ ]8 r" \- s: o0 `" W$ g[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

# C+ Z; I! l& P3 V  services:
# b7 W' w# D3 N' i' l7 B: {3 d    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
4 f- `1 V2 q( t& v2 T! n& E& Q; b$ E( c5 W    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

  data:
    pools:   7 pools, 1552 pgs
$ W/ @6 K- X7 t    objects: 8.30k objects, 42 GiB
- `. R: _, l8 r' Y) r    usage:   99 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
6 t! S, U& U% ^4 t% G5 u
  io:
    client:   13 MiB/s rd, 12 MiB/s wr, 7 op/s rd, 15 op/s wr

0 B# R" J% R" u1 P[root@compute01 ~]# ceph -s
5 p9 z& G  E" `  u( I3 H3 K8 l4 ?! {  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
- ?4 w! h8 @( J: e6 @    health: HEALTH_OK
0 b. f' L2 c  ]5 `
. x& G. I% V, b+ m4 A/ E  services:
. n8 C: ?+ L, ^6 [- {    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
8 u7 m5 K# U# ], q  t' e    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

! ?# h- V$ z* |! p1 p  data:
    pools:   7 pools, 1552 pgs
    objects: 8.43k objects, 42 GiB
    usage:   100 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
8 r6 d" m7 n* r% ]& ^
  io:
    client:   13 MiB/s rd, 52 MiB/s wr, 24 op/s rd, 110 op/s wr
- c, \9 `9 y+ f' j( }4 D8 U
[root@compute01 ~]# ceph -s
% ?* h& |/ c: i3 L/ @  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

( U9 B* t# J, q4 f$ Z/ o# K, |( X" x8 S  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

& G  X) Y. P  k; h6 p# j  data:
    pools:   7 pools, 1552 pgs
5 [4 Y5 i$ \& c# \" @    objects: 8.43k objects, 42 GiB
9 B, ]1 A6 b! ^2 D+ y    usage:   100 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
* S, l1 m+ k+ S+ S) d' {& W
; S. I/ B* b  }% h% V, A/ s  io:
. U: c+ n. l+ b( |8 D3 ^- v    client:   13 MiB/s rd, 52 MiB/s wr, 24 op/s rd, 110 op/s wr

6 {0 U4 z8 H. S" M[root@compute01 ~]# ceph -s
* K: q  [# D( R" {) I$ R1 Z  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
/ x! l5 b! J) V    health: HEALTH_OK
0 W. u7 J; H9 V  v$ c# N# J
- d% O$ k: N# l( M9 v  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
5 `* f9 w4 t" a" B. j% n4 e    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

5 S8 G0 o+ A+ o" \1 n4 [% f  data:
    pools:   7 pools, 1552 pgs
+ p) I, `" r, N7 e5 h6 H. [8 w    objects: 8.43k objects, 42 GiB
! `. F) C# B: ]2 X4 j    usage:   100 GiB used, 12 TiB / 13 TiB avail
% R* F+ V$ _$ P5 S- M: I    pgs:     1552 active+clean

  io:
    client:   8.7 MiB/s rd, 52 MiB/s wr, 23 op/s rd, 110 op/s wr

[root@compute01 ~]# ceph -s
! Q& C" J, }8 U* u+ f; @  cluster:
' v! V6 ^2 c  u4 I1 H: s    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

1 i7 S1 J3 I/ J) F$ F$ g* x! N  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

  data:
/ u& S% s. r7 d6 X" o) z* |    pools:   7 pools, 1552 pgs
' i( W, a% f, s) H    objects: 8.50k objects, 43 GiB
    usage:   101 GiB used, 12 TiB / 13 TiB avail
0 z8 Q8 l; d& H7 ~, h9 H2 Q0 N  R" q, l    pgs:     1552 active+clean

  io:
    client:   8.7 MiB/s rd, 73 MiB/s wr, 23 op/s rd, 225 op/s wr
7 H4 R: ~3 v/ b
) C8 P' P5 W0 G* |: [1 V, q[root@compute01 ~]# ceph -s
  I2 W0 ]) I, v$ d" [  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
- p" B) Q3 T* d! W" w. n+ x) c5 U. \
. s  P, Q% {: W& W8 ?& ^  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
% g7 u7 S% d1 n    mgr: compute03(active, since 24h), standbys: compute02, compute01
& |" M  v9 ^% \5 q- u4 }    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
% n# {: N' z( G6 V' X3 z
  data:
    pools:   7 pools, 1552 pgs
    objects: 8.50k objects, 43 GiB
    usage:   101 GiB used, 12 TiB / 13 TiB avail
: k& z( E+ x3 ?8 `. `% W- B    pgs:     1552 active+clean

  io:
    client:   8.7 MiB/s rd, 73 MiB/s wr, 23 op/s rd, 225 op/s wr
- Y4 b- `& I9 }) V! T
[root@compute01 ~]# ceph -s
  cluster:
' f9 y! v& D& i' J8 ?9 N1 V    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
& N8 q6 Y. c" y9 c+ f6 x% B    mgr: compute03(active, since 24h), standbys: compute02, compute01
% l3 R8 G2 \$ o2 k( F+ n' e, W- m    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
8 p& b& ~9 S& D) E1 h: B& Z" K
; D, K8 P3 G* C) D, r  data:
% C1 n. Z, }) g* g. c! ?    pools:   7 pools, 1552 pgs
    objects: 8.56k objects, 43 GiB
" B2 ?: a  Y. `2 q' F! Q    usage:   101 GiB used, 12 TiB / 13 TiB avail
, B' \6 a& m4 u& Z9 o, V! n    pgs:     1552 active+clean
8 ^7 f+ E! v* J
  io:
- X; s+ \  `* ?6 X, D8 G: W3 r    client:   13 MiB/s rd, 92 MiB/s wr, 23 op/s rd, 267 op/s wr
: u# z) q; T) p( m% f1 I6 V+ J4 _
. F+ @% ~# O* e" l8 ^[root@compute01 ~]# ceph -s
% m! W/ T0 e. U; _$ s  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
; n9 m9 u0 H$ K* q    health: HEALTH_OK

! w8 I, r' B6 N  G1 k: P1 W' R  services:
9 e$ X9 M( I; K8 W' h9 m) O* B    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
. h" C1 a) u* @$ T5 r    mgr: compute03(active, since 24h), standbys: compute02, compute01
3 k1 t7 b0 A4 A6 B. H- `. R    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

2 g2 q7 c. v7 G* `/ _' t. x* L  data:
! C8 r5 H4 |  [. G    pools:   7 pools, 1552 pgs
, o+ z% F' ~+ C9 N1 U    objects: 8.68k objects, 43 GiB
    usage:   102 GiB used, 12 TiB / 13 TiB avail
* B# ?* r6 i9 ?    pgs:     1552 active+clean

: L: A) y" t3 l) F5 N' C  W% u+ j& x  io:
    client:   7.3 MiB/s rd, 39 MiB/s wr, 4 op/s rd, 32 op/s wr
& W4 S2 ]; \8 B. o. X- g! g
[root@compute01 ~]# ceph -s
4 F1 B, Y) O9 R8 y: ^2 r% S  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
" U4 S2 Z- l  A
  services:
: H3 b2 \$ G, |+ I6 R- |    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
! u6 @7 @/ y, s/ n: y    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

) v  M- @; e5 m! U  data:
    pools:   7 pools, 1552 pgs
/ d$ W" \, s4 [# l# T- w4 T/ }- V    objects: 8.68k objects, 43 GiB
    usage:   102 GiB used, 12 TiB / 13 TiB avail
8 p2 p6 K4 K/ M6 {: b% O( E    pgs:     1552 active+clean
; D, r# B! O# O
  io:
    client:   11 MiB/s rd, 15 MiB/s wr, 4 op/s rd, 7 op/s wr
% R( g' E' U/ O