Volume f6c2389d-44ca-4dce-9d71-361bbeb112cd did not finish being created even af

admin

Volume f6c2389d-44ca-4dce-9d71-361bbeb112cd did not finish being created even after we waited 865 seconds or 121 attempts. And its status is creating.



, T$ c, D. V& V0 o; j
修改nova-compute.conf文件

. _! j& u' h# Y3 l: N
! Q) v3 b" F# u. g8 t4 O" ]On each compute node, edit the Nova configuration file:
Raw
# vi /var/lib/config-data/nova_libvirt/etc/nova/nova.conf
: J2 }( H1 J  y2 T- yFind the following two parameters and increase them:
0 m$ l: Z. M: D9 YRaw
block_device_allocate_retries=120  <---- Default is 60
block_device_allocate_retries_interval=6  <---- Default is 3
Restart the nova_compute container on each compute node:
+ D1 X/ P0 z! i0 Z% ^7 L3 I" zRaw
# docker restart nova_compute
Once all of the nova_compute containers have fully restarted, retry the instance creation.
5 p0 R2 y, V4 K7 ~! z2 _4 X! YNOTE: If editing the Nova configuration files is not feasible, then it is also possible to create the Cinder volume first, and then use it to boot the Nova instance. The following is an example:
Raw
2 [" |& U6 c# I! v6 m) y! d$ E& z# openstack volume create --size 60 --image big_image boot_volume
# openstack server create --flavor 2 --volume boot_volume big_server
Applying the changes through the templates
/ g4 k7 }9 |$ n& ^% c- J- In order to apply this through the templates, add the following to an environment file such as network-environment.yaml:
9 C5 r) g$ i! E) I! Q8 M( B2 oRaw
! f8 s% K9 b. s% X: n4 T) t( o  ComputeParameters:
: Z: ^/ V; [  c  h3 t    nova::block_device_allocate_retries: 120
    nova::block_device_allocate_retries_interval: 6
: g( k& B$ @8 _) W4 j  ControllerParameters:
    cinder::rpc_response_timeout: 1800
% W- F  [" q! Q# q2 T

admin

block_device_allocate_retries = 180
block_device_allocate_retries_interval = 12

admin

[root@compute01 ~]# ceph -s
  cluster:
3 q1 w3 O5 g8 k: [0 p! }1 e! U    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

: I; l; m9 v4 g- m  z  services:
4 \* C, I4 |: i9 G7 m    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
6 N& I3 Y9 U. L4 G) S    mgr: compute03(active, since 24h), standbys: compute02, compute01
: f4 y2 s1 V# z5 A1 |    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

  data:
    pools:   7 pools, 1552 pgs
  Q' N" j( {1 M5 Z0 k    objects: 8.26k objects, 42 GiB
5 C' ]" [, m+ K: _    usage:   99 GiB used, 12 TiB / 13 TiB avail
) g; ?  B; V+ {    pgs:     1552 active+clean

( }4 f) S3 j: l( G. v4 ?* u) l- s  v  io:
6 a- o- [' z2 O4 A    client:   11 MiB/s rd, 1 op/s rd, 0 op/s wr

# u5 Q- Y, `9 Y( @[root@compute01 ~]# ceph -s
7 R+ h6 V0 {9 S# B' n2 J( ?# L  cluster:
$ P0 Q( t) Z2 F8 ?+ \4 ]    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
& g! o- k) L. K, A- P
  services:
0 e4 \( U" B6 P0 q1 u/ B    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
/ j% A+ w) C2 l2 R5 z5 s1 C    mgr: compute03(active, since 24h), standbys: compute02, compute01
3 t; t$ V: I$ B( k" B3 `    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

  data:
    pools:   7 pools, 1552 pgs
    objects: 8.26k objects, 42 GiB
    usage:   99 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean

  c, h- y0 G! ]7 c  io:
    client:   11 MiB/s rd, 1 op/s rd, 0 op/s wr
; o) }7 A7 o; D+ ]" x2 ]9 e


主要原因是ceph的写入慢导致的问题，现在通过调整ceph cinder用户添加一个rx的权限，数据上来了。
" j1 T0 M: {$ p' o- k) Uallow rx pool=images
/ E) y+ Y2 [; @
+ q7 |- W. @1 L% W2 u[root@compute01 ~]# ceph -s
  cluster:
6 k0 X  |0 G. E2 C# ?    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
& m  v) p3 D( ?* C# p2 ]( k    health: HEALTH_OK

9 w1 O. w; K( }# P5 x  services:
* N5 B5 |5 g4 L- N    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
% \. P; p# f" Q: n    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
7 Y) Y$ D% K5 e: L8 i; O7 z
  data:
# K5 X; g) Y4 n% a+ `" c8 X    pools:   7 pools, 1552 pgs
    objects: 8.30k objects, 42 GiB
    usage:   99 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean

  io:
    client:   13 MiB/s rd, 12 MiB/s wr, 7 op/s rd, 15 op/s wr
; m- m- V# ?: X
  h+ A) O& A: Q[root@compute01 ~]# ceph -s
7 r8 o* b5 @: A! p! x9 i  cluster:
- s/ m5 D. M+ F" x# K% Q    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK
6 i: G1 ^/ Q5 z1 z/ X# |
  services:
7 g/ V( _$ K& J* [( a& D8 k    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
2 n# x1 h3 E6 U" m    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
) m; z6 m$ ~% h$ _- Z
  data:
7 u. H! z5 g0 T9 E    pools:   7 pools, 1552 pgs
    objects: 8.43k objects, 42 GiB
/ N6 D3 ~& }8 S! r9 x1 M    usage:   100 GiB used, 12 TiB / 13 TiB avail
) d( P! C/ M$ r    pgs:     1552 active+clean
0 g7 e3 }) ^* T; d' _: i3 E4 k+ C
5 H) K3 n% l' f  [( b, s$ `  io:
    client:   13 MiB/s rd, 52 MiB/s wr, 24 op/s rd, 110 op/s wr
' ^$ ?. [& D! `7 s
[root@compute01 ~]# ceph -s
6 _9 t, @. B- D9 n  s  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
% W* a- Q  I" _- }" G    health: HEALTH_OK
! ?9 [  B/ S2 N& o3 A: x2 C
  services:
2 v; I8 P8 u( J    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
" C; _! p8 v' v, l) L. }( {    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
, Q1 C" G6 M" R( W" |% d3 x0 z) R
4 H* f, `3 m$ y! O! S% M  data:
/ T1 H# Z/ g: M4 X; g7 Q- R; e    pools:   7 pools, 1552 pgs
    objects: 8.43k objects, 42 GiB
1 p, W! e" O! S. R$ M4 m    usage:   100 GiB used, 12 TiB / 13 TiB avail
, D8 D4 K4 L& O* m, j    pgs:     1552 active+clean

" A, T/ ^/ E) a( b% R, @- p6 b  io:
    client:   13 MiB/s rd, 52 MiB/s wr, 24 op/s rd, 110 op/s wr
1 Q7 W  t, Q/ `& I: p# K, P
8 x: G2 @5 T5 g[root@compute01 ~]# ceph -s
2 n7 F& Z* l) P/ k! [( F  cluster:
/ O! S8 t# o" l- r& B/ Q4 m    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
6 ~+ t4 D: V' P5 k9 y' I& g    health: HEALTH_OK

5 ^; ]% a$ b  i  n  services:
$ x& C5 @& l! D  T3 W1 p7 p    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
8 z, O# \' `7 Y5 z; o+ b    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
+ t/ s7 w5 h/ \* }
  data:
6 |8 P8 y6 b' o* y7 r    pools:   7 pools, 1552 pgs
8 I, P8 v& @- J. H+ f/ {    objects: 8.43k objects, 42 GiB
9 l' U, [6 B# n; |/ ^* p% n5 Y8 P    usage:   100 GiB used, 12 TiB / 13 TiB avail
# X5 w1 K% F8 p' h* X. J    pgs:     1552 active+clean

  io:
    client:   8.7 MiB/s rd, 52 MiB/s wr, 23 op/s rd, 110 op/s wr
3 |2 F; r, @) E3 r) q
4 r% I. |4 H9 Q; [6 o2 ?- j[root@compute01 ~]# ceph -s
* y) }  S3 w& K! a$ n# }2 c) N+ O  cluster:
' W9 W; J5 {) [$ B2 R2 j0 i- f4 ?    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
% n$ k: C3 z* z* ~! w3 W. J- i- }    mgr: compute03(active, since 24h), standbys: compute02, compute01
- f& s2 A5 ?% x    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)

- A6 |" I  j: d' w  O; X$ k  data:
    pools:   7 pools, 1552 pgs
    objects: 8.50k objects, 43 GiB
    usage:   101 GiB used, 12 TiB / 13 TiB avail
5 m' E: l8 G: ]$ S* }# D    pgs:     1552 active+clean
, L3 s* t4 U& y& E. T1 I, C
  io:
3 G7 m- Y# U1 ?6 I# D. o3 ?# b    client:   8.7 MiB/s rd, 73 MiB/s wr, 23 op/s rd, 225 op/s wr
  ~, ?" T3 T3 O
[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
: I" [* N' c" ?* K! \! p) U    health: HEALTH_OK

& V: |: `0 w1 i# i$ A  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
" h- `( {% q/ f3 _8 |; U3 ~    mgr: compute03(active, since 24h), standbys: compute02, compute01
* V9 s5 l7 U2 K; Z9 e    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
8 ?* i% {$ ?8 O* i
+ F0 N( Q7 }; ]' J4 z1 F4 F  data:
    pools:   7 pools, 1552 pgs
8 w( V# a4 F; o    objects: 8.50k objects, 43 GiB
    usage:   101 GiB used, 12 TiB / 13 TiB avail
: b6 U6 N- M- V, \9 ^6 R( o    pgs:     1552 active+clean

  io:
+ H$ |0 d" U0 [! Y2 \9 p# N    client:   8.7 MiB/s rd, 73 MiB/s wr, 23 op/s rd, 225 op/s wr
( o' i$ E2 n% P# {/ S
[root@compute01 ~]# ceph -s
  cluster:
    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

  services:
: K& t8 O2 k( p5 J8 D0 [' A" Y( p4 U    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
; Q  \, @4 G5 K9 Y% c9 ^3 |( s    mgr: compute03(active, since 24h), standbys: compute02, compute01
2 H1 ~  q! ^! ?# ~0 v1 n4 ]    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
  h3 m' M) V, @$ l1 K* y9 l8 C
2 _: |: ?' r6 T  data:
    pools:   7 pools, 1552 pgs
# z7 L% }3 j7 x1 a0 f6 Q8 U    objects: 8.56k objects, 43 GiB
    usage:   101 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
. i7 S; V. b# f
; P& m2 {0 f8 @& I2 @; d  io:
$ z  k6 \+ U9 r: t3 z    client:   13 MiB/s rd, 92 MiB/s wr, 23 op/s rd, 267 op/s wr
% X6 y8 Z/ V) p/ R
& G, I$ @4 f7 x[root@compute01 ~]# ceph -s
4 a+ s1 g! F* [0 r$ I+ `  cluster:
5 W% ^) |* [9 J/ E  p2 q" C3 Q5 C4 @3 D' `    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
. ]5 m2 @7 M% c    mgr: compute03(active, since 24h), standbys: compute02, compute01
: _) @* f) k" P) k) v    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
: z6 B4 u$ n( L# b4 W5 Z9 h
  data:
. d+ g; W& G* w1 [    pools:   7 pools, 1552 pgs
" P9 C- p; ]) {2 b2 {- d    objects: 8.68k objects, 43 GiB
7 I) k+ T5 ]- [7 Z, }# G$ F# @    usage:   102 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean

& }: S0 C7 }9 m0 ], ^. t) o  io:
! o) G+ f1 O! [) w    client:   7.3 MiB/s rd, 39 MiB/s wr, 4 op/s rd, 32 op/s wr
7 X( ~. N  l& e' f
[root@compute01 ~]# ceph -s
  cluster:
. Q2 w" ^7 f1 G( P    id:     2af51d38-db90-4a57-a43d-ea9f6ebd7482
    health: HEALTH_OK

, E, O! T: z: r) B  services:
    mon: 3 daemons, quorum compute01,compute02,compute03 (age 4h)
    mgr: compute03(active, since 24h), standbys: compute02, compute01
    osd: 14 osds: 14 up (since 7h), 14 in (since 24h)
7 j7 J; N) e) G7 ~, W% ?
0 \, f! j1 t6 F7 c# z) m! }  data:
. X. A1 V* W1 ?, c7 u% ]    pools:   7 pools, 1552 pgs
7 m$ H! Q& K7 h, i3 H0 K8 B6 G7 x    objects: 8.68k objects, 43 GiB
    usage:   102 GiB used, 12 TiB / 13 TiB avail
    pgs:     1552 active+clean
6 T$ N2 D% q& {0 Y% l# w! x$ ?$ O" ~
$ J$ W8 O$ ]; \" H6 E  io:
    client:   11 MiB/s rd, 15 MiB/s wr, 4 op/s rd, 7 op/s wr