|
|
2025-02-22 12:01:52.616 36892 ERROR cinder.volume.manager [None req-7ef5ffd0-64c7-4245-ae99-a626b4ece2dc - - - - - -] Cannot complete RPC initialization because driver isn't initialized properly.: cinder.exception.DriverNotInitialized: Volume driver not ready.
' _2 }0 M6 j8 A; W" O( K$ c; R: t( W" j. z) p) E2 W" J, ~0 }( r6 f
7 H' S7 P _; d+ L7 l8 r6 L
5 q p- d/ g6 t3 p1 r" `0 @2 i. [8 E v4 o
4 k! ]7 }" L( y1 Q$ p
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)5 ~& Q5 ^+ Q! O$ q
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager 6 G, `+ v& L- O' P% h1 Y
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager During handling of the above exception, another exception occurred:) M/ m w) B% }" `
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager
. D( t3 _$ t$ m5 ~& B# b" ~2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager Traceback (most recent call last): Y% q2 }0 k( E7 L. c8 S
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 525, in _init_host9 I/ A- z9 a! L6 ?; \# Z
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager self.driver.check_for_setup_error()9 j( H5 u! C2 r$ p- R3 d
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 545, in check_for_setup_error
" A* C8 p$ [ b3 A3 H2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager with RADOSClient(self):
2 q4 Z/ F# q4 K2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 270, in __init__. P1 q3 }% `, f' `
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager self.cluster, self.ioctx = driver._connect_to_rados(pool)8 e# A3 F5 p& r* \/ G
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 630, in _connect_to_rados
' r( ^/ \5 b) x( F0 }5 R2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager return _do_conn(pool, remote, timeout)
# X1 S& [ s6 u O, x& J2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 669, in _wrapper8 l! c3 q6 `3 u: V
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager return r(f, *args, **kwargs)
. v; ~% X4 Y+ z7 w/ q# K: b2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__2 v/ b7 E; }* I/ e" `/ L
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager do = self.iter(retry_state=retry_state)
; V! z% o; l6 c; l; ^0 t2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 372, in iter
1 |% b1 p* f- f# |/ p2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager raise retry_exc.reraise()9 I3 j$ b- v2 n1 v8 a
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 189, in reraise
" x6 [- D3 ?# n1 m2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager raise self.last_attempt.result()
- l: G: d4 g) Z2 D+ a2 I2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
* L% M1 h" W2 Z2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager return self.__get_result()$ y% U# J1 U) P/ j
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
. H" n# k2 b; q* a: U3 I2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager raise self._exception
4 `6 ~% `8 T% }, g2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
+ B o+ h: @$ A n2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager result = fn(*args, **kwargs)
3 I) ^; U: n( ~6 ]! p6 e4 V3 s7 h7 O; I2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 628, in _do_conn: ]* \5 i; j1 d' n( I, B/ i2 [3 ~
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager raise exception.VolumeBackendAPIException(data=msg)
* g6 L1 I# J0 b+ `2 b7 f v; R1 G2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster.! f$ N% ~& J2 D1 B
2025-02-22 12:06:53.209 37564 ERROR cinder.volume.manager % g" S& P$ Y1 z" o' I
2025-02-22 12:06:57.213 37564 INFO cinder.volume.manager [None req-6738215e-a136-4b82-a376-75651f4cb572 - - - - - -] Starting volume driver RBDDriver (1.3.0)
2 \( G( k/ d+ V: @; F! u1 ^2025-02-22 12:06:57.224 37564 ERROR cinder.volume.drivers.rbd [None req-6738215e-a136-4b82-a376-75651f4cb572 - - - - - -] Error connecting to ceph cluster.: rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)
4 Y Z( B3 L7 D' D9 l2 G3 x2025-02-22 12:06:57.224 37564 ERROR cinder.volume.drivers.rbd Traceback (most recent call last):
2 Q0 H$ q0 Y4 E2 C, u" M2025-02-22 12:06:57.224 37564 ERROR cinder.volume.drivers.rbd File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 621, in _do_conn
; }- ~7 x$ ^. D6 f2025-02-22 12:06:57.224 37564 ERROR cinder.volume.drivers.rbd client.connect()
8 B1 a. C. U6 X5 l2025-02-22 12:06:57.224 37564 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 690, in rados.Rados.connect
/ ]# u7 X5 G% J2 q K/ X2025-02-22 12:06:57.224 37564 ERROR cinder.volume.drivers.rbd rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)
9 D5 f4 g) q% y2025-02-22 12:06:57.224 37564 ERROR cinder.volume.drivers.rbd " U# F: ` B, ~, d4 h1 J/ C
2025-02-22 12:07:02.240 37564 ERROR cinder.volume.drivers.rbd [None req-6738215e-a136-4b82-a376-75651f4cb572 - - - - - -] Error connecting to ceph cluster.: rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)
" G3 x% g; d9 V. Q6 e9 R3 o2025-02-22 12:07:02.240 37564 ERROR cinder.volume.drivers.rbd Traceback (most recent call last):: J7 i' f, @1 T& s
2025-02-22 12:07:02.240 37564 ERROR cinder.volume.drivers.rbd File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 621, in _do_conn
2 |1 Z: `3 O1 W) F/ F$ Y2025-02-22 12:07:02.240 37564 ERROR cinder.volume.drivers.rbd client.connect()' M$ @4 w/ z; m' \+ M$ c
2025-02-22 12:07:02.240 37564 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 690, in rados.Rados.connect
' g( F( n* C: }) K2025-02-22 12:07:02.240 37564 ERROR cinder.volume.drivers.rbd rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)# s* a8 W3 r, q
2025-02-22 12:07:02.240 37564 ERROR cinder.volume.drivers.rbd 6 R3 a7 `6 c- E# R
2025-02-22 12:07:12.255 37564 ERROR cinder.volume.drivers.rbd [None req-6738215e-a136-4b82-a376-75651f4cb572 - - - - - -] Error connecting to ceph cluster.: rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)
1 K% y" E$ k# q1 M' V2025-02-22 12:07:12.255 37564 ERROR cinder.volume.drivers.rbd Traceback (most recent call last):
4 Y" \; {! x0 D* B7 `# b. o2025-02-22 12:07:12.255 37564 ERROR cinder.volume.drivers.rbd File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 621, in _do_conn
! S) k1 b% d0 R/ K2025-02-22 12:07:12.255 37564 ERROR cinder.volume.drivers.rbd client.connect()5 m9 |- N2 [9 E* K; W
2025-02-22 12:07:12.255 37564 ERROR cinder.volume.drivers.rbd File "rados.pyx", line 690, in rados.Rados.connect
/ ~" N. F; \5 y; ^2025-02-22 12:07:12.255 37564 ERROR cinder.volume.drivers.rbd rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)
" r7 j9 v1 i3 I4 Y/ L2025-02-22 12:07:12.255 37564 ERROR cinder.volume.drivers.rbd 6 }; D5 }1 C) s8 {4 U' ?
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager [None req-6738215e-a136-4b82-a376-75651f4cb572 - - - - - -] Failed to initialize driver.: cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster.; Z) @4 O: k: s; B4 X5 @
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager Traceback (most recent call last):
1 G6 c4 h f7 [2 Y; T, {1 Q' l' J2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 621, in _do_conn- m, R( F, A+ a& f" {
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager client.connect()
6 k+ |2 ^! ]1 @4 I! V! U3 h2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "rados.pyx", line 690, in rados.Rados.connect" a# O3 w8 V5 ^8 D" e
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)! J; @# I5 q/ B) _+ y2 h
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager ) E/ r# e" r2 Z1 |7 E/ K
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager During handling of the above exception, another exception occurred:- V+ B3 n. z# [+ \
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager
/ f9 o+ h7 t7 J2 |0 z5 u4 r$ ^2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager Traceback (most recent call last):" @& }9 B) j; r& ^" V8 B( R. S
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 525, in _init_host
& V% K# H6 H/ n; N0 V% X% Z2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager self.driver.check_for_setup_error()
, _. n6 e+ _* a) o4 F2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 545, in check_for_setup_error
5 C/ K. j: q) K& \4 ~1 N: t$ K8 s2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager with RADOSClient(self):
. X+ p: |3 N, F( I4 T2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 270, in __init__
; E7 b( g6 |2 N0 V2 D2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager self.cluster, self.ioctx = driver._connect_to_rados(pool)$ l* ^" I9 R& W) `' N) z- c
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 630, in _connect_to_rados
$ u6 |5 M. Q+ Z: u2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager return _do_conn(pool, remote, timeout)$ J7 K3 r( A1 C5 l* W; v! V9 N, x3 n
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 669, in _wrapper
- K& C, C2 I" \) o6 A$ s O2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager return r(f, *args, **kwargs)0 t& Y/ L* B9 \" L/ a! ^
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__: O, I3 M4 @/ L6 ]! \& u
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager do = self.iter(retry_state=retry_state)
7 l- H7 b6 v+ @. }5 e/ e2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 372, in iter
9 Y1 N' H* p9 O$ z9 r5 H# S4 Y h2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager raise retry_exc.reraise()* v1 I0 y8 l9 @8 _6 `+ U
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 189, in reraise
3 M" f! K' c2 o! w5 U5 W) q9 i; G9 Y2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager raise self.last_attempt.result()
4 ]+ |8 \) h# v1 q V2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result6 A% j! V: E* ^- I+ }5 z, l' N, F" j
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager return self.__get_result()" ?& D8 @3 w+ Y7 M; d- B5 P. Y: W
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result* D# ?4 @( H' y$ T( V- c- i# z
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager raise self._exception2 Y' z( ]; I: g% Q) E
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
1 I! p8 T2 b- L$ I5 G! V, B2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager result = fn(*args, **kwargs)3 x* H: Q$ U4 s+ M# z3 k5 I ^2 A
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 628, in _do_conn5 o* k. v @) |. T+ U
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager raise exception.VolumeBackendAPIException(data=msg)
0 c6 V2 L' ?2 y# S: _1 @, a2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error connecting to ceph cluster.* r- `7 C# U5 x+ @. ?# o/ n
2025-02-22 12:07:12.258 37564 ERROR cinder.volume.manager ]/ N& g9 n$ l) t8 W5 _6 [ W; r
2025-02-22 12:07:12.331 37564 INFO cinder.volume.manager [None req-6738215e-a136-4b82-a376-75651f4cb572 - - - - - -] Initializing RPC dependent components of volume driver RBDDriver (1.3.0)1 k: B* w2 u; D( C# M; {
2025-02-22 12:07:12.332 37564 ERROR cinder.volume.volume_utils [None req-6738215e-a136-4b82-a376-75651f4cb572 - - - - - -] Volume driver RBDDriver not initialized. b- w5 _8 h8 V: r9 b/ B5 H
2025-02-22 12:07:12.332 37564 ERROR cinder.volume.manager [None req-6738215e-a136-4b82-a376-75651f4cb572 - - - - - -] Cannot complete RPC initialization because driver isn't initialized properly.: cinder.exception.DriverNotInitialized: Volume driver not ready.1 Z/ G/ V& _: p( @2 U* l" R$ a) N3 U% U
2025-02-22 12:07:22.334 37564 ERROR cinder.service [-] Manager for service cinder-volume streamcontroller@ceph is reporting problems, not sending heartbeat. Service will appear "down".
- [# q6 K1 v" B2025-02-22 12:07:32.337 37564 ERROR cinder.service [-] Manager for service cinder-volume streamcontroller@ceph is reporting problems, not sending heartbeat. Service will appear "down".% d' n% ^( ]. Q. d& n1 a# b
2025-02-22 12:07:42.338 37564 ERROR cinder.service [-] Manager for service cinder-volume streamcontroller@ceph is reporting problems, not sending heartbeat. Service will appear "down".# h6 \; ~+ R& H7 Z/ Z+ l
2025-02-22 12:07:52.339 37564 ERROR cinder.service [-] Manager for service cinder-volume streamcontroller@ceph is reporting problems, not sending heartbeat. Service will appear "down"./ F4 X. o$ f( L( p* { x
2025-02-22 12:07:58.336 37564 WARNING cinder.volume.manager [None req-c6c8d5f3-ae9e-4485-9559-a2e528e8b5b9 - - - - - -] Update driver status failed: (config name ceph) is uninitialized.
4 [& _+ P/ ], R9 j: o" w2025-02-22 12:08:02.341 37564 ERROR cinder.service [-] Manager for service cinder-volume streamcontroller@ceph is reporting problems, not sending heartbeat. Service will appear "down".* q. D/ [0 [4 v9 ]* \
2025-02-22 12:08:12.342 37564 ERROR cinder.service [-] Manager for service cinder-volume streamcontroller@ceph is reporting problems, not sending heartbeat. Service will appear "down".1 `0 b. o: P1 w# d# H3 x
2025-02-22 12:08:22.344 37564 ERROR cinder.service [-] Manager for service cinder-volume streamcontroller@ceph is reporting problems, not sending heartbeat. Service will appear "down".6 t L0 c9 m7 c
3 S+ I' ]) ~9 \2 G8 _
0 z* e+ D( q( b
; _4 b. I, ^" N |
|