rabbit [-] A recoverable connection/channel error occurred,trying to reconnect:

admin

2021-10-20 13:54:27.337 56 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out
, c) d3 c  x/ W) W) A* F- b: c2021-10-20 13:54:37.247 41 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out
/ V' k5 }& v4 F6 i/ x

4 ~. s0 F; \4 [6 ?^C
[root@compute03 ~]# tail -f /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-volume.log
2021-10-20 13:54:11.270 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [5a4f1099-767d-49a4-bf4d-78ca8ac423c7] AMQP server on 192.168.6.236:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2021-10-20 13:54:11.270 56 ERROR oslo.messaging._drivers.impl_rabbit [-] [f34b98b1-d307-4bec-914e-8c96e5b60060] AMQP server on 192.168.6.236:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
2021-10-20 13:54:11.292 56 ERROR oslo.messaging._drivers.impl_rabbit [-] [5502af9b-3cde-4c0c-a4a7-9d700f3182d5] AMQP server on 192.168.6.236:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out
- [( R! d- [( J5 H6 K% `2021-10-20 13:54:12.325 41 INFO oslo.messaging._drivers.impl_rabbit [-] [bb4aede6-57e0-4e7f-9479-daede2b3d452] Reconnected to AMQP server on 192.168.6.234:5672 via [amqp] client with port 53456.
( D- X5 Y3 D9 }5 y) r2021-10-20 13:54:12.331 56 INFO oslo.messaging._drivers.impl_rabbit [-] [5502af9b-3cde-4c0c-a4a7-9d700f3182d5] Reconnected to AMQP server on 192.168.6.234:5672 via [amqp] client with port 53484.
2021-10-20 13:54:12.331 41 INFO oslo.messaging._drivers.impl_rabbit [-] [5a4f1099-767d-49a4-bf4d-78ca8ac423c7] Reconnected to AMQP server on 192.168.6.234:5672 via [amqp] client with port 53468.
9 ^* ~3 g& T! W4 T' F! v2021-10-20 13:54:12.334 56 INFO oslo.messaging._drivers.impl_rabbit [-] [f34b98b1-d307-4bec-914e-8c96e5b60060] Reconnected to AMQP server on 192.168.6.234:5672 via [amqp] client with port 53464.
+ t  o% K, j! m& {2021-10-20 13:54:17.228 41 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Server unexpectedly closed connection
2021-10-20 13:54:27.337 56 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out
2021-10-20 13:54:37.247 41 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out
2 a+ m" p, D, v0 I" s+ i0 h# i% p! G
  O$ s4 h- i. p解决办法：
, V( ~$ D( w) N: J- q$ i, s, P在cinder的配置文件中添加参数：
cinder.conf文件：

[DEFAULT]
" b8 V$ n) I. e6 J' V5 J' K
rabbit_max_retries = 5     #设置最大连接数为5
heartbeat_timeout_threshold = 0   #
3 l* L1 }7 q. Q! S3 s. j0 e
[oslo_messaging_notifications]
$ F2 o4 y3 v$ u& F5 drabbit_max_retries = 5
heartbeat_timeout_threshold = 0
5 k4 n& y, r3 |& D" V5 P
5 {0 |. V/ D+ b8 H1 \3 |重启cinder-volume服务，
systemctl restart cinder-volumes.service  
5 ^6 P) W- \( o
  ^+ z3 I$ L% G' ^
nova的参数也这样配置：
7 M  i6 [6 N3 W# P4 cnova.conf文件：
9 E; k2 y- c; K% P7 l/ L) ^[DEFAULT]
2 T+ h$ M" @# G- v6 E
rabbit_max_retries = 5     #设置最大连接数为5
heartbeat_timeout_threshold = 0   #

# 禁止heartbeat check

[oslo_messaging_notifications]
0 U& k7 w, f! B. y$ {rabbit_max_retries = 5
6 Z0 g& ^2 E0 C# x5 A0 uheartbeat_timeout_threshold = 0
重启nova-compute服务。
/ J' P6 V8 p. @$ S$ J( z重启cinder-volume服务，
systemctl restart openstack-nova-compute.service
- d0 G/ U6 c, ~1 N/ x8 z
: l7 \5 |5 a5 n7 H) q# g发现问题解决了：
+ F& G4 Y8 {( k5 O% j1 k: i; M
4 Y9 x; X1 t' b! a9 ?$ z

admin

[root@compute01 ~]# tail -f /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-volume.log
/ a+ m+ u4 ^5 K; p5 s  m2021-10-20 14:36:20.292 48 INFO cinder.keymgr.migration [req-ba591a1e-a6ea-4935-b630-e5c1e8a7f733 - - - - -] Not migrating encryption keys because the ConfKeyManager's fixed_key is not in use.
2021-10-20 14:36:20.326 48 WARNING oslo_config.cfg [req-957fc5d3-7162-4547-b4a3-de76221269f7 - - - - -] Option "rabbit_max_retries" from group "DEFAULT" is deprecated. Use option "rabbit_max_retries" from group "oslo_messaging_rabbit".
% A  o% d0 L$ A$ A- n  l7 ]" {2021-10-20 14:36:20.326 48 WARNING oslo_config.cfg [req-957fc5d3-7162-4547-b4a3-de76221269f7 - - - - -] Option "rabbit_max_retries" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2021-10-20 14:36:20.354 48 INFO cinder.volume.manager [req-957fc5d3-7162-4547-b4a3-de76221269f7 - - - - -] Driver initialization completed successfully.
6 d( o0 v/ D( k1 o, W2021-10-20 14:36:20.362 34 INFO cinder.volume.manager [req-9e80c786-e527-4af4-b4cf-ff57496dd21c - - - - -] Initializing RPC dependent components of volume driver RBDDriver (1.2.0)
2021-10-20 14:36:20.366 48 INFO cinder.manager [req-957fc5d3-7162-4547-b4a3-de76221269f7 - - - - -] Initiating service 11 cleanup
# z# |6 W6 v/ e0 V9 j2021-10-20 14:36:20.371 48 INFO cinder.manager [req-957fc5d3-7162-4547-b4a3-de76221269f7 - - - - -] Service 11 cleanup completed.
2021-10-20 14:36:20.475 48 INFO cinder.volume.manager [req-957fc5d3-7162-4547-b4a3-de76221269f7 - - - - -] Initializing RPC dependent components of volume driver RBDDriver (1.2.0)
2021-10-20 14:36:20.530 34 INFO cinder.volume.manager [req-9e80c786-e527-4af4-b4cf-ff57496dd21c - - - - -] Driver post RPC initialization completed successfully.
+ j# w, y5 b# J2021-10-20 14:36:20.625 48 INFO cinder.volume.manager [req-957fc5d3-7162-4547-b4a3-de76221269f7 - - - - -] Driver post RPC initialization completed successfully.
9 j9 @( M; p6 ~$ Z9 Q0 N: l1 w2021-10-20 14:40:36.808 34 INFO cinder.rpc [req-895aec0a-97d7-4a44-a3bf-cffb6038f36c 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Automatically selected cinder-backup objects version 1.35 as minimum service version.
2021-10-20 14:40:36.825 34 INFO cinder.rpc [req-895aec0a-97d7-4a44-a3bf-cffb6038f36c 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Automatically selected cinder-backup RPC version 2.1 as minimum service version.
2021-10-20 14:40:36.845 34 INFO cinder.rpc [req-895aec0a-97d7-4a44-a3bf-cffb6038f36c 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Automatically selected cinder-volume objects version 1.35 as minimum service version.
3 ~, z- u; N" Y- |; L2021-10-20 14:40:36.859 34 INFO cinder.rpc [req-895aec0a-97d7-4a44-a3bf-cffb6038f36c 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Automatically selected cinder-volume RPC version 3.15 as minimum service version.
2021-10-20 14:40:38.165 34 INFO cinder.volume.flows.manager.create_volume [req-895aec0a-97d7-4a44-a3bf-cffb6038f36c 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Volume 551a5f24-be37-401a-a607-2135c87812b6: being created as image with specification: {'status': u'creating', 'image_location': (None, [{u'url': u'rbd://31403b11-8a1e-432f-876e-5a2c852f9dcc/images/cceec6e8-29f7-4469-98d1-44131268b88b/snap', u'metadata': {}}]), 'volume_size': 10, 'volume_name': u'volume-551a5f24-be37-401a-a607-2135c87812b6', 'image_id': u'cceec6e8-29f7-4469-98d1-44131268b88b', 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fc1a0e18690>, 'image_meta': {u'status': u'active', u'file': u'/v2/images/cceec6e8-29f7-4469-98d1-44131268b88b/file', u'name': u'CentOS-7.9', u'tags': [], u'container_format': u'bare', u'created_at': datetime.datetime(2021, 10, 20, 5, 43, 50, tzinfo=<iso8601.Utc>), u'disk_format': u'raw', u'locations': [{u'url': u'rbd://31403b11-8a1e-432f-876e-5a2c852f9dcc/images/cceec6e8-29f7-4469-98d1-44131268b88b/snap', u'metadata': {}}], u'visibility': u'public', u'updated_at': datetime.datetime(2021, 10, 20, 5, 46, 52, tzinfo=<iso8601.Utc>), u'owner': u'fce718dfba31472a940816a41b8aed2a', u'protected': False, u'id': u'cceec6e8-29f7-4469-98d1-44131268b88b', u'min_ram': 0, u'checksum': u'29dbdf37822b1717bab1128a29d47945', u'min_disk': 0, u'virtual_size': None, 'properties': {u'cinder_img_volume_type': u'volumes', u'hw_qemu_guest_agent': u'yes', u'owner_specified.openstack.sha256': u'd3862804c0bad38564e1e108245098a82a43b9eda5679270ebf4965d6d4d9278', u'owner_specified.openstack.md5': u'29dbdf37822b1717bab1128a29d47945', u'owner_specified.openstack.object': u'images/CentOS-7.9', u'hw_ovirt_guest_agent': u'yes'}, u'size': 4294967296}}
2021-10-20 14:40:38.793 34 INFO cinder.volume.flows.manager.create_volume [req-895aec0a-97d7-4a44-a3bf-cffb6038f36c 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Volume volume-551a5f24-be37-401a-a607-2135c87812b6 (551a5f24-be37-401a-a607-2135c87812b6): created successfully
, y5 U( I# v* O2021-10-20 14:40:38.803 34 INFO cinder.volume.manager [req-895aec0a-97d7-4a44-a3bf-cffb6038f36c 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Created volume successfully.
# [1 [5 O) W7 z0 W9 a5 |2021-10-20 14:41:50.734 48 ERROR oslo_db.sqlalchemy.exc_filters [req-87ab34a8-80be-49a4-b720-442a9eef3968 - - - - -] DB exception wrapped.: ResourceClosedError: This Connection is closed
2021-10-20 14:41:50.734 48 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
* h! j( ?% n! c1 i9 T/ s" }: [2021-10-20 14:41:50.734 48 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
2021-10-20 14:41:50.734 48 ERROR oslo_db.sqlalchemy.exc_filters     return self._revalidate_connection()
2021-10-20 14:41:50.734 48 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
2 ]# _% Q0 w1 N, M( N& U( ^4 |2021-10-20 14:41:50.734 48 ERROR oslo_db.sqlalchemy.exc_filters     raise exc.ResourceClosedError("This Connection is closed")
2021-10-20 14:41:50.734 48 ERROR oslo_db.sqlalchemy.exc_filters ResourceClosedError: This Connection is closed
2021-10-20 14:41:50.734 48 ERROR oslo_db.sqlalchemy.exc_filters
+ p& b# w8 N. l6 X2021-10-20 14:41:50.737 48 WARNING py.warnings [req-87ab34a8-80be-49a4-b720-442a9eef3968 - - - - -] /usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py:74: SAWarning: An exception has occurred during handling of a previous exception.  The previous exception is:
) O7 Q* i8 \; Y% i9 W <class 'sqlalchemy.exc.ResourceClosedError'> This Connection is closed

6 G8 s; @4 ?8 i+ l4 O3 T  "is:\n %s %s\n" % (self._exc_info[0], self._exc_info[1]))
: DBAPIError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
2021-10-20 14:41:50.737 48 ERROR cinder.service [req-87ab34a8-80be-49a4-b720-442a9eef3968 - - - - -] Exception encountered: : DBAPIError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
3 G6 q) z6 t3 ^; Q! i! o2021-10-20 14:41:50.737 48 ERROR cinder.service Traceback (most recent call last):
& `2 e6 s& c9 M, b2 E9 n2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 482, in report_state
6 K! Q9 f' B8 N) I& e2021-10-20 14:41:50.737 48 ERROR cinder.service     service_ref.save()
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/objects/service.py", line 181, in save
4 t! @$ `" A  C2021-10-20 14:41:50.737 48 ERROR cinder.service     db.service_update(self._context, self.id, updates)
, t' S# O/ e$ f" k  o0 d2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 154, in service_update
/ `6 w$ ?1 ]( S; C8 q2021-10-20 14:41:50.737 48 ERROR cinder.service     return IMPL.service_update(context, service_id, values)
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 164, in wrapper
2021-10-20 14:41:50.737 48 ERROR cinder.service     return f(*args, **kwargs)
9 a, u; E) Z# e' a, S- q2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2021-10-20 14:41:50.737 48 ERROR cinder.service     ectxt.value = e.inner_exc
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
9 E2 C- }5 Z1 w2021-10-20 14:41:50.737 48 ERROR cinder.service     self.force_reraise()
8 q& ^5 V$ m4 a% y  Q3 g2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
0 I- \% R1 J, Y7 `5 j1 b% \2021-10-20 14:41:50.737 48 ERROR cinder.service     six.reraise(self.type_, self.value, self.tb)
0 a! M: u4 i$ e! W2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2021-10-20 14:41:50.737 48 ERROR cinder.service     return f(*args, **kwargs)
1 @) O! q" q! U& X2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 560, in service_update
2021-10-20 14:41:50.737 48 ERROR cinder.service     result = query.update(values)
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3369, in update
" L. K; ]* N1 m$ Q2021-10-20 14:41:50.737 48 ERROR cinder.service     update_op.exec_()
* ~. C; V3 u& d- U' ]/ k2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1325, in exec_
3 h( A& s& y8 n2021-10-20 14:41:50.737 48 ERROR cinder.service     self._do_exec()
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1492, in _do_exec
2021-10-20 14:41:50.737 48 ERROR cinder.service     self._execute_stmt(update_stmt)
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1332, in _execute_stmt
2021-10-20 14:41:50.737 48 ERROR cinder.service     mapper=self.mapper)
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1176, in execute
0 q+ q. E6 b2 x$ ~2021-10-20 14:41:50.737 48 ERROR cinder.service     bind, close_with_result=True).execute(clause, params or {})
- c. I% R% n6 a  f2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2021-10-20 14:41:50.737 48 ERROR cinder.service     return meth(self, multiparams, params)
, z  S8 c1 {2 N( a2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2021-10-20 14:41:50.737 48 ERROR cinder.service     return connection._execute_clauseelement(self, multiparams, params)
& n  q6 v+ \% _( K$ l7 k+ s. s2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2021-10-20 14:41:50.737 48 ERROR cinder.service     compiled_sql, distilled_params
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1220, in _execute_context
' s( y8 `. I! o% S8 @' y) o2021-10-20 14:41:50.737 48 ERROR cinder.service     self._root._commit_impl(autocommit=True)
- R- `5 v, G' e% u, u; B/ _3 H2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in _commit_impl
) V* V- H, ]& B( a: |2021-10-20 14:41:50.737 48 ERROR cinder.service     self.connection._reset_agent is self.__transaction:
9 Z7 y: ]. L  n! k& C3 m& \+ j2 w/ U2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 354, in connection
2021-10-20 14:41:50.737 48 ERROR cinder.service     self._handle_dbapi_exception(e, None, None, None, None)
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1406, in _handle_dbapi_exception
* i. h6 t( N4 _$ V/ p2021-10-20 14:41:50.737 48 ERROR cinder.service     self._autorollback()
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 76, in __exit__
6 D0 _5 F% X4 ?2021-10-20 14:41:50.737 48 ERROR cinder.service     compat.reraise(type_, value, traceback)
6 |, q- t5 L* _: Z9 x9 _5 p2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1406, in _handle_dbapi_exception
2021-10-20 14:41:50.737 48 ERROR cinder.service     self._autorollback()
$ \9 D3 V0 m+ N$ E4 N" V( R2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 827, in _autorollback
2021-10-20 14:41:50.737 48 ERROR cinder.service     self._root._rollback_impl()
' l: k4 M- ~, F3 F2 k2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 698, in _rollback_impl
2021-10-20 14:41:50.737 48 ERROR cinder.service     self.dispatch.rollback(self)
- S7 _. D& {6 s4 `( P1 C2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
; s) x9 ~6 u+ {! V/ f7 V' L1 x2021-10-20 14:41:50.737 48 ERROR cinder.service     fn(*args, **kw)
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/exc_filters.py", line 510, in pop_exc_tx
$ V% K5 X! \( q+ @/ d) o2021-10-20 14:41:50.737 48 ERROR cinder.service     conn.info.pop(ROLLBACK_CAUSE_KEY, None)
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 461, in info
6 j6 ?' J$ g! P! v1 ^2021-10-20 14:41:50.737 48 ERROR cinder.service     return self.connection.info
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 354, in connection
0 O) f* Z$ m2 p6 P$ h4 m2021-10-20 14:41:50.737 48 ERROR cinder.service     self._handle_dbapi_exception(e, None, None, None, None)
- w+ b9 ^9 {' T7 ]) Q% U2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
2021-10-20 14:41:50.737 48 ERROR cinder.service     exc_info
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
5 c7 S9 p6 e! F5 k! G2021-10-20 14:41:50.737 48 ERROR cinder.service     reraise(type(exception), exception, tb=exc_tb, cause=cause)
; @+ m5 ?6 ^' G6 w/ ~2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
; @' s, V* F/ a' h3 e2021-10-20 14:41:50.737 48 ERROR cinder.service     return self._revalidate_connection()
2021-10-20 14:41:50.737 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
2021-10-20 14:41:50.737 48 ERROR cinder.service     raise exc.ResourceClosedError("This Connection is closed")
2021-10-20 14:41:50.737 48 ERROR cinder.service DBAPIError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
# Q# N' j1 K/ n" G+ @) P# F8 h* M2021-10-20 14:41:50.737 48 ERROR cinder.service
; q! a+ w& _8 L% y0 {* t2021-10-20 14:42:00.658 48 ERROR cinder.service [req-87ab34a8-80be-49a4-b720-442a9eef3968 - - - - -] Recovered model server connection!
2021-10-20 14:42:51.919 48 ERROR oslo_db.sqlalchemy.exc_filters [req-87ab34a8-80be-49a4-b720-442a9eef3968 - - - - -] DB exception wrapped.: ResourceClosedError: This Connection is closed
2021-10-20 14:42:51.919 48 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2021-10-20 14:42:51.919 48 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
2021-10-20 14:42:51.919 48 ERROR oslo_db.sqlalchemy.exc_filters     return self._revalidate_connection()
2021-10-20 14:42:51.919 48 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
, T9 G5 X' S& T- l2021-10-20 14:42:51.919 48 ERROR oslo_db.sqlalchemy.exc_filters     raise exc.ResourceClosedError("This Connection is closed")
2021-10-20 14:42:51.919 48 ERROR oslo_db.sqlalchemy.exc_filters ResourceClosedError: This Connection is closed
" P8 x- Y2 a$ _' ]! z4 k" ]( L2021-10-20 14:42:51.919 48 ERROR oslo_db.sqlalchemy.exc_filters
2021-10-20 14:42:51.921 48 ERROR cinder.service [req-87ab34a8-80be-49a4-b720-442a9eef3968 - - - - -] Exception encountered: : DBAPIError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
2021-10-20 14:42:51.921 48 ERROR cinder.service Traceback (most recent call last):
8 v- R2 K* k) }2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 482, in report_state
0 r% x2 l" p0 m1 C! h3 t2021-10-20 14:42:51.921 48 ERROR cinder.service     service_ref.save()
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/objects/service.py", line 181, in save
$ y& M# d+ a, K) C& k2021-10-20 14:42:51.921 48 ERROR cinder.service     db.service_update(self._context, self.id, updates)
, t( g2 {( Z# z0 R6 I9 Q2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 154, in service_update
( ]! ^# N1 Y% D( c; `) B2021-10-20 14:42:51.921 48 ERROR cinder.service     return IMPL.service_update(context, service_id, values)
7 x5 t5 A  s! K7 N2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 164, in wrapper
3 W6 g/ ~/ \, @( I# H" M2021-10-20 14:42:51.921 48 ERROR cinder.service     return f(*args, **kwargs)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2021-10-20 14:42:51.921 48 ERROR cinder.service     ectxt.value = e.inner_exc
# X* [. A  E! T) o& g3 F2 B! Z2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
, Q4 s, v8 G+ O& p8 n8 N2021-10-20 14:42:51.921 48 ERROR cinder.service     self.force_reraise()
$ ?; }: r1 {, B0 k6 V8 r2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
6 y' n' ?) M+ P( ]2021-10-20 14:42:51.921 48 ERROR cinder.service     six.reraise(self.type_, self.value, self.tb)
' m: n4 [2 A( F( [8 d1 {8 M3 t2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
) |: X: z: |- {- W- O2021-10-20 14:42:51.921 48 ERROR cinder.service     return f(*args, **kwargs)
1 B% n* V9 w, Z) B# o' L1 T' X: l2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 560, in service_update
2021-10-20 14:42:51.921 48 ERROR cinder.service     result = query.update(values)
% ~( D" n; w' a# a# t8 R( x! W/ e2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3369, in update
2021-10-20 14:42:51.921 48 ERROR cinder.service     update_op.exec_()
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1325, in exec_
( y, x, e+ x8 I! \2021-10-20 14:42:51.921 48 ERROR cinder.service     self._do_exec()
5 {2 m! h$ g  c" Y8 a2 e2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1492, in _do_exec
5 u. ?) N! T% [! r3 c2021-10-20 14:42:51.921 48 ERROR cinder.service     self._execute_stmt(update_stmt)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1332, in _execute_stmt
2021-10-20 14:42:51.921 48 ERROR cinder.service     mapper=self.mapper)
) S3 G& M  s& H  b/ ^" L2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1176, in execute
6 h/ ~/ s( E' `! r9 S# R  w: ]2021-10-20 14:42:51.921 48 ERROR cinder.service     bind, close_with_result=True).execute(clause, params or {})
/ m) D. s3 k" J3 }2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
! O/ s, q" z- u& {2021-10-20 14:42:51.921 48 ERROR cinder.service     return meth(self, multiparams, params)
1 U7 n2 r8 \; U5 w& q8 g5 ]* E* Q2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
6 O5 v* k8 b4 f2 I; s/ d6 F2021-10-20 14:42:51.921 48 ERROR cinder.service     return connection._execute_clauseelement(self, multiparams, params)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
5 x" s, }( k8 ]2021-10-20 14:42:51.921 48 ERROR cinder.service     compiled_sql, distilled_params
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1220, in _execute_context
4 F: i" ~, ~8 m5 J0 h% _, S& n2021-10-20 14:42:51.921 48 ERROR cinder.service     self._root._commit_impl(autocommit=True)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in _commit_impl
2021-10-20 14:42:51.921 48 ERROR cinder.service     self.connection._reset_agent is self.__transaction:
% P: {" K1 ^! U0 r5 [2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 354, in connection
2021-10-20 14:42:51.921 48 ERROR cinder.service     self._handle_dbapi_exception(e, None, None, None, None)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1406, in _handle_dbapi_exception
8 n6 d4 I% t9 I3 x2021-10-20 14:42:51.921 48 ERROR cinder.service     self._autorollback()
$ B: y  i3 L6 ^- b8 ~2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 76, in __exit__
2021-10-20 14:42:51.921 48 ERROR cinder.service     compat.reraise(type_, value, traceback)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1406, in _handle_dbapi_exception
( s$ X; P( T5 q3 h2021-10-20 14:42:51.921 48 ERROR cinder.service     self._autorollback()
5 I4 i, a' X+ x- [) F9 z2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 827, in _autorollback
2021-10-20 14:42:51.921 48 ERROR cinder.service     self._root._rollback_impl()
& n% G. x% m0 [, y2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 698, in _rollback_impl
! h8 q9 U* W0 v+ k+ b7 w2021-10-20 14:42:51.921 48 ERROR cinder.service     self.dispatch.rollback(self)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
2021-10-20 14:42:51.921 48 ERROR cinder.service     fn(*args, **kw)
# U- e: g0 A+ y3 f4 p6 z- _2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/exc_filters.py", line 510, in pop_exc_tx
2021-10-20 14:42:51.921 48 ERROR cinder.service     conn.info.pop(ROLLBACK_CAUSE_KEY, None)
4 Z& K) Q" ]. f0 x0 B; |2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 461, in info
2021-10-20 14:42:51.921 48 ERROR cinder.service     return self.connection.info
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 354, in connection
2 v; k3 v' a/ t, P2021-10-20 14:42:51.921 48 ERROR cinder.service     self._handle_dbapi_exception(e, None, None, None, None)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
2021-10-20 14:42:51.921 48 ERROR cinder.service     exc_info
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2021-10-20 14:42:51.921 48 ERROR cinder.service     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
- ^: N4 U9 X& Q$ x2021-10-20 14:42:51.921 48 ERROR cinder.service     return self._revalidate_connection()
- u9 Z" i  T. b2021-10-20 14:42:51.921 48 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
2021-10-20 14:42:51.921 48 ERROR cinder.service     raise exc.ResourceClosedError("This Connection is closed")
  C. }4 i& i! a# S- D, D2021-10-20 14:42:51.921 48 ERROR cinder.service DBAPIError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
% U$ h/ g4 H' \4 w2021-10-20 14:42:51.921 48 ERROR cinder.service
2021-10-20 14:42:52.254 34 ERROR oslo_db.sqlalchemy.exc_filters [req-1ce13e05-9f9d-409c-9223-516b7ae2b7c8 - - - - -] DB exception wrapped.: ResourceClosedError: This Connection is closed
2021-10-20 14:42:52.254 34 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2021-10-20 14:42:52.254 34 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
2021-10-20 14:42:52.254 34 ERROR oslo_db.sqlalchemy.exc_filters     return self._revalidate_connection()
. n' i5 K# L: Z" h+ ~2021-10-20 14:42:52.254 34 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
+ U- \, A: \; C5 @2021-10-20 14:42:52.254 34 ERROR oslo_db.sqlalchemy.exc_filters     raise exc.ResourceClosedError("This Connection is closed")
& j* ?7 J# M6 c7 @0 m8 V2021-10-20 14:42:52.254 34 ERROR oslo_db.sqlalchemy.exc_filters ResourceClosedError: This Connection is closed
+ \% ]% _" |- T) B" _1 ]. x2021-10-20 14:42:52.254 34 ERROR oslo_db.sqlalchemy.exc_filters
$ t, H# c9 }3 ~9 y2021-10-20 14:42:52.259 34 WARNING py.warnings [req-1ce13e05-9f9d-409c-9223-516b7ae2b7c8 - - - - -] /usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py:74: SAWarning: An exception has occurred during handling of a previous exception.  The previous exception is:
& `  m( v' U0 }- | <class 'sqlalchemy.exc.ResourceClosedError'> This Connection is closed
( q% _' P' y7 p8 y+ b' P( N; [
  "is:\n %s %s\n" % (self._exc_info[0], self._exc_info[1]))
: DBAPIError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
$ J! C3 ?( L- Z9 n. ~. i! M. F2021-10-20 14:42:52.259 34 ERROR cinder.service [req-1ce13e05-9f9d-409c-9223-516b7ae2b7c8 - - - - -] Exception encountered: : DBAPIError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
2021-10-20 14:42:52.259 34 ERROR cinder.service Traceback (most recent call last):
6 \& P; L) W* d: p% ], d8 ^* W2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 482, in report_state
  X( \7 v1 d! U0 G2021-10-20 14:42:52.259 34 ERROR cinder.service     service_ref.save()
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/objects/service.py", line 181, in save
2021-10-20 14:42:52.259 34 ERROR cinder.service     db.service_update(self._context, self.id, updates)
# l8 L* G2 i$ e0 F2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 154, in service_update
2021-10-20 14:42:52.259 34 ERROR cinder.service     return IMPL.service_update(context, service_id, values)
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 164, in wrapper
; v. M' O+ L/ ]/ E- K2021-10-20 14:42:52.259 34 ERROR cinder.service     return f(*args, **kwargs)
! A" m' s" [- x9 d2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2021-10-20 14:42:52.259 34 ERROR cinder.service     ectxt.value = e.inner_exc
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
. U7 J: W2 [. u8 o5 F! E2021-10-20 14:42:52.259 34 ERROR cinder.service     self.force_reraise()
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
, \& l; `: r8 F! L; J1 U6 e2021-10-20 14:42:52.259 34 ERROR cinder.service     six.reraise(self.type_, self.value, self.tb)
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
, O" l' h/ S" P* ]# N* ^, X, b2021-10-20 14:42:52.259 34 ERROR cinder.service     return f(*args, **kwargs)
3 I3 T: }6 T9 T5 T8 j/ D5 m) I2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 560, in service_update
) x; Z2 d; J! U# R3 r1 h2021-10-20 14:42:52.259 34 ERROR cinder.service     result = query.update(values)
5 R- O6 K" M; X$ ]" k- Y  J) y2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3369, in update
2021-10-20 14:42:52.259 34 ERROR cinder.service     update_op.exec_()
2 I* U/ `1 t* d5 E2 u  o; j$ `2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1325, in exec_
7 G1 G  {) T( I8 O: n6 D& R) K2021-10-20 14:42:52.259 34 ERROR cinder.service     self._do_exec()
3 i0 U$ o- v+ Q- k+ d2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1492, in _do_exec
) w$ v! `, [5 R7 |+ M  L) }2021-10-20 14:42:52.259 34 ERROR cinder.service     self._execute_stmt(update_stmt)
7 z7 ~9 y  N: t" R4 E2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1332, in _execute_stmt
2021-10-20 14:42:52.259 34 ERROR cinder.service     mapper=self.mapper)
( \4 `2 C4 q# E' [# O  ]2 f2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1176, in execute
+ Z# H" [( Y' Z% ?3 n2021-10-20 14:42:52.259 34 ERROR cinder.service     bind, close_with_result=True).execute(clause, params or {})
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
$ ~: m( }, t  R' u1 i9 M0 d2021-10-20 14:42:52.259 34 ERROR cinder.service     return meth(self, multiparams, params)
7 Z% K; l, X. W/ X, z2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2021-10-20 14:42:52.259 34 ERROR cinder.service     return connection._execute_clauseelement(self, multiparams, params)
2 s% P' n  j% b5 O$ v9 ~2 Q2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
3 M9 \8 ?* G0 a# c/ k( q2021-10-20 14:42:52.259 34 ERROR cinder.service     compiled_sql, distilled_params
. F: Q  f0 j4 H2 n) F; t2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1220, in _execute_context
; M+ g5 K. {( V1 A4 D( g' J2021-10-20 14:42:52.259 34 ERROR cinder.service     self._root._commit_impl(autocommit=True)
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in _commit_impl
) ?% L& u4 e6 r2021-10-20 14:42:52.259 34 ERROR cinder.service     self.connection._reset_agent is self.__transaction:
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 354, in connection
2021-10-20 14:42:52.259 34 ERROR cinder.service     self._handle_dbapi_exception(e, None, None, None, None)
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1406, in _handle_dbapi_exception
- d; }0 V8 v& e; h$ I8 F$ n2021-10-20 14:42:52.259 34 ERROR cinder.service     self._autorollback()
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 76, in __exit__
4 d. I) L7 f! P9 n' Z5 m, z# _2021-10-20 14:42:52.259 34 ERROR cinder.service     compat.reraise(type_, value, traceback)
$ H4 L: C3 q% Q; c0 i6 a! P; b2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1406, in _handle_dbapi_exception
7 R  v+ ?0 Z1 n* i' q' G! [2021-10-20 14:42:52.259 34 ERROR cinder.service     self._autorollback()
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 827, in _autorollback
) p/ d( r. G1 W0 ]) e8 `7 u2021-10-20 14:42:52.259 34 ERROR cinder.service     self._root._rollback_impl()
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 698, in _rollback_impl
; K$ \" V9 ~/ k3 b0 ^+ T6 p, ]2021-10-20 14:42:52.259 34 ERROR cinder.service     self.dispatch.rollback(self)
. {& M- t6 r+ E7 q3 S0 i! C2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
: J$ Y0 I. ]6 k+ I6 G2021-10-20 14:42:52.259 34 ERROR cinder.service     fn(*args, **kw)
  S! C) ~$ ^2 X$ x; W0 o( Q+ a' _2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/exc_filters.py", line 510, in pop_exc_tx
  o" L! M7 |/ ^- `+ q$ c0 {2021-10-20 14:42:52.259 34 ERROR cinder.service     conn.info.pop(ROLLBACK_CAUSE_KEY, None)
) d# w& C7 F: C+ S- T- r8 @2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 461, in info
- i6 i* W8 \* N3 S& L" L. j! L1 ~2021-10-20 14:42:52.259 34 ERROR cinder.service     return self.connection.info
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 354, in connection
2021-10-20 14:42:52.259 34 ERROR cinder.service     self._handle_dbapi_exception(e, None, None, None, None)
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
2021-10-20 14:42:52.259 34 ERROR cinder.service     exc_info
# l# ~, k, l* c: z# I& p2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2021-10-20 14:42:52.259 34 ERROR cinder.service     reraise(type(exception), exception, tb=exc_tb, cause=cause)
7 C  E0 h8 N4 b8 n1 B2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
9 b* f/ @8 W8 W# q: f6 k2021-10-20 14:42:52.259 34 ERROR cinder.service     return self._revalidate_connection()
2021-10-20 14:42:52.259 34 ERROR cinder.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
2021-10-20 14:42:52.259 34 ERROR cinder.service     raise exc.ResourceClosedError("This Connection is closed")
- ?4 |6 w9 h* U! h2021-10-20 14:42:52.259 34 ERROR cinder.service DBAPIError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed (Background on this error at: http://sqlalche.me/e/dbapi)
2021-10-20 14:42:52.259 34 ERROR cinder.service
  D% p" \) A* v/ r! D+ L2021-10-20 14:43:00.607 34 ERROR cinder.service [req-1ce13e05-9f9d-409c-9223-516b7ae2b7c8 - - - - -] Recovered model server connection!
# U+ N; W) i& J) D1 o; `2021-10-20 14:43:00.674 48 ERROR cinder.service [req-87ab34a8-80be-49a4-b720-442a9eef3968 - - - - -] Recovered model server connection!
$ `* j/ z9 K7 Y& ^2021-10-20 14:44:54.078 48 WARNING py.warnings [req-c0b29795-7449-4c13-b995-314de3b95543 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] /usr/lib64/python2.7/site-packages/sqlalchemy/orm/evaluator.py:71: SAWarning: Evaluating non-mapped column expression 'updated_at' onto ORM instances; this is a deprecated use case.  Please make use of the actual mapped columns in ORM-evaluated UPDATE / DELETE expressions.
  "UPDATE / DELETE expressions." % clause)

2021-10-20 14:44:54.259 48 INFO cinder.volume.manager [req-c0b29795-7449-4c13-b995-314de3b95543 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Deleted volume successfully.
- y6 o/ n0 }4 z0 f( y- |; A2021-10-20 14:48:52.975 7 INFO oslo_service.service [req-6284bff7-c9e4-4c7c-b944-6f3b1580e6de - - - - -] Caught SIGTERM, stopping children
3 D9 y" }  v- I. g3 Z2 V2021-10-20 14:48:52.979 7 INFO oslo_service.service [req-6284bff7-c9e4-4c7c-b944-6f3b1580e6de - - - - -] Waiting on 2 children to exit
. i: N* ?) B7 @# c9 p2021-10-20 14:49:06.433 7 INFO cinder.rpc [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Automatically selected cinder-scheduler objects version 1.35 as minimum service version.
2021-10-20 14:49:06.440 7 INFO cinder.rpc [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Automatically selected cinder-scheduler RPC version 3.10 as minimum service version.
& c+ ?6 ^! `+ F2 l" G0 M2021-10-20 14:49:06.500 7 INFO cinder.volume.manager [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Determined volume DB was not empty at startup.
2021-10-20 14:49:06.515 7 INFO cinder.volume.manager [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Image-volume cache disabled for host rbd:volumes@volumes.
1 ]5 d/ J& L0 Z2021-10-20 14:49:06.533 7 INFO oslo_service.service [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Starting 1 workers
( `; h" M: E* ?  ]7 \2021-10-20 14:49:06.545 34 INFO cinder.service [-] Starting cinder-volume node (version 12.0.9)
2021-10-20 14:49:06.577 34 INFO cinder.volume.manager [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Starting volume driver RBDDriver (1.2.0)
2021-10-20 14:49:06.591 7 INFO cinder.volume.manager [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Determined volume DB was not empty at startup.
2021-10-20 14:49:06.601 7 INFO cinder.volume.manager [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Image-volume cache disabled for host rbd:vms@vms.
2 P( R* G6 Y7 t, a0 `% z; c8 N. ^2021-10-20 14:49:06.621 7 INFO oslo_service.service [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Starting 1 workers
) Y/ h. p( A& W2021-10-20 14:49:06.629 37 INFO cinder.service [-] Starting cinder-volume node (version 12.0.9)
! @8 |- O. P9 @5 k; u2021-10-20 14:49:06.635 7 WARNING oslo_config.cfg [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Option "rabbit_max_retries" from group "DEFAULT" is deprecated. Use option "rabbit_max_retries" from group "oslo_messaging_rabbit".
2021-10-20 14:49:06.636 7 WARNING oslo_config.cfg [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Option "rabbit_max_retries" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2021-10-20 14:49:06.646 37 INFO cinder.volume.manager [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Starting volume driver RBDDriver (1.2.0)
  N; C4 [% s% V5 |2021-10-20 14:49:06.649 7 WARNING oslo_config.cfg [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Option "auth_uri" from group "keystone_authtoken" is deprecated for removal (The auth_uri option is deprecated in favor of www_authenticate_uri and will be removed in the S  release.).  Its value may be silently ignored in the future.
2021-10-20 14:49:06.651 7 WARNING oslo_config.cfg [req-7711c8a0-ae4b-4d6c-bebe-bb1470e8e4fa - - - - -] Option "auth_uri" from group "keystone_authtoken" is deprecated. Use option "www_authenticate_uri" from group "keystone_authtoken".
2021-10-20 14:49:06.970 37 INFO cinder.keymgr.migration [req-3c885312-b5bf-4b0d-84e6-6a3efa9c4e61 - - - - -] Not migrating encryption keys because the ConfKeyManager's fixed_key is not in use.
: L% z: K+ j5 |% h  ?$ F2021-10-20 14:49:06.975 37 WARNING oslo_config.cfg [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Option "rabbit_max_retries" from group "DEFAULT" is deprecated. Use option "rabbit_max_retries" from group "oslo_messaging_rabbit".
! ^* B- Q  I9 s' G6 G. @; G& [4 U( j2021-10-20 14:49:06.976 37 WARNING oslo_config.cfg [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Option "rabbit_max_retries" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2021-10-20 14:49:07.009 37 INFO cinder.volume.manager [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Driver initialization completed successfully.
2021-10-20 14:49:07.017 37 INFO cinder.manager [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Initiating service 11 cleanup
- U, B) i# d1 Z2021-10-20 14:49:07.022 34 INFO cinder.keymgr.migration [req-77746f08-b601-4884-ab57-b0cbfe306998 - - - - -] Not migrating encryption keys because the ConfKeyManager's fixed_key is not in use.
2021-10-20 14:49:07.022 37 INFO cinder.manager [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Service 11 cleanup completed.
2021-10-20 14:49:07.060 34 WARNING oslo_config.cfg [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Option "rabbit_max_retries" from group "DEFAULT" is deprecated. Use option "rabbit_max_retries" from group "oslo_messaging_rabbit".
2021-10-20 14:49:07.061 34 WARNING oslo_config.cfg [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Option "rabbit_max_retries" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2021-10-20 14:49:07.100 34 INFO cinder.volume.manager [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Driver initialization completed successfully.
, D" B; A5 `& J( Z2021-10-20 14:49:07.110 34 INFO cinder.manager [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Initiating service 8 cleanup
, z% Q  }/ h- _) u; `2021-10-20 14:49:07.118 34 INFO cinder.manager [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Service 8 cleanup completed.
2021-10-20 14:49:07.120 37 INFO cinder.volume.manager [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Initializing RPC dependent components of volume driver RBDDriver (1.2.0)
2021-10-20 14:49:07.209 34 INFO cinder.volume.manager [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Initializing RPC dependent components of volume driver RBDDriver (1.2.0)
2021-10-20 14:49:07.248 37 INFO cinder.volume.manager [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Driver post RPC initialization completed successfully.
5 b* A) P' \1 E/ w- j2021-10-20 14:49:07.367 34 INFO cinder.volume.manager [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Driver post RPC initialization completed successfully.
^C
[root@compute01 ~]# tail -f /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-volume.log
2021-10-20 14:49:07.248 37 INFO cinder.volume.manager [req-0187f6fb-4f52-4820-8f0d-5aaa2a6fb49c - - - - -] Driver post RPC initialization completed successfully.
2021-10-20 14:49:07.367 34 INFO cinder.volume.manager [req-8ad266dc-5d0d-4971-a6fc-a8823c69dacb - - - - -] Driver post RPC initialization completed successfully.
0 f# e' u, M9 y  m! V8 K2021-10-20 14:50:12.395 34 INFO cinder.rpc [req-7b09fbe6-6e55-4d57-ad2b-66a8e28d38cc 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Automatically selected cinder-backup objects version 1.35 as minimum service version.
2021-10-20 14:50:12.410 34 INFO cinder.rpc [req-7b09fbe6-6e55-4d57-ad2b-66a8e28d38cc 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Automatically selected cinder-backup RPC version 2.1 as minimum service version.
2021-10-20 14:50:12.428 34 INFO cinder.rpc [req-7b09fbe6-6e55-4d57-ad2b-66a8e28d38cc 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Automatically selected cinder-volume objects version 1.35 as minimum service version.
7 i7 n2 B9 C( g8 ?$ n1 L! o. Y9 P2021-10-20 14:50:12.439 34 INFO cinder.rpc [req-7b09fbe6-6e55-4d57-ad2b-66a8e28d38cc 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Automatically selected cinder-volume RPC version 3.15 as minimum service version.
$ ^. c$ v+ Z! j$ N/ S2021-10-20 14:50:12.601 34 INFO cinder.volume.flows.manager.create_volume [req-7b09fbe6-6e55-4d57-ad2b-66a8e28d38cc 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Volume d760f9e9-77f8-466a-b53e-2e9de6e093ac: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-d760f9e9-77f8-466a-b53e-2e9de6e093ac'}
2021-10-20 14:50:12.747 34 INFO cinder.volume.flows.manager.create_volume [req-7b09fbe6-6e55-4d57-ad2b-66a8e28d38cc 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Volume volume-d760f9e9-77f8-466a-b53e-2e9de6e093ac (d760f9e9-77f8-466a-b53e-2e9de6e093ac): created successfully
' {6 S0 s' J: z& e; m/ O; _2021-10-20 14:50:12.793 34 INFO cinder.volume.manager [req-7b09fbe6-6e55-4d57-ad2b-66a8e28d38cc 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Created volume successfully.
2021-10-20 14:55:12.317 34 INFO cinder.volume.manager [req-915cef10-7f50-40f8-aea2-69df1ce6639a 14687393c47346beb4a6b95232e5d434 fce718dfba31472a940816a41b8aed2a - default default] Initialize volume connection completed successfully.

( _  E* [8 ~- N0 N# I