|
|
Kolla-Ansible ease of configuration and deployment highly depends on the uniformity of the network interface names which will be used by OpenStack on the controller/network/hypervisors/storage/etc. nodes, so it’s highly recommended to unify the network interface names across all the nodes, including the head node if it will be used as the deployment node. I- x( h1 y; }
Unifying the network interface names on CentOS 8 can be done in several ways:" O7 I5 u6 G& z0 y# p" T
Switch to traditional naming conventions for network interfaces:
9 ~6 q, b4 J* b/ ^, ?For the compute nodes, this can be done by appending net.ifname=0 to the kernel parameters in the software image:' l. T5 h) E, E7 J( k# g
# cmsh
* h( |: G, J. w, Q% softwareimage use <image-name>! X! A( \% ^8 ^1 F% L/ Z
% append kernelparameters ” net.ifnames=0″
8 n% ^2 v7 j0 k b% commit) |3 g4 }, |4 z
For the head node(s), this can be done by updating the GRUB_CMDLINE_LINUX parameters in /etc/default/grub.conf and re-writing the /boot/grub2/grub.conf
3 W! f% K! n: t* y8 D# grep net /etc/default/grub
: H! r2 _1 f( c/ u o/ [GRUB_CMDLINE_LINUX=”vconsole.keymap=us crashkernel=auto vconsole.font=latarcyrheb-sun16 rd.driver.blacklist=nouveau net.ifnames=0″1 [' y p- K4 Q0 b" P$ O
# grub2-mkconfig -o /boot/grub2/grub.cfg
& R+ V! Z* S5 w9 ^; L& o# reboot6 t5 v' |: S, s, M+ W. D+ x3 ]
Specify the MAC address and device name in the corresponding ifcfg-* files.
2 X( Y4 L# J4 JKolla-ansible is installed via “python-pip” so it’s mandatory to update “pip” to the latest:" k( p. @3 B0 @7 J6 c) j
# pip3 install -U pip/ p" H( Q9 n+ G
By default, Bright assumes that there are no important data stored on the compute nodes so each time a compute node is rebooted, the software image is re-synced to the local drive of the compute node resulting in adding any missing data or wiping out any data which don’t exist in the software image.
( ~* b( k( O: |; m! tTo avoid wiping out the data that will be introduced by the Kolla-ansible deployment, we will need to add the following paths to the sync/update exclude lists:
& ? @2 h# W4 P7 N– /var/lib/docker
' @- p7 a1 G# x% \( q+ X$ g– /var/lib/docker/*6 P3 m$ o1 j4 s4 I- G+ G. `
– /etc/kolla6 p1 o7 D( F O0 G
– /etc/kolla/*. F0 V( [! ~8 F4 G, C8 ~( ^# k) O
To avoid pulling unnecessary data from the compute nodes that will be deployed by the Kolla-ansible deployment, we will need to add the following paths to the grab exclude lists:
* C1 M; y* D1 e! A9 n/ M– /var/lib/docker
6 F) ]3 J9 {" g8 q– /var/lib/docker/*' c6 a. B+ A! D& @% X% _' L
– /etc/kolla$ v8 p% `, P! o2 W" I# F$ S
– /etc/kolla/*
) @* r+ a- o6 e$ D: }** Note: The exclude lists are better updated on the category level. If the setup will only include one compute node, then it may be better to update the exclude lists directly on the node.
- |2 J6 C9 z" v$ N/ KInstalling Ansible
$ \' g! _$ k, S6 `" tKolla-ansible requires ansible version 2.8 or higher and up to version 2.9 as of writing this document (June 2020)9 U" C. }5 f1 u! e/ [, C3 h5 D% V
yum install epel-release -y8 n) t8 c* y+ u+ j2 W: W$ X* I4 B
yum install ansible -y/ M4 x4 M7 k9 d# ^
Configuring Ansible (optional)
1 \. |+ o0 Q* bTuning the configurations of Ansible may improve the deployment time of Kolla-ansible so it’s suggested to do tune the ansible configurations for better results:
4 B+ F3 i0 p; s2 L. [ u% M8 X9 DIncrease the value of the forks as you see necessary. The forks parameter controls how many hosts can be configured in parallel, so if you have the processing power on the head node to run several forks, then it’s a good practice to increase the number of forks., I- ?- e' @0 n0 G4 h
Disable host_key_checking. Ansible has host key checking enabled by default. If a host is reinstalled and has a different key in ‘known_hosts’, then this will result in an error message until corrected. If a host is not initially in ‘known_hosts’ this will result in prompting for confirmation of the key, which will require an interactive intervention.
; |/ ^: K& X0 T& q1 E; PEnable pipelining. Enabling pipelining reduces the number of SSH operations required to execute a module on the remote server. This can result in a significant performance improvement when enabled. By default, this option is disabled. D" e6 a9 L* e2 k: f- I7 o% ^
# grep -E “forks|host_key_checking|pipelining” /etc/ansible/ansible.cfg | grep -v “^#”
/ a& ]8 k: w3 \+ ~: Kforks = 100" |+ G, X& P6 U. ~
host_key_checking = False0 \2 B0 J2 B" _7 p% q& n: {( G9 j8 I
pipelining = True
. G6 W2 i5 a3 {% gInstalling Kolla-ansible
% b$ n! `1 P9 l3 YKolla-ansible and its dependencies can be installed using python pip:
+ E4 h o9 R4 B! R* O# pip install kolla-ansible
% S9 b: B; z0 l. J( w! i& [Create default configuration directory. By default, Kolla-ansible expects the global configurations to be stored under /etc/kolla on the deployment node.
. E' |6 t( e/ _; b, @; d' Q# mkdir /etc/kolla' i v; r! n( T' e$ }) b6 M1 v5 t
Copy default globals.yml and password.yml templates from Kolla-ansible installation directory:4 G" F, k$ z9 v @
# cp -r /usr/local/share/kolla-ansible/etc_examples/kolla/* /etc/kolla
1 J$ i) r+ ]' ^9 X8 ?Copy all-in-one and multinode inventory files to the current directory.& U# w& K6 V M0 q H: z
# cp /usr/local/share/kolla-ansible/ansible/inventory/* .
! I" F; O" c: ~Preparing initial Kolla-ansible configurations
, P7 b2 @( V! X O* ZThe globals.yml is the main configuration file for Kolla-Ansible. There are a few parameters that are required to deploy Kolla-ansible:, d4 O2 F7 n$ [* p- F
Kolla-ansible provides container images for deploying OpenStack on several Linux distributions as host system: CentOS 7/8, Ubuntu 18.04, Debian and RHEL 7/8. For the purpose of this document, we will use CentOS 8 as the host system. Using kolla_base_distro parameter, “centos” can be specified to refer to CentOS as base distribution.
0 j H8 R* d) A2 S" }Type of installation can be either binary: using repositories like apt or yum or source: using raw source archives, git repositories, or local source directory. According to the official Kolla-ansible documentation, type source is proven to be slightly more reliable. The install type can be specified using the kolla_install_type parameter.
. w/ f/ C. u8 M- N1 nThe release of OpenStack to be deployed has to be specified using the openstack_release parameter. For the purpose of this document, we will use the OpenStack train.* K6 r3 b; h) \4 h7 [
Kolla-ansible requires two main networking options, network_interface: the default interface for multiple management-type networks (tenants networks) and neutron_external_interface: a dedicated network interface for Neutron external/public networks (provider networks) which can be vlan or flat, depends on how the networks are created. This interface should be active without an IP address on controller/network nodes. Otherwise, OpenStack instances won’t be accessible from the external networks. For the purpose of this document, we will use flat networks.4 b, k( \9 ?6 B/ g4 u _
A floating IP for management traffic should be specified using kolla_internal_vip_address. This IP will be managed by keepalived to provide high availability and should be a free IP address in the internal management network that is connected to the same network_interface.
. } k2 t3 M6 l% d. u1 lAdditional services can be enabled. For the purpose of this document, we will enable the cinder service using enable_cinder and enable_cinder_backend_nfs parameters. The cinder service with NFS backend requires an additional configuration file that has to be placed under /etc/kolla/config/ with the name of nfs_shares. This file includes the name/IP of the NFS servers and the path on the server. The file can include several NFS servers and different paths:
2 Y) }* p. k4 t# cat /etc/kolla/config/nfs_shares! I& J: h+ T$ |3 o4 A
10.141.0.1:/cinder6 T6 e. C. q/ I0 Y
# grep -vE “^#|$^” /etc/kolla/globals.yml
$ s% O. D8 @+ w: m- f0 m: k" L$ _—) X0 A5 O3 G8 S- b& V0 D/ ~
kolla_base_distro: “centos”
, s9 j, V/ r [5 d" X- }6 Y3 jkolla_install_type: “source”
9 A- Y& f% ~% L9 [+ R' M2 Mopenstack_release: “ussuri”1 v7 x; h2 ?5 g0 ~) B
kolla_internal_vip_address: “10.141.255.245”
6 Y ?1 V3 L: s* F+ H2 Z5 c1 ]network_interface: “eth0”9 k7 d8 V# i. ^2 X2 e' Y
neutron_external_interface: “eth1”
2 n* @, U. J# W2 T& y+ p9 Yenable_cinder: “yes”
( x8 B0 l3 N6 O6 c: uenable_cinder_backend_nfs: “yes”8 q" a/ y* O$ [* Y& W' z) ]5 E0 x
** Please make sure that the NFS server can accept mount requests from the OpenStack nodes. Otherwise, OpenStack will fail to launch instances because it will fail to allocate storage for them.2 I, f! a6 `! Y2 v; D. [# H
** OpenStack release “train” is used with kolla-ansible 9.1.0. If a higher version of kolla-ansible is to be used, then another release for OpenStack should be used. For example, kolla-ansible version 10.0.0 can be used with OpenStack release “ussuri”.( I0 A& W( y: U; Q( t9 N
The multinode template file is an ansible inventory file that has the main server groups for different OpenStack components that will be installed.
* K* g7 M3 v( [2 W5 |** DO NOT REMOVE THE CONTENT OF THE “multinode” FILE. WHAT IS SHOWN IN THE FOLLOWING SNIPPET IS JUST THE FIRST 20 LINES WHICH YOU NEED TO EDIT. THE REST WILL REMAIN THE SAME AND SHOULD NOT BE REMOVED.
8 I4 t. F# H; e- G, u# grep -vE “^#|$^” multinode | head -n 201 l! R/ u1 B& E7 a; f6 z
[control]
4 B' O. K( S; o5 S/ Q0 v. znode001node002node003! _& s6 ?; D6 o; f' k6 n* m* |6 e- e$ T& `
[network]% T& a( c1 Q6 s& G
node001node002node003
! J' w1 U- |5 ][compute]3 K# p! M+ A8 o- V) x3 w
node001node002node003' w0 [$ b: d! ~& Y1 i
[monitoring]8 S, J) K; }1 V: M& I& k: q
node001node002node003
6 |/ k* y5 S+ f& a2 _; F[storage]% K. }% T. g6 V- ^( Q; u. x2 z7 g
localhost ansible_connection=local
4 G) r/ {& z# S3 g[deployment]
5 b' C( R( M" i3 l( {; ^# Glocalhost ansible_connection=local
) l2 d' K$ n# q/ }! \; V[baremetal:children]! D% t! J. l. Z0 @8 h- ^2 j) K6 B( C
control
' b- b& f8 r! g# w! Snetwork) W2 G6 e3 {5 U4 U% P+ q
compute
7 a7 m/ a5 @2 C1 `4 e& [7 i7 Ustorage
! Q/ t6 p% Z8 w5 Z6 ]; t. l* {; [; g8 Hmonitoring8 e3 w9 f! M! g3 B8 |2 u" K) P
Deploying Kolla-Ansible
( P, T$ Z" L8 a% Q; mMake sure that the nodes specified in the inventory are all reachable:
. H( m; Z. [1 u( J# ansible -i multinode all -m ping6 l( e( |$ ^/ F; X
The template file /etc/kolla/passwords.yml is a placeholder for all the passwords that will be used during the kolla-ansible deployment:% F9 f+ s6 o! Z# \4 u" L
# grep -vE “^#|$^” /etc/kolla/passwords.yml | head
% Y1 x0 D# t3 {, o+ Q—
" u! V! L* b; ^2 v/ @ceph_cluster_fsid:
( T3 \: U% E0 \6 Oceph_rgw_keystone_password:' y- F2 }/ k) T: \* b9 Y% H: H
rbd_secret_uuid:
: D4 h) I0 r/ s. I& ~; O0 ]- d, z3 Tcinder_rbd_secret_uuid:
) n6 o# `9 g& ^* c% }. |3 ddatabase_password:
) b; m3 J9 a+ g& W1 Smariadb_backup_database_password:
/ Q( e5 |/ J% t) `docker_registry_password:3 ?5 F7 M( M+ i# ]: ?7 ^
opendaylight_password:
9 v9 J" [& Z" P" Q; D3 yvmware_dvs_host_password:) b" U/ F* h+ H4 y
All passwords in passwords.yml are blank and have to be filled either manually or by running a random password generator:) f8 H9 X# M8 }3 w% L
# kolla-genpwd5 y, E, i; U4 A/ k' f' [
Bootstrap all nodes with Kolla deploy dependencies. This will include installing packages and writing out configuration files:% t% l5 R. w7 G/ H, v6 u0 |$ m& _% f
# kolla-ansible -i ./multinode bootstrap-servers
Z# U1 e1 u, WRun the prechecks for deployment for hosts. This will make sure that the packages have been installed successfully and the configuration files do exist in the expected location:1 O3 b# b L( s0 i7 {
# kolla-ansible -i ./multinode prechecks! s. X* S D5 x
If the previous two steps did run without failures, then you can proceed to deploy OpenStack:
4 [5 `7 m$ E+ R# kolla-ansible -i ./multinode deploy
3 @1 W+ X2 z: Q3 C, g9 lAfter deployment is successful, an openrc file must be generated wherein the credentials for admin users are listed. By running the following command, an admin-openrc.sh file will be generated under /etc/kolla directory on the deployment (head) node: c Q. l& ~# k, C, C' G; r/ s
# kolla-ansible post-deploy3 ]! S4 r% y8 Q# h: _/ g
Now the changes which has been applied to the compute nodes have to be grabbed back to the software image of the compute nodes to avoid losing them:
7 e& S& W" [; |* q, w: V# cmsh9 _7 N; E M) i
% device use node001
" ^. Z6 F6 x1 t( A, r) {7 w/ t+ b% grabimage -w8 t% Y+ d( R0 u! w, L$ k& `
Make sure that SELinux is still disabled and not set to permissive:9 |4 i% q5 B' J1 W- P
# grep -w SELINUX /cm/images/default-image/etc/selinux/config | grep -v “^#”
: C& H) [- V0 L/ U pSELINUX=disabled/ z$ s1 t1 ?/ v4 d2 H7 h) d
Installing OpenStack CLI client; b w0 Z6 T5 I+ m5 ~/ H: I% ~ J; `
To start using OpenStack which has just been deployed, OpenStack CLI needs to be installed first and the openrc file needs to be sourced:
: u2 E( y" L/ |, r) e# pip install python-openstackclient
- L( J: T% v1 E9 I2 E# . /etc/kolla/admin-openrc.sh
) O r3 `9 v1 N: X# v" _Initializing the OpenStack environment for the first use
2 f- m8 \0 h0 H5 WTo start using OpenStack to create instances and access them from the external network, the “/usr/local/share/kolla-ansible/init-runonce” has to be run. This script uses “10.0.2.0/24” as the external network. In most cases, the init-runonce has to be adjusted to match the external network:4 E7 p2 {& {, t* P" t( Z4 \7 c
# This EXT_NET_CIDR is your public network,that you want to connect to the internet via.- d. O0 N6 Q+ F+ m
ENABLE_EXT_NET=${ENABLE_EXT_NET:-1}
6 O3 b, W: ~% c U5 nEXT_NET_CIDR=${EXT_NET_CIDR:-‘10.0.2.0/24’}9 L9 b/ W+ H6 f$ ?, M+ v2 [
EXT_NET_RANGE=${EXT_NET_RANGE:-‘start=10.0.2.150,end=10.0.2.199’}9 x+ R: N1 U* ^% G( e1 y
EXT_NET_GATEWAY=${EXT_NET_GATEWAY:-‘10.0.2.1’}
" h6 l* F; Z6 n- w[…]
1 Q3 Q9 j! ^$ gif [[ $ENABLE_EXT_NET -eq 1 ]]; then
! N3 E8 C+ Z$ m7 _8 O openstack network create –external –provider-physical-network physnet1 \4 D5 B8 T# g7 w) p/ P0 x
–provider-network-type flat public1
' d7 [, e2 L0 {- k+ B- j openstack subnet create –no-dhcp \% g5 e8 C2 l9 \
–allocation-pool ${EXT_NET_RANGE} –network public1 \
0 X' m8 l/ W+ Q" b V+ L. V –subnet-range ${EXT_NET_CIDR} –gateway ${EXT_NET_GATEWAY} public1-subnet- X+ r; U) `' k, T$ q& J- y5 I
fi5 R) p; h, X" o/ \, C& c
openstack network create –provider-network-type vxlan demo-net" x# H0 @' Q! M; t* g, i
openstack subnet create –subnet-range 10.0.0.0/24 –network demo-net \
/ `+ O; w* T5 u –gateway 10.0.0.1 –dns-nameserver 8.8.8.8 demo-subnet
0 I3 }. w6 Q t; ?0 L/ T: ATypically, provider networks are directly associated with a physical network but that is not a requirement. Users create tenant networks for connectivity within projects. By default, these tenant networks are fully isolated and are not shared with other projects. OpenStack Networking supports the following types of network isolation and overlay technologies:0 F W2 @8 z4 a2 `+ V! D
Flat
; N, X5 s; [+ A: H: d uAll instances reside on the same network, which can also be shared with the hosts. No VLAN tagging or other network segregation takes place.
, B* [2 P, v& EVLAN
( i% U0 J2 n+ uNetworking allows users to create multiple provider or tenant networks using VLAN IDs (802.1Q tagged) that correspond to VLANs present in the physical network. This allows instances to communicate with each other across the environment. They can also communicate with dedicated servers, firewalls, load balancers, and other networking infrastructure on the same layer 2 VLAN.
+ }- F' y8 H# \! D6 }4 s9 v U8 r+ [( d/ cGRE and VXLAN1 V/ _; I, m% q& X% P( R
VXLAN and GRE are encapsulation protocols that create overlay networks to activate and control communication between compute instances. A Networking router is required to allow traffic to flow outside of the GRE or VXLAN tenant network. A router is also required to connect directly-connected tenant networks with external networks, including the Internet. The router provides the ability to connect to instances directly from an external network using floating IP addresses.
# T4 U9 O3 H- |" SFor the purpose of this document, we will create a provider network of type flat. After adjusting the init-runonce script and running it, there will be provider and tenant networks created connected by a router. Next, we will deploy the first OpenStack instance and access it from the external network:3 g/ u3 Z- n# O, S1 c @& |9 J
Deploying the first Instance# E& m5 Z& K0 q" P
OpenStack dashboard can be accessed via the Virtual IP (VIP) which has been configured earlier in the process. The following steps will show how to create the first instance and be able to access it from the external network using the OpenStack CLI. Similar steps can be following from the OpenStack dashboard to achieve similar results.
5 k$ \6 S3 n3 V" l. s- V& z7 fCreate a demo2 instance and attach it to the demo-net created by the init-runonce script:+ i" e0 m9 B4 I# r" V& t" [ J, J) G/ F5 S
# openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo22 C- e/ @, y9 f' t( J; n
Allocate a floating IP on the public (provider) network created by the init-runonce script:
$ }9 u9 F* W$ k4 U& u# f# openstack floating ip create public1
# A! h/ j! \$ N! Y" {% i, u# m Q' h# openstack floating ip list -c "Floating IP Address" -c "Port" ; x5 h5 N; |1 @+ m/ t' }
+---------------------+--------------------------------------+ & ]: n" |# t) [
| Floating IP Address | Port |
( q% b5 Q, l$ l- q# J5 m+---------------------+--------------------------------------+ ; t Z' G# Z" o# y( g, ^0 O
| 192.168.1.183 | ab1fa4b7-3e8d-4bcc-8b48-a53ad3360fa2 | " z+ e' @2 k4 d/ ~$ o) A" p, ]8 e
| 192.168.1.158 | None |
. k8 w/ C+ K: h' h8 y9 b+---------------------+--------------------------------------+ ; i( G. u2 M! H4 d5 h
Associate the floating IP with the demo2 instance created in the first step:0 j3 C: G! n1 ^4 a3 k
# openstack server add floating ip demo2 192.168.1.158% N f# T7 a" a
As a sanity check, make sure that the name spaces are updated correctly on the controller/network node:( V! @' I1 e& i' v% d* \* Z* Z
# ip netns exec qrouter-21c38859-4375-4bb4-9207-c74e45e12602 ip a7 W: }# b" w$ T; W3 @! {1 l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
/ r8 N6 N# Q7 d2 k* ^2 ? link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00- c2 |- d: U: u3 Z
inet 127.0.0.1/8 scope host lo- O, d0 K+ p$ Z$ J/ b
valid_lft forever preferred_lft forever) [ p( j0 O0 n" r* H, \7 g8 L9 ^) w
inet6 ::1/128 scope host' \: L" ~& X6 n
valid_lft forever preferred_lft forever: Q( b, d8 b% J. u: J' A: E% [5 K
16: qr-6f76c4b6-21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000( n( q/ _. g; x; f+ L2 E- l1 w
link/ether fa:16:3e:8e:f1:7c brd ff:ff:ff:ff:ff:ff- t6 s# h8 n# b/ g( s
inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-6f76c4b6-21
* P4 o( L* u* ^: P0 o valid_lft forever preferred_lft forever
8 E# z) l( a; ^ V; Y- b inet6 fe80::f816:3eff:fe8e:f17c/64 scope link
' {& o. n3 |! m; U$ R valid_lft forever preferred_lft forever* f* d8 x& I Z' H0 a4 K! R3 C
17: qg-3660ae4c-b8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
- H- l" c5 v; J% q( \: p" l2 V( C link/ether fa:16:3e:51:81:b3 brd ff:ff:ff:ff:ff:ff/ l; N) H' I0 D2 Q
inet 192.168.1.189/24 brd 192.168.1.255 scope global qg-3660ae4c-b8* G, ^# C' S% n0 m
valid_lft forever preferred_lft forever7 C9 e) Z5 U4 ]" n; k" o9 z
inet 192.168.1.183/32 brd 192.168.1.183 scope global qg-3660ae4c-b8
$ E9 f; W2 C0 e# F: m valid_lft forever preferred_lft forever: O6 m, z; Q+ U. Y& J$ l" O
inet 192.168.1.158/32 brd 192.168.1.158 scope global qg-3660ae4c-b8& x3 Y: {: i: ~; W. x
valid_lft forever preferred_lft forever" k7 h& w( p* U2 v) x7 q
inet6 fe80::f816:3eff:fe51:81b3/64 scope link
# m; s3 S( a0 J' c5 G4 x# u valid_lft forever preferred_lft forever" ^7 J: d( G: y* x
SSH into the OpenStack instance using the associated floating IP address from the external network:/ x3 g" | u6 ~5 l) U
# ssh -l cirros 192.168.1.158
: U+ ?' ^8 \5 t. _( eWarning: Permanently added '192.168.1.158' (ECDSA) to the list of known hosts.4 R( r4 O2 B8 P
$ ip a2 H/ w5 x& ^: O/ S9 c
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1% X' y& F, ~ R" O) h8 z
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
& C9 {% J- i& w$ s inet 127.0.0.1/8 scope host lo" c, _" C7 M6 P% |$ p- Q; w
valid_lft forever preferred_lft forever! R$ h6 v8 @) z# X+ O0 z2 Z) W
inet6 ::1/128 scope host2 U% e% I5 X4 ~2 `( ?
valid_lft forever preferred_lft forever ^4 H7 K! P1 N7 _ g7 d
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
' Q+ ^3 S3 ^( g! J4 X- K% W link/ether fa:16:3e:08:bd:30 brd ff:ff:ff:ff:ff:ff
& {; _: V1 E! y4 r inet 10.0.0.67/24 brd 10.0.0.255 scope global eth0
& ?2 g) M4 K1 O" w% U! s: O valid_lft forever preferred_lft forever! j9 x5 L6 P3 p$ F6 L( N( v0 h
inet6 fe80::f816:3eff:fe08:bd30/64 scope link% C4 V7 p7 I' J0 h5 T# ^
valid_lft forever preferred_lft forever7 j$ f+ }. w( M% m- q
$ 2 a- I7 q' P/ \) \$ Y
*** NEW SECTION **** g. p' P8 w+ P7 E
How to integrate Kolla-ansible with an external Ceph Storage
) c& H$ x) c5 \2 w& I: KUsing Ceph as backend storage for different OpenStack services greatly reduces network traffic and increases performance since Ceph can clone images/volumes/ephemeral disks instead of copying them. In addition, it makes migrating between OpenStack deployments much simpler.
` _6 z9 Z/ K+ B2 ?6 ^5 M% hPreparing the Ceph Storage Pools for OpenStack Services
: R4 J' h0 K/ ?/ \Glance Service
6 L0 J7 K0 K" m7 \- Y; R- r# sBy default, i.e. when not using Ceph, Glance images are stored locally on the controller nodes. When they are needed for creating a VM they are copied to the compute hosts (hypervisors). The hypervisors can then cache those images. But the images need to be copied again, every time an image is updated, or whenever a hypervisor undergoes a FULL install., W. Z$ }4 F- I( k1 z, c1 ?" ?2 q
If Glance is configured with Ceph as the storage backend, things work differently. In such case, Glance stores images in Ceph, not on the head node. Depending on the format of the image, they might still be downloaded from Ceph to the hypervisor node (and get cached there).
( e; B5 B) ^. ~( LIf both Glance and Cinder are configured to use Ceph, it’s possible to avoid copying the image from Ceph into the Hypervisors. This can be done using ‘copy-on-write’ (CoW).
2 x) z7 R6 |- c: c9 V# [* K! [With CoW, the storage volume of a VM is thinly pre-provisioned (from the image) directly in the Ceph backend. In other words, the image does not have to be copied from glance into the hypervisor, nor from hypervisor to Cinder. J5 h0 u! j, d) o, l0 X1 R5 w, g9 W' M
If Ceph is available we recommend to use it both for Glance, and Cinder so that the two services can make use of CoW. Note that for CoW to work, image format of images stored in glance needs to be “raw”. Using CoW often results in best VM creation times, and least load on the infrastructure. The drawback of using CoW is increased disk I/O latency if the VM ends up modifying existing files on it’s CoW-created volume. This is because data needs to be deduplicated the first time a unique block is written.( s( o4 f# F* c9 ^
To configure Glance with Ceph, on the Ceph cluster, run the following commands:
: r, {' }* u9 Q `: t6 ^/ S9 hCreate a pool for glance images:. R5 I) K4 P$ d2 e2 O y0 u! s! D
# ceph osd pool create images 64
2 D# W* J5 d6 c1 ]Create a user for glance service which has write privileges to the pool created in the previous step and store its keyring:
+ `7 N5 n4 W: Y. `# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' -o /etc/ceph/ceph.client.glance.keyring
* `1 J. n3 U; ? H5 CVerify if the pool has been created properly and user has the required privileges:
. B3 I+ V6 N$ f R! i9 `# ceph osd pool ls
* [# Z ~1 |2 j' J# E- Zimages
8 I3 {# o2 o' [. P* u% {6 i" d# rados lspools
: G8 X# {+ m t) C: m X" \, Z2 |images0 o9 G. a6 i7 d
# ceph auth ls/ i& b7 `! Y' v" \( \
[...]
" {: Q$ O" q7 e" Xclient.glance' o, e6 W+ u3 P, ~9 v5 {$ ^3 t
key: AQDniwRfNVzlJRAAKa9QDXlMGDerCPY7J8pAGw==0 j! q/ U; V# \; Q
caps: [mon] allow r
. O8 P! n7 Z' Q$ o& K, _ o" t caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=images" t4 l/ A3 Y: A; [6 B5 L0 |
[...]. e9 F; q( I" U+ t+ ~/ L9 x7 j2 _
** Note: How to choose Placement Group Number for a Ceph pool:
: y0 }" y. R. x8 `6 {4 oWhen creating a Ceph OSD pool, it is mandatory to specify a value for placement group number (pg_num) because it cannot be calculated automatically. From the Ceph online manual, we list here some recommendations:1 U! |& E3 k9 ]( z: m
Less than 5 OSDs set pg_num to 128.% y1 D) l9 O7 i& P' O. F
Between 5 and 10 OSDs set pg_num to 512.
% J* B2 _& w% W+ X0 @Between 10 and 50 OSDs set pg_num to 4096.
6 M6 x' U6 q5 z! S8 K/ xIf you have more than 50 OSDs, you need to understand the tradeoffs and how to calculate the pg_num value by yourself.
3 q0 ^2 n9 ]( jFor calculating pg_num value by yourself please take help of pgcalc tool.
6 E7 z4 d/ j4 o( uAs the number of OSDs increases, choosing the right value for pg_num becomes more important because it has a significant influence on the behavior of the Ceph cluster as well as the durability of the data when something goes wrong (i.e. the probability that a catastrophic event leads to data loss).! G+ V. t0 |& s. ?
Cinder Service
0 w. \4 P8 T% G0 p( {6 [# NCinder is the block storage service in OpenStack. It’s responsible for storing volumes and volume snapshots that can be attached to VMs or can be used to launch instances. In this document, we have demonstrated how to use NFS as a backend storage for Cinder service.
0 P3 f; u: P, ^5 Q* b2 B7 {Cinder can be configured to use Ceph as its backend storage instead of NFS.
" X3 ^ Q+ V+ A3 @+ y5 V6 YOn the Ceph cluster, run the following commands:; {' |1 r$ q) u( T
Create a pool for cinder volumes:
8 J" j/ V( t$ r! o" y2 J- Z# ceph osd pool create volumes 32$ ~& q" m; f$ ?) J: h
Create a user for cinder service which has write privileges to the pool created in the previous step and store its keyring:
* y, s9 w4 q. ^* L5 `/ F# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images' -o /etc/ceph/ceph.client.cinder.keyring" I Q e) _8 Z j. h4 u5 V! h& g
Verify if the pool has been created properly and user has the required privileges:
5 O0 I* V+ P0 b: H) S1 d$ e# ceph osd pool ls
! A0 H; K. g% U* W; ^8 _8 `# R# Rimages7 w& a% J4 w" h4 B. d: y# E
volumes
' m0 u4 Q- ^1 x#ceph auth ls
& z* T" u# a8 W/ W[...]& P" K( C8 W- |$ C1 E. `! c
client.cinder
1 a1 e) ~5 W+ P# T key: AQAZeQRfpfx0DRAAigGXZf4jbas9jQNDMUcIog==( [9 ^+ n, e7 J7 A$ H: D
caps: [mon] allow r3 _9 Z' V0 y$ T/ H3 ]6 t3 q3 k
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images
( `# s+ ^. t* g" U+ X' B& R# t! W[...]; ?+ _1 Z: A) B& o, P7 l- ?* S
Nova Service
; I9 ^: o5 g3 S+ k/ qNova is the compute service within OpenStack. By default, Nova stores ephemeral disks associated with running VMs locally on the hypervisors under /var/lib/nova/instances. There are a few drawbacks to using local storage on hypervisors, large images can cause filesystem to fill up, thus crashing compute nodes and a disk crash on hypervisor could cause loss of virtual disk and as such a VM recovery would be impossible. (if VMs are used with Cinder-manager volumes, those volumes are not stored on the hypervisors, they are stored in whatever’s configured as the Cinder backend)
+ _% S- F4 B. ]! WNova can be configured to use Ceph as its backend storage.4 m: J" |, |0 _7 |+ C6 z+ _1 n
On the Ceph cluster, run the following commands:
( ?) w- P. O; {1 SCreate a pool for nova ephemeral disks:1 v. d; c" H+ s+ T! T8 ^6 L
# ceph osd pool create vms 128
5 w1 [( @2 `& n1 M. KCreate a user for nova service which has write privileges to the pool created in the previous step and store its keyring:
) f" N# u' {) s3 p4 i# ceph auth get-or-create client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images' -o /etc/ceph/ceph.client.nova.keyring
8 H% E2 b. s! r# ~8 ^' HVerify if the pool has been created properly and user has the required privileges:9 e4 F! l( a' {- | c j( R2 x
# ceph osd pool ls9 G. \2 h# @" v3 Q0 t7 x$ L
vms
9 d4 J+ u5 w# s* K5 D! o9 C4 @images4 Q$ _' ~# n" n9 f P# G7 B
volumes0 m1 D$ ]( k- c, e
[...]/ ^* [. n# m7 w+ j. r
# ceph auth ls
3 U1 \0 S; X6 k( y[...]% i. f' W' _0 j5 L% N/ V: H8 F" s6 \" \
client.nova
. O; t: s; ?2 E2 j1 |; w7 n key: AQB6dwRfgrhaGRAAxXB/6YKcXIW+eYdoiSSMlg==* p4 O- u' e8 S( k1 p& \& k
caps: [mon] allow r* z# c7 Z% T6 x6 |
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images
A r" K/ y% U1 S" R# o$ L$ L& \[...]" ]. J: {! k7 Q6 s, t
Preparing Kolla-ansible configurations to use Ceph Storage as backend storage for Glance/Cinder/Nova services
$ s( z2 \8 {2 g- n+ p. aThe globals.yml is the main configuration file for Kolla-Ansible. In addition to the parameters mentioned at the very beginning of this document, there are a few other parameters that are required to enable several OpenStack services to use Ceph as their backend storage:
. S% i, p3 ?$ N4 k* J0 W7 I+ eThe parameter ceph_{ service }_user is used to specify the Ceph user that will be used to write the Ceph pool for a particular service. The variable “{ service }” should be replaced with the service name like glance, cinder, or nova.
0 j4 v2 e2 @* K3 q1 O" Z1 D; v7 f/ OThe parameter ceph_{ service }_pool_name is used to specify the name of the pool created in Ceph which will be used by a particular service. The variable “{ service }” should be replaced with the service name like glance, cinder, or nova.
. J- K& `. \! R1 [& IFor Ceph users to be able to read/write Ceph pools a keyring is required. The keyring is specified by ceph_{ service }_keyring which points to the location of the keyring file for specific service. The variable “{ service }” should be replaced with the service name like glance, cinder or nova.
" a8 m% n* Q5 J b3 c' _9 KThe parameter { service }_backend_ceph is used to enable/disable a particular service to use Ceph as its backend storage. The variable “{ service }” should be replaced with the service name like glance, cinder, or nova.
: p7 H# ~7 L2 ~# grep -vE "^#|$^" /etc/kolla/globals.yml; k. B C8 j' r; w
---2 \. R( U9 O) r( ~1 P- x) H8 |6 {
kolla_base_distro: "centos"
6 F% c8 i' N; w+ G6 r1 Jkolla_install_type: "source"
0 U$ R8 R. d9 f5 @5 W8 y1 Jopenstack_release: "train"1 U q+ x' ~1 Q* l5 t2 N9 d2 I( p
kolla_internal_vip_address: "10.141.255.245"
3 K9 m8 f) L0 F8 U* f+ {+ Wnetwork_interface: "eth0"
0 U# q1 B/ \ K6 L' `/ nneutron_external_interface: "eth1"
' G1 T6 H- H8 [+ @* uenable_cinder: "yes"2 o% _6 R5 C: p; \
ceph_glance_keyring: ceph.client.glance.keyring J1 M8 s8 \0 j2 e {
ceph_glance_user: glance
5 E/ Y- |; z/ q) I4 B0 `0 wceph_glance_pool_name: images
- t2 }" p q5 D s' R; x. r) u- jceph_cinder_keyring: ceph.client.cinder.keyring. j0 }1 n1 `* \2 _ j5 ~4 J+ z- E
ceph_cinder_user: client.admin6 V# F" R6 a. F( ^! r1 R
ceph_cinder_pool_name: volumes& l9 j3 E0 D; _! q
ceph_cinder_backup_keyring: ceph.client.cinder-backup.keyring* Q7 s/ d" T. s; D
ceph_cinder_backup_user: client.admin
C& a' G# c; o! q" }ceph_cinder_backup_pool_name: backups
+ R3 T5 I6 o1 j* V6 yceph_nova_keyring: ceph.client.nova.keyring
* d7 E0 T# t% ^! J2 q7 N: P/ Oceph_nova_user: nova! t% `9 ^, b! C& V* V1 ?9 M
ceph_nova_pool_name: vms
. m* l, B1 z# oglance_backend_ceph: "yes"
& H( {. x& H' Z d; W8 i+ A8 ccinder_backend_ceph: "yes": r7 Q1 Y1 Y1 X S2 v% f
nova_backend_ceph: "yes"$ L2 v# z# e6 \2 n$ r8 P, n
** Note: the “ceph_cinder_backup_pool_name” is the same as the “ceph_cinder_pool_name”, however, it’s not strictly necessary to have both pools the same. You can create a separate pool for cinder_backup_pool.
4 \8 x% v) N$ `8 O! EGlance Configuration
/ n; L/ z6 Z/ g# q# Y7 t: E6 HThe configuration files which will tell Glance how to use Ceph should be placed in the location where Kolla-ansible would expect to find them:
& P0 O4 ?6 N9 _7 j6 @+ W- k3 tCreate a configuration directory for glance:/ _8 G5 B2 x; o. w
# mkdir -p /etc/kolla/config/glance4 d& ?3 P9 {+ |1 H4 H- ^8 B
Copy ceph.conf to the same directory created in the previous step:
, ]* @) }8 o" _+ b, W; C# cp /etc/ceph/ceph.conf /etc/kolla/config/glance/) g* }# @% V6 n( x+ q% ]
Copy the keyring to the same directory:! t6 q0 R6 R# r2 Q6 B2 i- O: W
# cp /etc/ceph/ceph.client.glance.keyring /etc/kolla/config/glance/
4 [# E' u: A, K! l) H' h7 f" H- vCreate glance-api.conf in the same directory with the following contents:
4 y+ f& J' Y9 s- n6 ^. t; ?: F3 x" G# cat /etc/kolla/config/glance/glance-api.conf
' {" [5 B Z5 A$ o[glance_store]: f6 Z2 m) |6 |1 t# P& l
stores = rbd0 P0 v8 J7 h( L6 I
default_store = rbd
: e. u9 l8 E* trbd_store_pool = images# e0 h( ?3 C; e8 C+ T+ b& }
rbd_store_user = glance
( p$ {) W: J/ J' s Crbd_store_ceph_conf = /etc/ceph/ceph.conf
' h! A% X" O9 N. X4 MCinder Configuration. @" q# V! W G% K6 z- j$ @
The configuration files which will tell cinder how to use Ceph should be placed in the location where Kolla-ansible would expect to find them:
* f' f- X0 X: v5 DCreate a configuration directory for cinder:
4 O' X2 k8 ]* K r# mkdir -p /etc/kolla/config/cinder/{cinder-volume,cinder-backup}# i/ x) ~9 Y ?+ N# ~7 ^. {
Copy ceph.conf to the same directory created in the previous step:
& e: Z' g7 r+ ~# cp /etc/ceph/ceph.conf /etc/kolla/config/cinder/ceph.conf* f5 y) u. i) D! y1 u
Copy the keyring to the same directory:$ T/ D- S/ T4 D1 f6 O: Y# g5 R
# cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring7 Y8 K" X; I+ k% Q& j
# cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring
l2 [8 n2 b& y- J1 {# cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring# ~# v( _4 T: K
Create cinder-volume.conf and cinder-backup.conf under /etc/kolla/config/cinder directory with the following contents:
5 \. r5 H! Q# U1 j* g' ^8 O# cat /etc/kolla/config/cinder/cinder-volume.conf" ?! X! o& {4 Q1 s- l! A
[DEFAULT]
) C0 I1 r: r N0 [3 h9 g* x( W& I }$ ?enabled_backends=rbd-1
2 e# z1 F/ |9 J8 G; `/ S[rbd-1]4 U" K+ |4 ^# a" |* H- x
rbd_ceph_conf=/etc/ceph/ceph.conf# \5 W. A, O Z A' {- `
rbd_user=cinder! B% E' `7 g% Z! M) }
backend_host=rbd:volumes
2 R, s' ~- h1 h4 rrbd_pool=volumes7 P: c+ `* U" d8 `2 v5 E1 x
volume_backend_name=rbd-1
9 I. [4 C) N9 ^volume_driver=cinder.volume.drivers.rbd.RBDDriver
4 [& A& v% R& m0 b, }8 ?7 i/ Y+ n( Irbd_secret_uuid = {{ cinder_rbd_secret_uuid }}
) }& x! Z. U8 {. ]# R' @, o# cat /etc/kolla/config/cinder/cinder-backup.conf
' L2 A8 X$ L+ t[DEFAULT]
' N5 h! F5 v. [backup_ceph_conf=/etc/ceph/ceph.conf
& `( G& [# u3 d+ y. f0 a6 {backup_ceph_user=cinder
X# [- B: z8 a+ p& [+ s! x$ ibackup_ceph_chunk_size = 134217728
( A! t: W: i# g8 `8 Mbackup_ceph_pool=volumes0 d$ ^! A3 _/ l" E- Q
backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
/ \7 `, _+ o' C. V" n' P* t2 Pbackup_ceph_stripe_unit = 0
2 H/ P: h* d* } y7 ubackup_ceph_stripe_count = 0% U T- f* ~5 R) T$ K
restore_discard_excess_bytes = true/ L8 ~; ~* g4 W1 w; v( x
Note: {{ cinder_rbd_secret_uuid }} can be found in /etc/kolla/passwords.yml9 j9 N" F% z4 A; N- K% U& P5 l/ x$ T
Nova Service
1 ^8 R; l1 B1 b" JThe configuration files which will tell cinder how to use Ceph should be placed in the location where Kolla-ansible would expect to find them:
, ?8 b1 N7 F' w8 V \5 m* \Create a configuration directory for nova:
5 z6 p. p8 i9 J" p- L" P8 c: X# mkdir -p /etc/kolla/config/nova7 U1 Y) {; \, l; W0 c
Copy ceph.conf to the same directory created in the previous step:$ L! P3 Z0 {- @
# cp /etc/ceph/ceph.conf /etc/kolla/config/nova/
, k1 b6 Q. u5 t& Y' Q1 ^. }Copy the nova and cinder client keyrings to the same directory:
8 W, o0 p' q- }' n; R c A# K% }# cp /etc/ceph/ceph.client.nova.keyring /etc/kolla/config/nova/ceph.client.nova.keyring
/ p5 v7 o& O" T# cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/nova/ceph.client.cinder.keyring/ V) Q- ^$ Z0 ^: l$ r
Create nova-compute.conf in the same directory with the following contents:+ ?3 {! M* ^' ?8 i4 \
# cat /etc/kolla/config/nova/nova-compute.conf/ }: }' q3 X% G& V7 K
[libvirt]/ q, ^7 r0 z% N# p& _* _/ D
images_rbd_pool=vms K% U( s: d8 @* V0 q f
images_type=rbd
6 P# X9 c1 Y$ F5 j5 E0 c, z( kimages_rbd_ceph_conf=/etc/ceph/ceph.conf
6 C4 R1 c) ~ a% H- j8 p; prbd_user=nova
# H2 [ a! x1 ~* n3 W' ~Verify if OpenStack is writing to Ceph Backend Storage; S& ?6 b, P u3 ~
After deploying kolla-ansible following the steps mentioned earlier, you can verify if OpenStack is writing data to Ceph by inspecting Ceph storage itself:
; s' T6 {: r( v5 P! L( ATest creating images (glance service)/ H$ p; p( q* o `
# openstack image create --disk-format qcow2 --container-format bare --file /tmp/cirros-0.4.0-x86_64-disk.img cirros
1 Z1 X& L! P# J! f3 k7 w; n# rados df. h) W% l* U% o% {
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
% P( _5 x1 |& i6 j7 j1 Eimages 37 MiB 8 0 24 0 0 0 90 70 KiB 21 12 MiB 0 B 0 B9 C- [* @& b/ |1 d. c( d
vms 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
, T& k( z) P; G# C6 Qvolumes 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B# e/ M. b: X. v6 g6 Z
total_objects 8
: U8 n* k. s2 J, O; R; ~total_used 3.0 GiB2 P9 r$ Q0 v* ]5 |6 i' Y
total_avail 297 GiB2 k* o# v3 V6 Y+ A. k
total_space 300 GiB
5 P3 Y3 z+ B' }- c# rados -p images ls2 } |- I& L2 |5 w& O1 y% L
rbd_data.179c773a98d0.00000000000000010 O1 R8 d3 m% K% a
rbd_object_map.179c773a98d0.0000000000000004: ^& Z" [2 e/ G" p/ @
rbd_directory1 T. U& J/ Y5 B# L
rbd_info6 S6 y; N( w2 r
rbd_object_map.179c773a98d0
/ i, l* v6 \8 s, P2 Hrbd_header.179c773a98d0
# P, R& k! e2 C( ~4 }4 c3 Trbd_id.9223ba8b-ddbd-4b69-a5ea-db1fb31ee1e0
: b5 h7 d* w# N+ g. ~0 Nrbd_data.179c773a98d0.0000000000000000
. c! Z9 q* V# Z# V1 SAs you can see in the previous output, after creating an image, it is present in the Ceph storage.5 a: k0 m/ m$ E( @. ]1 f
Test creating a volume (cinder service)
" P+ E2 H- S/ R T7 d) |; k( I. o' k# openstack volume create --size 10 --availability-zone nova mynewvolume
' w& V* }& U4 X+---------------------+--------------------------------------+
8 ]/ y% L; q, _6 r8 K3 R, {| Field | Value |, a# t' V5 c4 \3 i7 F
+---------------------+--------------------------------------+
: r" W3 F" ]# ^" d9 A| attachments | [] |
' R/ u) f1 ~# O- l" J6 m2 _| availability_zone | nova |
! ], Y7 o# m( f V. ?| bootable | false |
0 K4 A) I% Q# N7 Z| consistencygroup_id | None |) Y# u2 ~" d0 l( X( \1 N+ G8 s
| created_at | 2020-07-09T15:59:17.000000 |. ]' E0 o2 W% e2 W
| description | None |8 g' ]; K" s. d j X
| encrypted | False |
' ^. G- a6 I; d| id | 1ee10849-f948-46b0-ae33-d7bbd5733c6a |4 W+ N& R; u" M0 e
| migration_status | None |
3 k- A$ M+ [4 N| multiattach | False |3 O" p# @7 k; N W% s
| name | mynewvolume |
* l4 B6 j0 J! Z/ T8 O) A/ q| properties | |
" R8 J& w1 H3 r1 w| replication_status | None |
6 K+ X6 V: W* m; q" g& R& y1 d| size | 10 |
* ?& `7 V5 a$ x3 z" x| snapshot_id | None |
) e8 c! w; c# i/ j N| source_volid | None |1 M3 [6 a/ Z/ S- T! K# `
| status | creating |
3 W& y+ H6 }) ] }| type | __DEFAULT__ |) s+ s$ [2 ?# c0 [0 s$ X
| updated_at | None |
: Q5 Z7 ]% R; X! J& X6 H| user_id | be4c99e41dab462a881b55585dbcfe15 |( @) @+ u7 M. V$ _4 K l. p
+---------------------+--------------------------------------+. b A6 `) Q: Q7 ` G" g
# rados df
8 r, c$ c- h4 `- F# Y7 U' LPOOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
; r( o: U+ a: R5 E0 nimages 112 MiB 20 0 60 0 0 0 267 209 KiB 59 36 MiB 0 B 0 B
& m# d: Y4 L4 ~% e& Rvms 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
# p4 f) ?: }+ e- T+ rvolumes 576 KiB 5 0 15 0 0 0 36 27 KiB 6 6 KiB 0 B 0 B; u* Y4 i" P4 w2 p6 l3 }
# rados -p volumes ls
- \! w6 d! Q( T- |6 G4 Q! irbd_header.a554d675abc8# X1 O( w9 r- f* K2 @# U
rbd_directory J! [- W+ o3 S' ?& Y
rbd_object_map.a554d675abc8
5 `" m. `8 z& J0 arbd_id.volume-1ee10849-f948-46b0-ae33-d7bbd5733c6a% F: V& I1 ^$ H# \& b
rbd_info
$ B) Z. n) l0 D: iAs you can see in the previous output, after creating a volume, it is present in the Ceph storage.0 X5 `* f" W+ _# C6 ]" U
Test creating a VM (nova service)
( H3 l. _: d2 J" X6 T# openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo11 t" ^. a/ ~+ R+ R: s
+-------------------------------------+-----------------------------------------------+& M( z+ i, `- y* b
| Field | Value |
: V$ u% `' B1 L K! u- F+-------------------------------------+-----------------------------------------------+9 M3 a/ h. \' G
| OS-DCF:diskConfig | MANUAL |! a5 x; D+ ]. u
| OS-EXT-AZ:availability_zone | |
, S, g, ~! X$ h& b/ U d| OS-EXT-SRV-ATTR:host | None |
7 z |2 j [# |; I8 N* \$ || OS-EXT-SRV-ATTR:hypervisor_hostname | None |
# M+ y! \* m3 S9 B| OS-EXT-SRV-ATTR:instance_name | |8 K( x$ ^. p7 T( i9 }2 q9 I
| OS-EXT-STS:power_state | NOSTATE |- X4 S, W/ _- }1 Q
| OS-EXT-STS:task_state | scheduling |( ~2 C- Y! N1 C& r+ X
| OS-EXT-STS:vm_state | building |
2 O9 m7 M2 s2 P| OS-SRV-USG:launched_at | None |
3 v7 I6 ]4 V9 |0 H/ c| OS-SRV-USG:terminated_at | None |
3 e0 ^5 P3 d. A| accessIPv4 | |; k% @ I6 m5 X; K: Z' i; R
| accessIPv6 | |
, ?- l% g1 v+ s- _9 A! C| addresses | |' a s# I5 {& P! K" s
| adminPass | JqLnjPKb5Awj |. H( X; @& |1 \
| config_drive | |
' @6 X# Y; N# x) b. U }+ N" @| created | 2020-07-09T16:07:47Z |1 v' E4 H2 F! S/ a
| flavor | m1.tiny (1) |) W4 Q- L4 {. X
| hostId | |1 M' Q4 b2 l3 J' P# m: F
| id | a2fdb85d-d6bf-4150-ab96-dd22b7d4e9d4 |
# C2 s) A) r, p| image | cirros (2de81533-2410-4c67-baa4-5260cfb9f1a8) |. A: M" h6 u( h
| key_name | mykey |
" H. h# w, U! Q' ^( @- G| name | demo1 |
4 D) a- h, [/ h B5 @| progress | 0 |" }7 n r' V! y% J& e" P
| project_id | b108a02fe6b24fffa88b7de24e05b685 |# S$ z. J( p, h* H* v
| properties | |
4 p- g( K" Q2 P' U# Q6 B| security_groups | name='default' |
) i4 n& F/ t% k- i| status | BUILD |7 d2 M' y+ K" c/ X
| updated | 2020-07-09T16:07:48Z |( F! T( w0 L# \. m& {
| user_id | be4c99e41dab462a881b55585dbcfe15 |
+ I& u2 K0 E, e2 b' @| volumes_attached | |+ I+ Y" p0 f7 s
+-------------------------------------+-----------------------------------------------+0 Z0 ?: \. u. l
# openstack server list1 X( O* v3 O3 j9 z( Z2 a: }: B
+--------------------------------------+-------+--------+---------------------+--------+---------++ X C, ^" U( `2 p; S& i0 J5 K: i
| ID | Name | Status | Networks | Image | Flavor |
2 G* q1 W6 |, I; P! _' c+--------------------------------------+-------+--------+---------------------+--------+---------+5 G" P/ V0 e' A( t- p. [6 G8 A1 W
| a2fdb85d-d6bf-4150-ab96-dd22b7d4e9d4 | demo1 | ACTIVE | demo-net=10.0.0.134 | cirros | m1.tiny |& C; M% H4 S/ G8 C: e; O* k2 s: h
+--------------------------------------+-------+--------+---------------------+--------+---------+
0 u* n6 Y: J$ A- m8 z7 b; U: ~2 J# rados df / S. |/ \# c+ M
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR $ D. T7 K; j% r8 P* |* I
images 37 MiB 8 0 24 0 0 0 520 13 MiB 105 49 MiB 0 B 0 B
1 \* c% |( j# j3 }vms 207 MiB 138 0 414 0 0 0 1880 19 MiB 987 58 MiB 0 B 0 B & v0 m; A6 q, F# z* k9 L7 I! H
volumes 576 KiB 5 0 15 0 0 0 306 243 KiB 6 6 KiB 0 B 0 B
8 k: c( S% ~. R2 M. \ 9 }! K( b* ^ ]' x. r- g
total_objects 151
) e" @: u; Y2 |- I* Q) dtotal_used 3.3 GiB
6 y6 \/ H. f6 B; E1 b+ k% Wtotal_avail 297 GiB
# \: R& d2 b6 ~# ktotal_space 300 GiB
' d" X4 m. U# E3 n* F3 Xrados -p vms ls | head" k- T7 p' w9 J
rbd_data.a6b31261fa8f.0000000000000003) Q+ r% p. n, x2 s6 A1 @
rbd_data.a6b31261fa8f.000000000000009c
: g" L, i4 _% U9 [1 X9 l4 p* frbd_data.a6b31261fa8f.000000000000007c8 P+ l8 K& ]% U; E" o5 @
rbd_data.a6b31261fa8f.00000000000000284 Z# g0 V0 `0 ]2 F! R) v
rbd_data.a6b31261fa8f.0000000000000022
* p, P) K' j, \rbd_data.a6b31261fa8f.00000000000000c4
: `/ x' M# o1 U5 m, |' f: i! Prbd_header.a6b31261fa8f
: E/ n) a$ u$ C Arbd_data.a6b31261fa8f.000000000000002a/ u5 }2 c7 w% V1 w7 |8 f0 n
rbd_data.a6b31261fa8f.00000000000000fe: I7 W* {, X* C' x- I
rbd_data.a6b31261fa8f.0000000000000006
0 |* I) K% ~1 h0 w4 l U7 ?9 t$ kTroubleshoot
& V+ ]3 o* B7 |( h0 iIssue: “kolla-ansbible deploy” step failed with the following error:! ]- [/ d; s( V" r8 `. F( d" ], N
TASK [nova : Running Nova API bootstrap container] *********************************************************************************************************************************************************+ N0 C/ E5 i1 |: n
fatal: [node002]: FAILED! => {"changed": false, "msg": "Unknown error message: open /var/lib/docker/tmp/GetImageBlob027787082: no space left on device"}' q4 g: N! S: @4 R8 L% n
NO MORE HOSTS LEFT *****************************************************************************************************************************************************************************************
" E9 b8 S; h# l( r$ ~$ m A2 b m4 wPLAY RECAP *************************************************************************************************************************************************************************************************7 J. S1 q# t: G8 v1 Q2 K6 L( r, \
localhost : ok=48 changed=12 unreachable=0 failed=1 skipped=17 rescued=0 ignored=0
/ w" `9 ]9 I' [0 i. j( a1 Nnode001 : ok=40 changed=7 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0& Z. \) V" h" l( v5 l4 ?2 S
node002 : ok=202 changed=99 unreachable=0 failed=1 skipped=79 rescued=0 ignored=1 - i6 X7 t" W* P2 v
node003 : ok=78 changed=10 unreachable=0 failed=0 skipped=76 rescued=0 ignored=0
( ~2 H* i. H, Q: a+ h9 v% nResolution:
8 X: E0 b8 N& `The openstack node(s) (controller or hypervisors) ran out of space and no more containers can be run. A bigger disk at, least 60G, should be used for controller and hypervisor nodes.. x v. V3 p$ ?1 h
Issue: kolla-ansible deploy fails after haproxy container fails to start1 f3 o4 X2 a! w
RUNNING HANDLER [Waiting for haproxy to start] ************************************************************************************
4 T( h- s% t9 g. Y' r4 h9 bfatal: [node003]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 10.141.0.3:61313"} Y. [- b6 `+ y. r7 {9 F) r
RUNNING HANDLER [haproxy : Waiting for virtual IP to appear] **********************************************************************# o5 s$ S( ?/ G) P& U" [9 \
NO MORE HOSTS LEFT ****************************************************************************************************************7 j/ ^0 w1 a$ Y1 B
PLAY RECAP ************************************************************************************************************************
5 A; p7 X7 N5 T5 H' elocalhost : ok=35 changed=0 unreachable=0 failed=0 skipped=7 rescued=0 ignored=04 C9 K3 k4 S: h9 F
node001 : ok=35 changed=0 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0
4 u; V8 ~8 g. j- p6 y0 k) j! c- Enode002 : ok=35 changed=0 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0( E: _7 h$ s. \3 }: r8 t. K
node003 : ok=71 changed=3 unreachable=0 failed=1 skipped=76 rescued=0 ignored=0 _5 ?9 R: R1 ^
Command failed ansible-playbook -i ./multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /usr/local/share/kolla-ansible/ansible/site.yml1 C, [3 O3 l3 W4 W! }- \# n
Resolution:
T1 Y) z+ H- \' @These sort of issues usually happen when a higher kolla-ansible version is used with an older openstack release. Always make sure that the openstack_release in globals.yml points to a release which is supported by the currently installed kolla-ansible version.
+ \8 \& F/ m4 KExample, kolla-ansible version 9.1.0 supports OpenStack “train”, while kolla-ansible version 10.0.0 supports OpenStack “ussri”.. A/ t/ ^. P4 t9 u( m6 ]% w
Issue: I have rebooted all my OpenStack controller nodes at the same time and now mariadb container is continuously failing.
' q8 t7 W: T5 ~ iResolution:2 f2 X( x* ]+ e* {3 g
It is highly likely that the database cluster has become inactive and thus a database recovery should be attempted:
! ~. n k' W+ L/ h# kolla-ansible -i ./multinode mariadb_recovery
9 _1 }4 k4 z1 f. U+ @9 f1 m- ?6 xTips and tricks
$ e* H. }" h. L1 L/ G" sWhen running the kolla-ansible CLI, additional arguments may be passed to ansible-playbook via the EXTRA_OPTS environment variable.1 J, k) F* `% r* ~+ L
kolla-ansible -i INVENTORY deploy is used to deploy and start all Kolla containers.. n/ x8 I1 r5 Y
kolla-ansible -i INVENTORY destroy is used to clean up containers and volumes in the cluster.
; }7 ~. _9 J4 D& E% a6 P) d( Mkolla-ansible -i INVENTORY mariadb_recovery is used to recover a completely stopped mariadb cluster.! j% w4 H E; q; f
kolla-ansible -i INVENTORY prechecks is used to check if all requirements are meet before deploy for each of the OpenStack services.$ B9 ]3 M5 n( Y, D1 c' a, p
kolla-ansible -i INVENTORY post-deploy is used to do post deploy on deploy node to get the admin openrc file.7 E9 ~, k( v% D; r. U. D( }5 C
kolla-ansible -i INVENTORY pull is used to pull all images for containers.6 `( Z/ H: t0 {* G8 e* K1 n
kolla-ansible -i INVENTORY reconfigure is used to reconfigure OpenStack service.
% ?! R" w4 V1 h& |! Ykolla-ansible -i INVENTORY upgrade is used to upgrades existing OpenStack Environment.
! y* L$ o3 [1 p7 zkolla-ansible -i INVENTORY check is used to do post-deployment smoke tests.
, p( ~6 G _ @ `* Nkolla-ansible -i INVENTORY stop is used to stop running containers.
& `) I. r, S% F3 t3 d% S. U- Hkolla-ansible -i INVENTORY deploy-containers is used to check and if necessary update containers, without generating configuration.
; a4 \& ^) L0 W. u" V0 f, Kkolla-ansible -i INVENTORY prune-images is used to prune orphaned Docker images on hosts.& L z' d& q- s5 b, f
|
|
发表于 2021-5-31 11:27:23