|
|
Kolla-Ansible ease of configuration and deployment highly depends on the uniformity of the network interface names which will be used by OpenStack on the controller/network/hypervisors/storage/etc. nodes, so it’s highly recommended to unify the network interface names across all the nodes, including the head node if it will be used as the deployment node.
* s6 ]- K% x2 QUnifying the network interface names on CentOS 8 can be done in several ways:0 G2 k3 }$ `. m7 [* |/ s9 Z# s
Switch to traditional naming conventions for network interfaces:
0 t8 O# m& B) _0 ^) b+ E" NFor the compute nodes, this can be done by appending net.ifname=0 to the kernel parameters in the software image:
4 K+ g. t& g3 C4 ~" f1 _- d# cmsh
* S1 I* B: |& Y! [% softwareimage use <image-name>
6 h8 Z2 t+ m. e9 | N9 X+ h. t, H8 N* f% append kernelparameters ” net.ifnames=0″
" ~/ B: O- Y+ }% commit
& c- b) Z; [! nFor the head node(s), this can be done by updating the GRUB_CMDLINE_LINUX parameters in /etc/default/grub.conf and re-writing the /boot/grub2/grub.conf
' p8 C( n' b) e0 y# grep net /etc/default/grub; y y# ?' x" F9 e; |8 r9 _2 v3 t
GRUB_CMDLINE_LINUX=”vconsole.keymap=us crashkernel=auto vconsole.font=latarcyrheb-sun16 rd.driver.blacklist=nouveau net.ifnames=0″
9 D& v9 i4 h. }# y# grub2-mkconfig -o /boot/grub2/grub.cfg
& G& K5 A# G$ @% i# reboot
/ E& l8 i5 z9 }) n e7 RSpecify the MAC address and device name in the corresponding ifcfg-* files.
5 |. V+ j* f, U2 g! ZKolla-ansible is installed via “python-pip” so it’s mandatory to update “pip” to the latest:' [, W' c- s7 i. E' j6 m
# pip3 install -U pip
4 X$ O- c' ] q$ ]By default, Bright assumes that there are no important data stored on the compute nodes so each time a compute node is rebooted, the software image is re-synced to the local drive of the compute node resulting in adding any missing data or wiping out any data which don’t exist in the software image.
% X9 N2 y+ ?: r! K& [, TTo avoid wiping out the data that will be introduced by the Kolla-ansible deployment, we will need to add the following paths to the sync/update exclude lists: j+ b" C; i- m6 h( ^* d4 e
– /var/lib/docker% f: R# s f* x+ o
– /var/lib/docker/*/ q/ d' c* J% P" h( ^2 G L. T
– /etc/kolla4 U) s {. q( ^# g
– /etc/kolla/*% L6 ~( p' L1 B* L6 S0 x
To avoid pulling unnecessary data from the compute nodes that will be deployed by the Kolla-ansible deployment, we will need to add the following paths to the grab exclude lists:
2 u; u; z$ a2 S( G1 n7 W4 E( Z– /var/lib/docker
# D- o5 x2 L R% ^1 k! X6 t: P) T– /var/lib/docker/*
; @9 u: L/ {0 Z9 h& }# u& i– /etc/kolla
- s) v1 K8 }4 j9 R, M: E) ?! n– /etc/kolla/*
d0 A6 T, q) @" ?: I$ V$ D/ `** Note: The exclude lists are better updated on the category level. If the setup will only include one compute node, then it may be better to update the exclude lists directly on the node.% M( m5 P) a2 Q
Installing Ansible
+ q/ B1 f0 o- s3 r$ d5 ?' MKolla-ansible requires ansible version 2.8 or higher and up to version 2.9 as of writing this document (June 2020)
5 w1 q1 c& P T# j$ kyum install epel-release -y
6 Y! d9 K, y- b# U& O5 Oyum install ansible -y7 Q. e! u( }$ T; |! E
Configuring Ansible (optional); u& L# J0 G N/ a7 R
Tuning the configurations of Ansible may improve the deployment time of Kolla-ansible so it’s suggested to do tune the ansible configurations for better results:
' L/ \; H2 v; Q: n+ n+ SIncrease the value of the forks as you see necessary. The forks parameter controls how many hosts can be configured in parallel, so if you have the processing power on the head node to run several forks, then it’s a good practice to increase the number of forks.+ ]: @6 d" o/ [$ n
Disable host_key_checking. Ansible has host key checking enabled by default. If a host is reinstalled and has a different key in ‘known_hosts’, then this will result in an error message until corrected. If a host is not initially in ‘known_hosts’ this will result in prompting for confirmation of the key, which will require an interactive intervention.0 k- o3 @( X4 {; j4 _( y
Enable pipelining. Enabling pipelining reduces the number of SSH operations required to execute a module on the remote server. This can result in a significant performance improvement when enabled. By default, this option is disabled.
" ?" z& q/ o* b, k# grep -E “forks|host_key_checking|pipelining” /etc/ansible/ansible.cfg | grep -v “^#”
" B7 |+ G! t8 s: f) Gforks = 100
2 O0 n7 r$ J3 q6 V4 fhost_key_checking = False/ w! x6 i& _( c2 x }5 I
pipelining = True5 o6 \( ~. v9 f# J; V8 w, x
Installing Kolla-ansible+ p/ d: @5 d9 L: r" o: L$ j
Kolla-ansible and its dependencies can be installed using python pip:- l; q: n% {/ w
# pip install kolla-ansible
. X9 a' ~$ R: t9 m8 O9 ICreate default configuration directory. By default, Kolla-ansible expects the global configurations to be stored under /etc/kolla on the deployment node.
9 M- N/ j: A% J! B' M! Y# mkdir /etc/kolla3 k/ b# J5 p, x
Copy default globals.yml and password.yml templates from Kolla-ansible installation directory:
! t, m+ M4 Y- S, b: c# cp -r /usr/local/share/kolla-ansible/etc_examples/kolla/* /etc/kolla
; O' T! T5 H0 y- v* \$ e1 r9 BCopy all-in-one and multinode inventory files to the current directory.
5 @$ |8 q( U/ ^" G# cp /usr/local/share/kolla-ansible/ansible/inventory/* .
+ U- a ~' w4 Y, ^1 d( |0 yPreparing initial Kolla-ansible configurations
9 }1 J: P1 v- ]& v- @/ e/ sThe globals.yml is the main configuration file for Kolla-Ansible. There are a few parameters that are required to deploy Kolla-ansible:
; m" [3 s* O# m7 [) o8 dKolla-ansible provides container images for deploying OpenStack on several Linux distributions as host system: CentOS 7/8, Ubuntu 18.04, Debian and RHEL 7/8. For the purpose of this document, we will use CentOS 8 as the host system. Using kolla_base_distro parameter, “centos” can be specified to refer to CentOS as base distribution.
. G7 i6 c$ a) M' t( D- Y% ]! lType of installation can be either binary: using repositories like apt or yum or source: using raw source archives, git repositories, or local source directory. According to the official Kolla-ansible documentation, type source is proven to be slightly more reliable. The install type can be specified using the kolla_install_type parameter.4 E6 o! a! U5 z8 V, n; [) Y
The release of OpenStack to be deployed has to be specified using the openstack_release parameter. For the purpose of this document, we will use the OpenStack train.
/ G2 _! Q1 N B2 E. [) j1 @Kolla-ansible requires two main networking options, network_interface: the default interface for multiple management-type networks (tenants networks) and neutron_external_interface: a dedicated network interface for Neutron external/public networks (provider networks) which can be vlan or flat, depends on how the networks are created. This interface should be active without an IP address on controller/network nodes. Otherwise, OpenStack instances won’t be accessible from the external networks. For the purpose of this document, we will use flat networks.
0 a$ H' j( C- n' }! k% ]A floating IP for management traffic should be specified using kolla_internal_vip_address. This IP will be managed by keepalived to provide high availability and should be a free IP address in the internal management network that is connected to the same network_interface.' f4 q7 E! o8 l6 T
Additional services can be enabled. For the purpose of this document, we will enable the cinder service using enable_cinder and enable_cinder_backend_nfs parameters. The cinder service with NFS backend requires an additional configuration file that has to be placed under /etc/kolla/config/ with the name of nfs_shares. This file includes the name/IP of the NFS servers and the path on the server. The file can include several NFS servers and different paths:$ X# d9 ~" k1 ?+ D4 ^$ v1 o
# cat /etc/kolla/config/nfs_shares
' B3 A0 d4 {1 z8 S5 ?! d10.141.0.1:/cinder
! _/ g; {- a! t/ ]5 m# grep -vE “^#|$^” /etc/kolla/globals.yml
+ n2 z* E" M$ @3 {- o1 [—. }4 ]: \; i% R3 h; B
kolla_base_distro: “centos”( d `0 M! d3 ]$ v- F
kolla_install_type: “source”
; r3 Y* e% h' y+ K5 [2 c7 M3 S; popenstack_release: “ussuri”) Z3 n3 c3 D' K1 T P& ^* g, M
kolla_internal_vip_address: “10.141.255.245”6 W9 b' B. |$ G- l& g \8 ?
network_interface: “eth0”" c5 c8 L2 ?' @( t# k* S& ]0 t* m/ A
neutron_external_interface: “eth1”
3 ~- v1 X: L7 {# ]enable_cinder: “yes”' T# y! e" \6 L" w+ I( {& w
enable_cinder_backend_nfs: “yes”
& R: J" n0 K7 t1 J6 ~' l: K** Please make sure that the NFS server can accept mount requests from the OpenStack nodes. Otherwise, OpenStack will fail to launch instances because it will fail to allocate storage for them.: s [6 I/ Q% f6 ]7 d
** OpenStack release “train” is used with kolla-ansible 9.1.0. If a higher version of kolla-ansible is to be used, then another release for OpenStack should be used. For example, kolla-ansible version 10.0.0 can be used with OpenStack release “ussuri”.
2 _: ^: f" b4 t# }7 c# QThe multinode template file is an ansible inventory file that has the main server groups for different OpenStack components that will be installed.
6 Q7 {# M% V8 Q/ \! L: q** DO NOT REMOVE THE CONTENT OF THE “multinode” FILE. WHAT IS SHOWN IN THE FOLLOWING SNIPPET IS JUST THE FIRST 20 LINES WHICH YOU NEED TO EDIT. THE REST WILL REMAIN THE SAME AND SHOULD NOT BE REMOVED.2 Q- z* c: m& G! Z7 W. u
# grep -vE “^#|$^” multinode | head -n 20
: N1 t& @$ L5 u, N( R+ j8 [9 g[control]
5 m+ L0 r, ? c0 D) J9 {node001node002node0039 S, k, A* e7 A; K% n# [+ E
[network]
' X% B5 B+ q3 Unode001node002node003, ]! a z7 Q+ h, l3 f
[compute]
$ b; o- u% t. S8 y+ g0 P* wnode001node002node003
; Z9 a: X) I5 o, u[monitoring]8 u: Y) X& O/ o9 l% J
node001node002node003
0 N2 T* j' s4 B[storage]
) V" Z) w# P- ?! @localhost ansible_connection=local) ^2 e& @& w# p2 E1 r9 d
[deployment]; e H* k; I- d6 `6 N
localhost ansible_connection=local
. k+ d6 T+ l1 F: x4 Y7 u# ?[baremetal:children]" s$ m- V3 o# M+ c( [6 G1 e+ ^! F
control
- g- `3 o3 [6 t# I1 I& D' pnetwork
, \' i6 u$ d' u6 m4 }, Ncompute
% ]2 @* }+ V0 u9 w3 wstorage
" J4 V$ L+ l8 m, q2 k0 Q6 Wmonitoring
2 k, {6 f' d2 B/ v1 ?Deploying Kolla-Ansible
/ X% z/ C: u/ n+ _Make sure that the nodes specified in the inventory are all reachable:8 \9 i8 @3 t) q* T' {
# ansible -i multinode all -m ping: j9 ~6 ^& ^" ~5 |( v
The template file /etc/kolla/passwords.yml is a placeholder for all the passwords that will be used during the kolla-ansible deployment:
0 ]3 I6 s, U/ J! H# grep -vE “^#|$^” /etc/kolla/passwords.yml | head
% E4 i* M+ Y& l—6 m+ V2 I+ H( ?$ B
ceph_cluster_fsid:9 n8 g! V- E, { A; i c2 }
ceph_rgw_keystone_password:
( r0 x; r+ n# crbd_secret_uuid:9 u$ p( `, G" V# g6 B; ~
cinder_rbd_secret_uuid:- R" Z, _9 A+ b" r; j* o
database_password:
" W1 l0 x) x% Gmariadb_backup_database_password:
' B: W% A$ x% J( M7 A6 R% l! ?( qdocker_registry_password:1 ]6 b9 J: G% P. W2 l8 Y$ l
opendaylight_password:
5 r" R9 z2 \, e+ m$ r: Vvmware_dvs_host_password:" Q" k" k- t0 m0 L: c$ j; Q
All passwords in passwords.yml are blank and have to be filled either manually or by running a random password generator:
2 m+ a, C& n- v2 I0 P/ N# c# kolla-genpwd
7 Y6 B) L; } @Bootstrap all nodes with Kolla deploy dependencies. This will include installing packages and writing out configuration files:6 N$ O* `$ X" d( W6 G! `% u
# kolla-ansible -i ./multinode bootstrap-servers
: L5 q2 {- s5 V2 Y- K3 IRun the prechecks for deployment for hosts. This will make sure that the packages have been installed successfully and the configuration files do exist in the expected location:
+ s4 q- \( K- ?# kolla-ansible -i ./multinode prechecks6 `4 ]# @- `3 c: p0 B
If the previous two steps did run without failures, then you can proceed to deploy OpenStack:
% E7 c. q6 }- b. X4 _ r- b9 h# kolla-ansible -i ./multinode deploy( d0 \) z" G" e
After deployment is successful, an openrc file must be generated wherein the credentials for admin users are listed. By running the following command, an admin-openrc.sh file will be generated under /etc/kolla directory on the deployment (head) node:
0 k* W0 y! s9 _! ~: e# [# kolla-ansible post-deploy* ^' T+ j7 a' L3 v- E
Now the changes which has been applied to the compute nodes have to be grabbed back to the software image of the compute nodes to avoid losing them:/ R8 u5 r7 I: q, v7 T
# cmsh
. l3 ^8 G" L: a/ I, |( x& n/ g% device use node001- K& ?+ _8 c4 M9 l& Q! A. d6 N
% grabimage -w
0 j, V8 }% r( i* z$ t5 W4 fMake sure that SELinux is still disabled and not set to permissive:2 w' X1 S1 w1 b9 X2 A' p
# grep -w SELINUX /cm/images/default-image/etc/selinux/config | grep -v “^#”, u9 M( ^6 W/ Q* @3 S5 m. B, U
SELINUX=disabled
: i) j. @2 J+ Z: [( T5 p0 K7 eInstalling OpenStack CLI client
" ^" M+ a5 V( h: ZTo start using OpenStack which has just been deployed, OpenStack CLI needs to be installed first and the openrc file needs to be sourced:6 b3 T3 H5 i j( {8 t! ]
# pip install python-openstackclient
! J) W& ^% o4 }& O) p2 p' U X# . /etc/kolla/admin-openrc.sh* t; g2 l2 n8 e' r, c; H
Initializing the OpenStack environment for the first use
3 c' L! I) u" ?$ G( w* LTo start using OpenStack to create instances and access them from the external network, the “/usr/local/share/kolla-ansible/init-runonce” has to be run. This script uses “10.0.2.0/24” as the external network. In most cases, the init-runonce has to be adjusted to match the external network:
& P- |: w' b3 f# This EXT_NET_CIDR is your public network,that you want to connect to the internet via.
, I( ~& p5 L) n8 ]ENABLE_EXT_NET=${ENABLE_EXT_NET:-1}
+ Y% A e* S/ Q/ z- |; M' |# uEXT_NET_CIDR=${EXT_NET_CIDR:-‘10.0.2.0/24’}
3 ]0 B+ v& o4 hEXT_NET_RANGE=${EXT_NET_RANGE:-‘start=10.0.2.150,end=10.0.2.199’}4 ~/ N9 V, z" q) S+ r
EXT_NET_GATEWAY=${EXT_NET_GATEWAY:-‘10.0.2.1’}3 b: S) J, {2 t1 ]5 ?% l
[…]
% B9 x2 Y$ J$ r# cif [[ $ENABLE_EXT_NET -eq 1 ]]; then+ U* v: G: H3 X) M N
openstack network create –external –provider-physical-network physnet1 \
* p" z$ y0 ` J0 |" P6 C5 M –provider-network-type flat public1
, Q" }) t5 h; D7 S/ }+ ~' f openstack subnet create –no-dhcp \
^5 U) l0 H4 u! G7 F! v" ] –allocation-pool ${EXT_NET_RANGE} –network public1 \
4 J8 k5 n; x9 f- s9 i, V –subnet-range ${EXT_NET_CIDR} –gateway ${EXT_NET_GATEWAY} public1-subnet
& x: h: F/ E1 q$ D* pfi
- y5 q6 j! W$ ~5 Dopenstack network create –provider-network-type vxlan demo-net) k8 k0 N: X! N8 ~+ J* B
openstack subnet create –subnet-range 10.0.0.0/24 –network demo-net \( c) T! N3 x- K; {
–gateway 10.0.0.1 –dns-nameserver 8.8.8.8 demo-subnet
$ a N3 R# t' f' j% G1 X& RTypically, provider networks are directly associated with a physical network but that is not a requirement. Users create tenant networks for connectivity within projects. By default, these tenant networks are fully isolated and are not shared with other projects. OpenStack Networking supports the following types of network isolation and overlay technologies:
) T2 s' C0 f3 n6 F# zFlat( I) e- z; i8 Z, l; ^1 ^
All instances reside on the same network, which can also be shared with the hosts. No VLAN tagging or other network segregation takes place.
* F) @, b& I9 I. _+ q5 x' I1 C" sVLAN8 G# J4 ]$ N5 ]) J
Networking allows users to create multiple provider or tenant networks using VLAN IDs (802.1Q tagged) that correspond to VLANs present in the physical network. This allows instances to communicate with each other across the environment. They can also communicate with dedicated servers, firewalls, load balancers, and other networking infrastructure on the same layer 2 VLAN.& k6 M: z4 S% g. N6 A, ?
GRE and VXLAN
1 K$ b+ U7 y9 ^- b$ g6 T b+ HVXLAN and GRE are encapsulation protocols that create overlay networks to activate and control communication between compute instances. A Networking router is required to allow traffic to flow outside of the GRE or VXLAN tenant network. A router is also required to connect directly-connected tenant networks with external networks, including the Internet. The router provides the ability to connect to instances directly from an external network using floating IP addresses. c8 s' h" C8 n; [: I3 B6 O
For the purpose of this document, we will create a provider network of type flat. After adjusting the init-runonce script and running it, there will be provider and tenant networks created connected by a router. Next, we will deploy the first OpenStack instance and access it from the external network:6 a$ J0 e* N1 F0 p% e, z
Deploying the first Instance9 _" X1 \* Q; w( Q# m
OpenStack dashboard can be accessed via the Virtual IP (VIP) which has been configured earlier in the process. The following steps will show how to create the first instance and be able to access it from the external network using the OpenStack CLI. Similar steps can be following from the OpenStack dashboard to achieve similar results.
! g% l4 w1 @1 n) OCreate a demo2 instance and attach it to the demo-net created by the init-runonce script:
f: y( |9 c3 i% T7 f# openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo2
* t4 P( u: b% M9 [& w. F; JAllocate a floating IP on the public (provider) network created by the init-runonce script: p( C; m/ d' G) I) C
# openstack floating ip create public1
# c# S6 @3 L6 @( {# openstack floating ip list -c "Floating IP Address" -c "Port"
G6 x* I) Y, o( i% p q9 V1 D+---------------------+--------------------------------------+
5 E7 k5 z6 V4 l4 K, Z| Floating IP Address | Port | + J* M9 N- K% G% ?5 z V* h# I
+---------------------+--------------------------------------+ ! N Y& m% a+ w; n6 m* Q9 S
| 192.168.1.183 | ab1fa4b7-3e8d-4bcc-8b48-a53ad3360fa2 |
; A$ R& Y: v) \! Z6 W| 192.168.1.158 | None | # x( f8 V9 a! M; y# Y: E
+---------------------+--------------------------------------+ " W) o* r! w+ h; ~) h* K; z
Associate the floating IP with the demo2 instance created in the first step:) n: j i& m, i( Z
# openstack server add floating ip demo2 192.168.1.158
# G5 |$ ~8 a6 xAs a sanity check, make sure that the name spaces are updated correctly on the controller/network node:
, v' u" b- N( h4 z8 z p1 {0 X# ip netns exec qrouter-21c38859-4375-4bb4-9207-c74e45e12602 ip a- E* L1 M: ?8 Z: O
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
% F& U. A c |* u link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00: H/ ^0 p. g! G, g
inet 127.0.0.1/8 scope host lo
; o) ?5 l; x0 w5 H3 z3 W3 l valid_lft forever preferred_lft forever$ I4 m9 Y1 Q- _, p
inet6 ::1/128 scope host
- U; N2 A4 k# f valid_lft forever preferred_lft forever; _3 a! K) o& s7 z# @2 E
16: qr-6f76c4b6-21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
8 j$ d+ k0 k9 u: \$ Y) F4 D; ^ link/ether fa:16:3e:8e:f1:7c brd ff:ff:ff:ff:ff:ff! p% K% D& a7 b( |6 e# M
inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-6f76c4b6-21
% l8 h2 D/ N4 }. w5 Y valid_lft forever preferred_lft forever
( Y* R+ W: A% _0 X. m3 p8 z inet6 fe80::f816:3eff:fe8e:f17c/64 scope link" q. t1 S3 c% e
valid_lft forever preferred_lft forever
5 e! A7 b: ^% r5 n$ z17: qg-3660ae4c-b8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
! o! c7 h; x. Q1 L: O link/ether fa:16:3e:51:81:b3 brd ff:ff:ff:ff:ff:ff- H$ g9 i9 _/ I$ g& D: s1 t
inet 192.168.1.189/24 brd 192.168.1.255 scope global qg-3660ae4c-b8
3 S3 h+ r4 F9 e2 V" G valid_lft forever preferred_lft forever4 n- M% z& m- _6 D" L1 d3 W7 a
inet 192.168.1.183/32 brd 192.168.1.183 scope global qg-3660ae4c-b8
5 P7 U: @+ ~! G valid_lft forever preferred_lft forever' c& L# f' m+ k$ |' t- |# N
inet 192.168.1.158/32 brd 192.168.1.158 scope global qg-3660ae4c-b8
/ j2 v0 W" I, R valid_lft forever preferred_lft forever
S, c; ^' o' s6 R9 a inet6 fe80::f816:3eff:fe51:81b3/64 scope link6 }+ N9 H& `' Q! @: [& @' G
valid_lft forever preferred_lft forever2 L: a- I( F+ U D: `
SSH into the OpenStack instance using the associated floating IP address from the external network:
, `. Z9 k* U& F4 W4 `, j2 F6 }# ssh -l cirros 192.168.1.158
& ^3 Z8 k( l3 ?& l q( TWarning: Permanently added '192.168.1.158' (ECDSA) to the list of known hosts.
; A2 P$ z0 P' b$ ip a
. w9 `4 L9 r) z5 P% J1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1. W$ c; f( g: M& A1 r( U/ g
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
. r6 w; t6 T% R5 ?! i. A7 u inet 127.0.0.1/8 scope host lo* G" [4 M. j+ I- G' B7 Q* O
valid_lft forever preferred_lft forever
3 ~2 L: z( T* a C, Z inet6 ::1/128 scope host/ [; k! r4 X: x3 m2 I
valid_lft forever preferred_lft forever3 c6 G0 j9 g( `; V' b
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
( b% n! S' p" o" v$ N link/ether fa:16:3e:08:bd:30 brd ff:ff:ff:ff:ff:ff8 v" N9 _1 H1 B2 c( f. \- D3 [
inet 10.0.0.67/24 brd 10.0.0.255 scope global eth0: K* g! N Q2 `8 q/ y \4 z6 Z
valid_lft forever preferred_lft forever# E# T0 w7 W0 g) B
inet6 fe80::f816:3eff:fe08:bd30/64 scope link1 T+ p6 s+ I; a( s, W% b; o
valid_lft forever preferred_lft forever
/ Q M5 @. z' R/ N$
0 r! V5 M$ e ]2 n- o/ O& ^*** NEW SECTION ***$ Q2 U# Q; H( y+ N# m' R
How to integrate Kolla-ansible with an external Ceph Storage
0 p& b6 v; P6 A5 o, J$ C2 m4 [. U4 X, FUsing Ceph as backend storage for different OpenStack services greatly reduces network traffic and increases performance since Ceph can clone images/volumes/ephemeral disks instead of copying them. In addition, it makes migrating between OpenStack deployments much simpler.: C2 }5 W; d4 ~2 m) X, B6 u6 y
Preparing the Ceph Storage Pools for OpenStack Services8 U4 y& \# a3 y: A( U
Glance Service5 K/ Z1 L- q/ c2 P- a, F, p* Y
By default, i.e. when not using Ceph, Glance images are stored locally on the controller nodes. When they are needed for creating a VM they are copied to the compute hosts (hypervisors). The hypervisors can then cache those images. But the images need to be copied again, every time an image is updated, or whenever a hypervisor undergoes a FULL install.! F( h- q: |0 C' ~' f+ H
If Glance is configured with Ceph as the storage backend, things work differently. In such case, Glance stores images in Ceph, not on the head node. Depending on the format of the image, they might still be downloaded from Ceph to the hypervisor node (and get cached there).
+ U4 j1 k& `7 i9 G v7 AIf both Glance and Cinder are configured to use Ceph, it’s possible to avoid copying the image from Ceph into the Hypervisors. This can be done using ‘copy-on-write’ (CoW).
! \: p" K5 K$ ]3 zWith CoW, the storage volume of a VM is thinly pre-provisioned (from the image) directly in the Ceph backend. In other words, the image does not have to be copied from glance into the hypervisor, nor from hypervisor to Cinder. & n% d( p8 i* |
If Ceph is available we recommend to use it both for Glance, and Cinder so that the two services can make use of CoW. Note that for CoW to work, image format of images stored in glance needs to be “raw”. Using CoW often results in best VM creation times, and least load on the infrastructure. The drawback of using CoW is increased disk I/O latency if the VM ends up modifying existing files on it’s CoW-created volume. This is because data needs to be deduplicated the first time a unique block is written.
; W8 q2 T2 ]. `7 sTo configure Glance with Ceph, on the Ceph cluster, run the following commands:
; G" {4 y7 Y8 C8 }& O" Y" {+ YCreate a pool for glance images:( h5 I0 G& L2 m5 w* r8 Y
# ceph osd pool create images 64" z# c. s+ H2 A( q- t n: v p) O
Create a user for glance service which has write privileges to the pool created in the previous step and store its keyring:
" U1 E6 o9 |( W5 H; m1 {4 M# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' -o /etc/ceph/ceph.client.glance.keyring
4 [* O N! w9 s5 S6 nVerify if the pool has been created properly and user has the required privileges:
* M4 W4 p+ ]( ^ U1 y# ceph osd pool ls0 t3 [5 P, s# @2 [& }
images2 f& v4 _1 F* k( L/ v( L9 d5 K
# rados lspools
: _3 [) r* d+ l7 b3 F8 S `" f" Nimages
3 Z$ S& g9 [0 w! i- ?( Q- z% K# ceph auth ls" C" n) h# S: T) U/ O, s& C: k& o
[...]
* |+ Y* z4 }5 E; b v* jclient.glance
/ r$ P( X) t9 b; S8 E key: AQDniwRfNVzlJRAAKa9QDXlMGDerCPY7J8pAGw==: o( T) J0 Q4 ?
caps: [mon] allow r
7 t/ J( Q H' z% X7 k% T6 r caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=images
8 ?' M9 V/ w+ y4 V% c: o[...]( s W, j. y# c/ @7 G
** Note: How to choose Placement Group Number for a Ceph pool:
6 h3 ~3 u9 a) w5 TWhen creating a Ceph OSD pool, it is mandatory to specify a value for placement group number (pg_num) because it cannot be calculated automatically. From the Ceph online manual, we list here some recommendations:
3 e* m+ \9 H. ZLess than 5 OSDs set pg_num to 128.
+ y2 m' a2 R! L* W; F9 u' T; SBetween 5 and 10 OSDs set pg_num to 512.
3 @6 j3 I5 L# i% IBetween 10 and 50 OSDs set pg_num to 4096.
" u Z$ j' v7 R4 d4 JIf you have more than 50 OSDs, you need to understand the tradeoffs and how to calculate the pg_num value by yourself.
9 M" G% ?# ]" D' Z" \2 u" @1 HFor calculating pg_num value by yourself please take help of pgcalc tool.
4 l' ~$ r; w! dAs the number of OSDs increases, choosing the right value for pg_num becomes more important because it has a significant influence on the behavior of the Ceph cluster as well as the durability of the data when something goes wrong (i.e. the probability that a catastrophic event leads to data loss).
+ I+ M7 A; h6 l0 u9 ]Cinder Service G7 b+ D0 T/ g
Cinder is the block storage service in OpenStack. It’s responsible for storing volumes and volume snapshots that can be attached to VMs or can be used to launch instances. In this document, we have demonstrated how to use NFS as a backend storage for Cinder service.
6 I, n. s6 Q+ gCinder can be configured to use Ceph as its backend storage instead of NFS.
, i6 m% ]+ A$ u5 n7 B3 z0 b* |: q. wOn the Ceph cluster, run the following commands:* J% @) O& z8 S& b
Create a pool for cinder volumes:+ l$ s# x& r- Y( l: w2 \( x
# ceph osd pool create volumes 32
& S6 r+ T( ^/ Z6 GCreate a user for cinder service which has write privileges to the pool created in the previous step and store its keyring:
. P, K* K: r4 h; o. o& T" M( ?# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images' -o /etc/ceph/ceph.client.cinder.keyring
( i0 n3 z" b7 [$ R0 G+ S$ V9 BVerify if the pool has been created properly and user has the required privileges:
: Z F) p F3 Z2 B- j% e( M# ceph osd pool ls$ q R1 L5 C* k, n9 S- d0 B5 F" h
images; H+ U) i) M. Z1 S
volumes+ f( f4 Y9 e- @4 x n
#ceph auth ls: [" z& H: e$ s5 h
[...]- m, f! e# c9 C
client.cinder
0 {! Q8 r K1 b0 H8 C key: AQAZeQRfpfx0DRAAigGXZf4jbas9jQNDMUcIog==( Y- Q" B5 o; I | L" M
caps: [mon] allow r& B5 Q; r. _6 _% b7 L1 M) J
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images
5 b" u" T' y4 ?. A4 N[...]7 N( S2 \* f d3 X. D
Nova Service: l4 w- @5 h* T5 D9 v
Nova is the compute service within OpenStack. By default, Nova stores ephemeral disks associated with running VMs locally on the hypervisors under /var/lib/nova/instances. There are a few drawbacks to using local storage on hypervisors, large images can cause filesystem to fill up, thus crashing compute nodes and a disk crash on hypervisor could cause loss of virtual disk and as such a VM recovery would be impossible. (if VMs are used with Cinder-manager volumes, those volumes are not stored on the hypervisors, they are stored in whatever’s configured as the Cinder backend)4 p4 B) E5 w- ?
Nova can be configured to use Ceph as its backend storage.
; g$ ?1 P" D5 i8 O: gOn the Ceph cluster, run the following commands:
! g" Q1 i; G" [( |Create a pool for nova ephemeral disks:0 I2 `; J" b+ I2 U; ?
# ceph osd pool create vms 128! g* ~3 z* _2 v
Create a user for nova service which has write privileges to the pool created in the previous step and store its keyring:
. R7 l. C* i7 C ^" E# ceph auth get-or-create client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images' -o /etc/ceph/ceph.client.nova.keyring- |* S: o M+ @9 Q' b" x
Verify if the pool has been created properly and user has the required privileges:
- S7 v; z5 l+ W6 k; B# ceph osd pool ls5 |+ M3 Y7 }" S
vms
( c; M$ x" M" a( l5 Limages2 E! z. M- w7 u; p* Z
volumes
. e1 G- k1 ^4 T) l! ]2 f[...]
8 y! D& q* ^5 T7 i* x$ L# ceph auth ls
% e2 b' B* ?' ^2 r+ W$ Z[...]) e* K3 S' d: |2 q# ~1 k- B0 P! e; l$ ]
client.nova
1 v3 T5 w/ |9 Z key: AQB6dwRfgrhaGRAAxXB/6YKcXIW+eYdoiSSMlg==! j$ C' J6 N, Y. j
caps: [mon] allow r
, W! F3 i% u- V% Y caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images3 s5 L& |4 U* G: N
[...]4 ?# R9 a$ S/ d2 z$ a/ d( j/ |
Preparing Kolla-ansible configurations to use Ceph Storage as backend storage for Glance/Cinder/Nova services
0 c5 d% s+ E2 x% X' i) p6 rThe globals.yml is the main configuration file for Kolla-Ansible. In addition to the parameters mentioned at the very beginning of this document, there are a few other parameters that are required to enable several OpenStack services to use Ceph as their backend storage:( T! C2 O* i* E6 y1 T8 r
The parameter ceph_{ service }_user is used to specify the Ceph user that will be used to write the Ceph pool for a particular service. The variable “{ service }” should be replaced with the service name like glance, cinder, or nova.
" g+ X" l3 J) B' {The parameter ceph_{ service }_pool_name is used to specify the name of the pool created in Ceph which will be used by a particular service. The variable “{ service }” should be replaced with the service name like glance, cinder, or nova.) H# [% X9 x# \) R2 k, l I# G
For Ceph users to be able to read/write Ceph pools a keyring is required. The keyring is specified by ceph_{ service }_keyring which points to the location of the keyring file for specific service. The variable “{ service }” should be replaced with the service name like glance, cinder or nova.( c9 ^: y! ^2 p/ O
The parameter { service }_backend_ceph is used to enable/disable a particular service to use Ceph as its backend storage. The variable “{ service }” should be replaced with the service name like glance, cinder, or nova.
+ T0 B5 \# J: o% Y# grep -vE "^#|$^" /etc/kolla/globals.yml
& _8 v# F7 Z r6 p! i9 b: z---
2 j3 X) D2 j" `1 v6 D, j% jkolla_base_distro: "centos"
4 }" P- z2 }, y D, t5 f dkolla_install_type: "source"
. D1 @0 S2 l v1 K2 D% {! kopenstack_release: "train"
: T% q, i7 N1 A9 A7 Pkolla_internal_vip_address: "10.141.255.245": |7 v1 D* x( v; {
network_interface: "eth0"
; b, d# k. o3 G6 Q3 { n$ lneutron_external_interface: "eth1"4 }6 I; {. m7 U3 k/ }8 M: w
enable_cinder: "yes"; j& h* N/ {, h. v
ceph_glance_keyring: ceph.client.glance.keyring
9 i3 Z8 v' Z7 Rceph_glance_user: glance& o7 D; u! A% k; i8 V7 Q
ceph_glance_pool_name: images7 e6 d3 z' M% y3 f3 e i5 w5 L
ceph_cinder_keyring: ceph.client.cinder.keyring8 R; x$ ^& v3 m+ o9 T$ B4 g. X& t) t
ceph_cinder_user: client.admin
. K0 e7 `- ~8 Y) tceph_cinder_pool_name: volumes
2 Z+ ^3 i0 r" v5 s5 Fceph_cinder_backup_keyring: ceph.client.cinder-backup.keyring. D) T) H6 b4 ^
ceph_cinder_backup_user: client.admin
$ P/ Y% [5 ?) F+ D2 R) S/ Iceph_cinder_backup_pool_name: backups
4 F/ M; V6 ^% D8 u7 J: bceph_nova_keyring: ceph.client.nova.keyring6 n6 `0 t& I6 D' {0 j
ceph_nova_user: nova7 J2 C; G( ^& U/ [1 N- c
ceph_nova_pool_name: vms9 y0 [; G7 Y* d; n, w6 \* L9 s/ r
glance_backend_ceph: "yes"+ `- a; ?& u" r0 w7 h4 X9 w* X
cinder_backend_ceph: "yes"
/ M, Z$ Y( b1 L% T- @nova_backend_ceph: "yes"
; h% e% _* Q: a: c: o1 |: T7 [** Note: the “ceph_cinder_backup_pool_name” is the same as the “ceph_cinder_pool_name”, however, it’s not strictly necessary to have both pools the same. You can create a separate pool for cinder_backup_pool.$ a% f* `+ w$ i: C2 ]6 ]; a+ Z
Glance Configuration
/ n' a* b: f* n6 g5 ?The configuration files which will tell Glance how to use Ceph should be placed in the location where Kolla-ansible would expect to find them:) h5 F: B1 x+ P( ^; W
Create a configuration directory for glance:; D5 R8 k5 ~3 J* S+ U. n
# mkdir -p /etc/kolla/config/glance
" F. e6 d% c8 l: D& \Copy ceph.conf to the same directory created in the previous step:% K' t7 h+ ], a+ [- h, y
# cp /etc/ceph/ceph.conf /etc/kolla/config/glance/) s4 n+ a: c& A0 k2 |# y
Copy the keyring to the same directory:
) Q! v0 p: S2 W1 P# cp /etc/ceph/ceph.client.glance.keyring /etc/kolla/config/glance/
) l' |* x% ]# ~" }/ B% P6 \. cCreate glance-api.conf in the same directory with the following contents:
$ t# h4 t8 e, W; k7 ~2 n: Q# cat /etc/kolla/config/glance/glance-api.conf) N) ?8 Q. V' ^% @& B
[glance_store]" C8 a3 b" t5 R
stores = rbd: a5 S) c) K E; b1 |% g1 f9 F
default_store = rbd
* x& \$ x# J6 @8 l5 T$ C0 S3 X% Trbd_store_pool = images
1 d. d. i( C. f7 X' P" w# `' g; i7 |rbd_store_user = glance
2 \$ e5 d9 ^. Yrbd_store_ceph_conf = /etc/ceph/ceph.conf
# T) j7 _% G. H! }Cinder Configuration5 n* |6 r( N5 [ B# J( K
The configuration files which will tell cinder how to use Ceph should be placed in the location where Kolla-ansible would expect to find them:0 Z4 _' k# Y! p( c" p
Create a configuration directory for cinder:. O! V- @2 y" G7 l2 H
# mkdir -p /etc/kolla/config/cinder/{cinder-volume,cinder-backup}
% h) w' {$ ~0 g& uCopy ceph.conf to the same directory created in the previous step:3 }& }( ^+ e# n: g1 |
# cp /etc/ceph/ceph.conf /etc/kolla/config/cinder/ceph.conf' P! W% j' p- A
Copy the keyring to the same directory:
. _! _3 S$ R! a7 k# cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring
) s6 @- g) y" L& `5 ?7 C# cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring9 J& G* P# s+ f1 T
# cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring- l, \# [! [; y* N; |; y r
Create cinder-volume.conf and cinder-backup.conf under /etc/kolla/config/cinder directory with the following contents:
r# g$ r& \4 Z3 j, A8 Z# cat /etc/kolla/config/cinder/cinder-volume.conf2 G; w, G; k# e( h$ [. j) g
[DEFAULT]
8 S& h. s; {& ~: W7 ^5 U1 Tenabled_backends=rbd-1/ s7 V3 C% U2 }; f+ x
[rbd-1], P3 P4 f1 x2 w2 e
rbd_ceph_conf=/etc/ceph/ceph.conf
$ t: b" S" G# \8 z6 \* W5 {; Vrbd_user=cinder5 \" O( J2 ?# L K
backend_host=rbd:volumes
& T- @6 d6 @8 z$ Wrbd_pool=volumes
, w2 Z) l9 x1 A n: \* Z/ f# gvolume_backend_name=rbd-19 F2 D. t, J$ @
volume_driver=cinder.volume.drivers.rbd.RBDDriver
H1 e& F; A, J0 `4 D7 Trbd_secret_uuid = {{ cinder_rbd_secret_uuid }}
* |: P# q5 X( y3 D5 `. v# cat /etc/kolla/config/cinder/cinder-backup.conf
, n$ F$ ?% w$ I1 y[DEFAULT]8 D. U" ?( w: F! ?
backup_ceph_conf=/etc/ceph/ceph.conf
' R2 p) A7 q( wbackup_ceph_user=cinder
8 I' T0 q' p) j( ubackup_ceph_chunk_size = 134217728% O) \# q5 J) O$ Q6 B/ O3 H
backup_ceph_pool=volumes, q/ {$ F; {$ c1 O3 {
backup_driver = cinder.backup.drivers.ceph.CephBackupDriver' E3 v( L Q, X/ l
backup_ceph_stripe_unit = 0 w" i! M1 F; Y$ }
backup_ceph_stripe_count = 0, }0 `9 f! }( V) G. H
restore_discard_excess_bytes = true
& b; u# t9 F* r+ SNote: {{ cinder_rbd_secret_uuid }} can be found in /etc/kolla/passwords.yml# `1 t( |, y" N* ~: u( x
Nova Service2 }- C* T) e; M: c$ P" D4 g5 c
The configuration files which will tell cinder how to use Ceph should be placed in the location where Kolla-ansible would expect to find them:
3 V% {/ E/ C! d' A/ @4 E0 TCreate a configuration directory for nova:
/ \% A" u% C8 n& n. e5 J/ b) G v# mkdir -p /etc/kolla/config/nova
6 w. K# e# ?: a1 `7 c( gCopy ceph.conf to the same directory created in the previous step:0 \& Z; h; I$ `( y& T. I7 ~, t
# cp /etc/ceph/ceph.conf /etc/kolla/config/nova/. R# J8 X8 F _( A0 x
Copy the nova and cinder client keyrings to the same directory:
0 [7 E7 ]9 K# [7 |3 W3 g8 h# cp /etc/ceph/ceph.client.nova.keyring /etc/kolla/config/nova/ceph.client.nova.keyring
1 p# I5 b/ | N# cp /etc/ceph/ceph.client.cinder.keyring /etc/kolla/config/nova/ceph.client.cinder.keyring
) y& P1 J6 s( y3 x2 B5 `Create nova-compute.conf in the same directory with the following contents:
" f1 O' ?0 _# t9 b+ x; D# cat /etc/kolla/config/nova/nova-compute.conf
' B0 U0 O4 c" W( A7 V[libvirt]5 N. L0 W+ ~' f; {$ w
images_rbd_pool=vms$ i6 y0 F0 G. y' W7 k5 d7 Q
images_type=rbd
. B" P/ K" [- D; ximages_rbd_ceph_conf=/etc/ceph/ceph.conf
$ ^$ [# F; d) K9 Irbd_user=nova
1 f, J3 j: @0 [2 RVerify if OpenStack is writing to Ceph Backend Storage
% h# i: V2 w8 v8 HAfter deploying kolla-ansible following the steps mentioned earlier, you can verify if OpenStack is writing data to Ceph by inspecting Ceph storage itself:- k" w9 p7 J h5 E: G
Test creating images (glance service)( R, l) X9 C& F6 O/ h
# openstack image create --disk-format qcow2 --container-format bare --file /tmp/cirros-0.4.0-x86_64-disk.img cirros
3 L! K1 K# D: x8 u5 @) d. |# rados df" W* C8 h/ ~$ Y3 G
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
5 X9 p; y5 P1 n# uimages 37 MiB 8 0 24 0 0 0 90 70 KiB 21 12 MiB 0 B 0 B/ y2 f' x, b7 m. }0 N# E
vms 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B
6 v& P0 d8 y( vvolumes 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B6 H4 q/ j9 q6 i) `# n. r
total_objects 8
0 U: G% s1 A* A2 A7 V/ x$ _+ e' Itotal_used 3.0 GiB# X3 M9 |/ Z) D& C+ A
total_avail 297 GiB
2 e0 c" ]/ {, h; t' c3 wtotal_space 300 GiB1 V2 p- z5 n6 O; }. |
# rados -p images ls
# `4 \6 o. g! e. L! trbd_data.179c773a98d0.0000000000000001
2 p( {9 U) `* L( i( G) f8 o/ krbd_object_map.179c773a98d0.0000000000000004
" K# S8 y Y4 n& }. Trbd_directory
" [: m$ r ?& X( T( E: e L5 urbd_info
4 T' M* }7 P; ]6 q1 Y+ U+ _rbd_object_map.179c773a98d00 S# a; J/ E) c E: Q' J+ _# I) i
rbd_header.179c773a98d0
F9 p: z1 U( D+ ^$ Z; R5 D0 u- srbd_id.9223ba8b-ddbd-4b69-a5ea-db1fb31ee1e0& X6 |9 e8 x ]% R0 s3 v
rbd_data.179c773a98d0.00000000000000008 {4 H3 H, [% X0 V7 l% W. Y4 V6 p
As you can see in the previous output, after creating an image, it is present in the Ceph storage./ d5 T0 l& Q" ~9 }- N0 C% y
Test creating a volume (cinder service)) [4 K6 @" q6 l- P
# openstack volume create --size 10 --availability-zone nova mynewvolume
* v3 V1 I0 p# x: N& ?' E+ J+---------------------+--------------------------------------+
* `% C" g# g- {. |8 o) {| Field | Value |, y7 e h1 S5 F' ]# I
+---------------------+--------------------------------------+. l4 N: l6 ^8 \. q( Q/ W& Z. e5 D
| attachments | [] |
8 x5 K9 x' q7 |) |7 p| availability_zone | nova |
# G7 k& M. o: w" N' o2 j, L# `- @# W1 f| bootable | false |
8 d4 J+ m$ q9 X| consistencygroup_id | None |' T. O; V& S* i4 Q m( M$ o, C/ h' b% G0 M
| created_at | 2020-07-09T15:59:17.000000 |
" E/ Y3 D5 e2 s3 q: ]! z3 f& S| description | None |) @3 w2 Q: f' i8 q E( e0 a) i: r
| encrypted | False |
3 _ ]( X/ H. F$ N" t' G6 J& D| id | 1ee10849-f948-46b0-ae33-d7bbd5733c6a |
2 g! A# b0 u, ?- ]+ I/ Q| migration_status | None |3 \( @3 l# D# v$ N& d4 e
| multiattach | False |
' Y0 D2 I: T: n0 U* O2 H| name | mynewvolume |+ R+ ?1 W8 @# B" E" u6 \
| properties | |
1 _" S( E5 f( X N9 ~% N+ t; o1 i| replication_status | None |2 f/ {( y5 C" S, T$ J1 e
| size | 10 |/ ~8 _ J+ n) l$ K/ L& ~$ a0 ~; w
| snapshot_id | None |
) o) g9 Q9 e) A/ X. \2 Q| source_volid | None |+ S; w+ T' ^4 j6 Y* c' Z; ?
| status | creating |
' }: S) P7 C# }0 u3 e/ u0 |/ B| type | __DEFAULT__ |
4 d% W: h- U) y| updated_at | None |
# r; Z6 [% ?: ~; W| user_id | be4c99e41dab462a881b55585dbcfe15 |
( f4 H0 ^6 N8 }+---------------------+--------------------------------------+/ [5 w# C8 l' K# N+ d5 j
# rados df C0 |" T" o0 ?' [- V
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR; Q: M& E. ^4 O; x1 j
images 112 MiB 20 0 60 0 0 0 267 209 KiB 59 36 MiB 0 B 0 B
( K6 {+ J$ t+ m' b, |vms 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B* v" F) V. H6 o1 b
volumes 576 KiB 5 0 15 0 0 0 36 27 KiB 6 6 KiB 0 B 0 B3 i4 b, w: T- U v* a+ K. ]3 `; q
# rados -p volumes ls/ r/ ?$ s* q$ F# {. u3 e: [
rbd_header.a554d675abc8- o, H/ N7 b0 s
rbd_directory6 g2 [" H3 _- S
rbd_object_map.a554d675abc8
0 Q! U( c& w( m. u* q, Z1 _rbd_id.volume-1ee10849-f948-46b0-ae33-d7bbd5733c6a3 j6 u: y% @# C2 I3 W
rbd_info
~: I1 _9 h1 d: E5 v# x6 P6 YAs you can see in the previous output, after creating a volume, it is present in the Ceph storage.+ i0 t ? w3 k2 r0 j$ j5 r
Test creating a VM (nova service)
+ k$ R2 U$ |+ Z% c1 o: |3 J# openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo1
, \7 B! F. b0 p$ y& N+-------------------------------------+-----------------------------------------------+
5 t' U7 @! h. \/ y8 O1 z! j( L* O6 d| Field | Value |! _4 d' V6 a3 X6 e' @4 @8 H- ?
+-------------------------------------+-----------------------------------------------+
. b* t" X% `6 B1 T| OS-DCF:diskConfig | MANUAL |" O6 x1 D! c/ ^2 Z1 v
| OS-EXT-AZ:availability_zone | |+ L. E+ U: c. l4 j, L2 \
| OS-EXT-SRV-ATTR:host | None |
7 i. g- o8 P0 z1 D4 E; B* k7 }| OS-EXT-SRV-ATTR:hypervisor_hostname | None |/ Z. q, t6 t0 v% q/ @3 H
| OS-EXT-SRV-ATTR:instance_name | | o; k( z( ]: \7 S* S. ?
| OS-EXT-STS:power_state | NOSTATE |
/ K4 G( H7 V2 Q: c| OS-EXT-STS:task_state | scheduling |- r# I) l4 G& d& G6 Z; ]: ?3 S' }& m
| OS-EXT-STS:vm_state | building |
/ Y9 b7 w4 ^3 R" B) |0 J* n1 @ R) \| OS-SRV-USG:launched_at | None |( T: n. T4 S5 F; W, L. n
| OS-SRV-USG:terminated_at | None |
! C) X& ?0 m" R4 k% r| accessIPv4 | |
' b* V. K+ E% U2 @ O4 a* Z; L| accessIPv6 | |2 Z0 N# i9 {* V! a7 W8 K- z' X
| addresses | |, e! p2 t$ k7 }% q4 u& Q- p
| adminPass | JqLnjPKb5Awj |
. k' Z Z: X7 N| config_drive | |" |+ p- ~' _9 I1 t$ s x5 ?
| created | 2020-07-09T16:07:47Z |
B/ U! B) E" X3 k* L| flavor | m1.tiny (1) |" F* Q! t: M3 K" v6 l! u( ^
| hostId | |( H0 d$ R! u' [1 q& E$ O1 W
| id | a2fdb85d-d6bf-4150-ab96-dd22b7d4e9d4 |) _! Z) D8 j( n1 u) a W
| image | cirros (2de81533-2410-4c67-baa4-5260cfb9f1a8) |
. m5 b9 O- ~* T( p* N3 x* G| key_name | mykey |
: q' I2 W3 }! f# o| name | demo1 |
! J, ~& Q' K( ]* D4 p( L# G( z# k' a3 u| progress | 0 |
; f' F/ g1 v& r" | |' `| project_id | b108a02fe6b24fffa88b7de24e05b685 |+ s$ i. g. b: y* J+ N
| properties | |2 R; W& d$ D, Q) B+ y9 S
| security_groups | name='default' |
+ n0 n {+ X4 A2 s| status | BUILD |
2 d, X/ \' M/ m, f+ @, |) i| updated | 2020-07-09T16:07:48Z |2 f3 _! x: X' z" l
| user_id | be4c99e41dab462a881b55585dbcfe15 |
7 p% `9 X+ l" c( I& B+ {0 X2 U/ S| volumes_attached | |% h9 |% J; Q3 u$ ]' L7 Y; i7 V0 ~3 F
+-------------------------------------+-----------------------------------------------+
* E. d# ^& U4 w# openstack server list
$ l& C) J+ m1 b# f9 z+--------------------------------------+-------+--------+---------------------+--------+---------+8 P# s+ X; [3 | e& n+ t
| ID | Name | Status | Networks | Image | Flavor |
. o9 ?1 ?9 X+ A; {' c. ^ n. p9 p+--------------------------------------+-------+--------+---------------------+--------+---------+
1 w! `: x9 r! A! ?$ a* r/ @/ Z| a2fdb85d-d6bf-4150-ab96-dd22b7d4e9d4 | demo1 | ACTIVE | demo-net=10.0.0.134 | cirros | m1.tiny |
: u* m& Z I: t2 X* x) V8 r+--------------------------------------+-------+--------+---------------------+--------+---------+: B* i7 l& B8 A; y) }
# rados df " Z, S1 ~0 l8 _: g) L- T. ]
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
. I9 b' C# @- w( ?& n; Himages 37 MiB 8 0 24 0 0 0 520 13 MiB 105 49 MiB 0 B 0 B
' W- q, ]+ }& e& ~1 Tvms 207 MiB 138 0 414 0 0 0 1880 19 MiB 987 58 MiB 0 B 0 B
) x3 }* q. B2 f# _5 E1 Z# Kvolumes 576 KiB 5 0 15 0 0 0 306 243 KiB 6 6 KiB 0 B 0 B 6 D: V2 j- f3 `
- Q& l' Z) ~4 {3 k7 |) l" g
total_objects 151
9 N9 _" q) U8 Z. v6 x7 x2 dtotal_used 3.3 GiB
& [1 c K$ y2 _: h% }2 x8 J& Ttotal_avail 297 GiB $ z( P; T9 c5 _, ?8 b3 H
total_space 300 GiB & ]3 \. J$ w& B+ y5 }8 b, ]8 K
rados -p vms ls | head! }" T6 M5 h0 g' `& Z- k( Q
rbd_data.a6b31261fa8f.0000000000000003" V [9 L7 E* Z- Q! W
rbd_data.a6b31261fa8f.000000000000009c' i4 J) f0 R1 V* \: J3 S% Z& _
rbd_data.a6b31261fa8f.000000000000007c
& p& w& _! c$ t+ ?5 ^) U! G" ]rbd_data.a6b31261fa8f.0000000000000028+ A% ]$ ~) F: U
rbd_data.a6b31261fa8f.00000000000000225 }& S3 B0 M& [
rbd_data.a6b31261fa8f.00000000000000c4
0 t! O6 v* w! Z0 h+ M& vrbd_header.a6b31261fa8f$ C2 q$ m8 N! q$ M
rbd_data.a6b31261fa8f.000000000000002a5 w+ U+ ?9 q7 K2 g! n9 H3 F- a
rbd_data.a6b31261fa8f.00000000000000fe4 c; _( z. j- {- Z' C
rbd_data.a6b31261fa8f.0000000000000006' B. \: g& {: _& E4 G* M4 Q
Troubleshoot
/ L. q) h9 f: ]1 t; p5 y9 ]Issue: “kolla-ansbible deploy” step failed with the following error:. }7 K; e" I2 @4 D- n3 v
TASK [nova : Running Nova API bootstrap container] *********************************************************************************************************************************************************+ ?2 @' O3 P# f4 d
fatal: [node002]: FAILED! => {"changed": false, "msg": "Unknown error message: open /var/lib/docker/tmp/GetImageBlob027787082: no space left on device"}
$ h3 C c4 V0 H- [5 C7 N4 tNO MORE HOSTS LEFT ***************************************************************************************************************************************************************************************** N& q* j* s; ~
PLAY RECAP *************************************************************************************************************************************************************************************************. G8 c W$ W2 l& t
localhost : ok=48 changed=12 unreachable=0 failed=1 skipped=17 rescued=0 ignored=00 a2 q& R# q `
node001 : ok=40 changed=7 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0
: r6 p2 Q4 ^; U# Pnode002 : ok=202 changed=99 unreachable=0 failed=1 skipped=79 rescued=0 ignored=1 ( @4 C5 c; _ R& d( [; I
node003 : ok=78 changed=10 unreachable=0 failed=0 skipped=76 rescued=0 ignored=0
* ~* u) T+ Y. c$ Z0 N+ R. EResolution:
& L! ^3 e5 o5 [: U" O% f- K3 gThe openstack node(s) (controller or hypervisors) ran out of space and no more containers can be run. A bigger disk at, least 60G, should be used for controller and hypervisor nodes.
: A0 ~" [; d* e! y a# ~4 N+ S( iIssue: kolla-ansible deploy fails after haproxy container fails to start
% }1 G, b! [* ORUNNING HANDLER [Waiting for haproxy to start] ************************************************************************************. K0 P8 \0 l/ ~/ B
fatal: [node003]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 10.141.0.3:61313"}+ t+ r8 E" l, F7 C
RUNNING HANDLER [haproxy : Waiting for virtual IP to appear] **********************************************************************# w V- R& v1 C& O
NO MORE HOSTS LEFT ****************************************************************************************************************2 I- S4 H! G7 V* b9 g
PLAY RECAP ************************************************************************************************************************
+ A0 D8 M4 N8 G$ Ulocalhost : ok=35 changed=0 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0
+ l0 q: Z; h4 a5 q% g) {6 p' r6 Ynode001 : ok=35 changed=0 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0% G3 M2 G3 h( a$ `9 h
node002 : ok=35 changed=0 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0! V0 B- D, o: U; l) g% H- v1 T
node003 : ok=71 changed=3 unreachable=0 failed=1 skipped=76 rescued=0 ignored=0" H! l0 V* }' ?, ~5 O5 Z
Command failed ansible-playbook -i ./multinode -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /usr/local/share/kolla-ansible/ansible/site.yml
( [" I7 N7 Y( b* wResolution:
- V6 h2 B. }1 \; D$ y- b1 {, ZThese sort of issues usually happen when a higher kolla-ansible version is used with an older openstack release. Always make sure that the openstack_release in globals.yml points to a release which is supported by the currently installed kolla-ansible version.
3 g/ |; L& O$ V) ]- ^+ x5 h! HExample, kolla-ansible version 9.1.0 supports OpenStack “train”, while kolla-ansible version 10.0.0 supports OpenStack “ussri”.: z% o' w& [! \, u/ |) Y/ ?, K! a
Issue: I have rebooted all my OpenStack controller nodes at the same time and now mariadb container is continuously failing.. L; E `9 u I3 n( N
Resolution:3 C E: V4 s) n' c# f2 P, Z
It is highly likely that the database cluster has become inactive and thus a database recovery should be attempted:3 S' c& ~$ o1 f/ H/ P: X/ O
# kolla-ansible -i ./multinode mariadb_recovery W; _% U( X1 Z5 v, G) l( i4 k- H
Tips and tricks' V) {* a. F: C. g& n. B7 T) f
When running the kolla-ansible CLI, additional arguments may be passed to ansible-playbook via the EXTRA_OPTS environment variable.
* V1 t" L! @$ Q) j/ pkolla-ansible -i INVENTORY deploy is used to deploy and start all Kolla containers.2 G0 i0 ~& T9 F% p# x( J2 B; `+ f
kolla-ansible -i INVENTORY destroy is used to clean up containers and volumes in the cluster.
' r; J% w q9 d) ?) y Ikolla-ansible -i INVENTORY mariadb_recovery is used to recover a completely stopped mariadb cluster.7 |8 _8 A \' r) R0 Z, ?" ^
kolla-ansible -i INVENTORY prechecks is used to check if all requirements are meet before deploy for each of the OpenStack services.) x8 B# u9 A/ I4 T6 q- ^% l. A5 N
kolla-ansible -i INVENTORY post-deploy is used to do post deploy on deploy node to get the admin openrc file., Z: I; t { u0 K/ J
kolla-ansible -i INVENTORY pull is used to pull all images for containers./ x" L8 O4 v, |# i$ {
kolla-ansible -i INVENTORY reconfigure is used to reconfigure OpenStack service.
! P# E/ {# C; V/ Tkolla-ansible -i INVENTORY upgrade is used to upgrades existing OpenStack Environment.
" G3 C! |( ^- n4 G; n! |, {kolla-ansible -i INVENTORY check is used to do post-deployment smoke tests.
" T6 C" O% _: Q; k6 t+ Ykolla-ansible -i INVENTORY stop is used to stop running containers.7 m1 m9 c. y! D" R s5 @! x
kolla-ansible -i INVENTORY deploy-containers is used to check and if necessary update containers, without generating configuration.
% n( a) P% C+ w4 j0 qkolla-ansible -i INVENTORY prune-images is used to prune orphaned Docker images on hosts.1 `+ [# P* P# }2 a
|
|
发表于 2021-5-31 11:27:23