- 积分
- 16843
在线时间 小时
最后登录1970-1-1
|

楼主 |
发表于 2021-8-2 10:51:52
|
显示全部楼层
1 Ansible管理docker
$ {9 R# |7 |8 v6 p( N近年来Linux容器技术越来越受欢迎,通过容器技术,可以保持程序运行环境的一致性,快速启动并高效率运行,涉及到的开销也比较小,此外,在系统层次上完成容器级别的资源隔离非常快速。
. L3 Z$ g- n: g8 I; x6 ]Docker是管理Linux容器最流行的工具,它为管理Linux容器提供了许多方便的工具,比如创建、销毁Linux容器,还提供了一些除管理Linux容器之外的工具,比如管理镜像、编排。通过它的易用性,Docker已经成为管理容器的最流行的方法之一。& [+ \- M0 W% ^8 N' Z/ {
题外话:关于容器和Docker; r! _# Y7 B! y0 Q! C2 \% B
Linux容器是内核的几种功能组合在一起实现的。换句话说,Linux容器技术是内核层次的功能,Docker只是提供了一系列工具,包括从底层和内核交互到高层和用户交互的一条龙。除了Docker外,也还有其它操作Linux容器的工具,只是对大众来说,Docker是最流行的。8 b$ ~# Y( H# }& c5 E/ B9 `. S
Ansible为Docker提供了一整套工具,包括相关模块、连接插件(ansible_connection: docker)和inventory脚本,因此Ansible可在许多方面与Docker进行交互。例如Ansible可构建Docker镜像、启动或停止容器、组合多个容器服务、连接到活动容器并与之交互,甚至可以从容器中获取inventory。
, Y7 N9 D/ v5 r7 h9 m8 v如下是Ansible官方目前提供的和Docker相关的模块:
* ]. `9 s; |& T" e4 l/ zCode
# {6 t# A6 H @$ R1 w) P: u3 r8 g* N! u) [7 E# ]. T7 d/ P
docker_compose – Manage multi-container Docker applications with Docker Compose H. e8 o, \/ p) w$ l% _
docker_config – Manage docker configs; O& e) m) R( F) Y) n; \1 R
docker_container – manage docker containers" q9 n6 b2 w6 U+ R) [2 h; G! ~: o
docker_container_info – Retrieves facts about docker container1 K2 v1 J x. l) P4 [( Z! w
docker_host_info – Retrieves facts about docker host and lists of objects of the services5 P0 b8 E- b/ g3 e+ {# H Y1 g# R+ m
docker_image – Manage docker images4 g; a9 d6 ^5 U1 H$ N0 M! Z
docker_image_info – Inspect docker images& @) E8 k2 {: E, k7 x9 Z% p( Z7 ?, ?
docker_login – Log into a Docker registry
. [6 M! C9 f# V C. ^docker_network – Manage Docker networks4 s5 k1 Z6 C4 q8 n `1 l
docker_network_info – Retrieves facts about docker network
( W4 S2 r C! m0 C) Q& `docker_node – Manage Docker Swarm node; H7 |( L* ^* t- u& {
docker_node_info – Retrieves facts about docker swarm node from Swarm Manager/ u5 b! i; x6 q
docker_prune – Allows to prune various docker objects) } {. m/ q# y" D \( z d+ q
docker_secret – Manage docker secrets& q) l* u, x+ P1 t
docker_stack – docker stack module
5 d5 j! x x5 D8 t" pdocker_swarm – Manage Swarm cluster
! j: C4 `( D. i: C" odocker_swarm_info – Retrieves facts about Docker Swarm cluster
) h9 z8 J+ V9 d! \5 wdocker_swarm_service – docker swarm service( [/ y( a; ?' i0 a
docker_swarm_service_info – Retrieves information about docker services from a Swarm Manager
+ Y: }: b9 p! _ }- Adocker_volume – Manage Docker volumes
5 f1 ~$ `" t( \8 ldocker_volume_info – Retrieve facts about Docker volumes
5 v( J' P( m s9 g; g要使用Ansible管理连接Docker,要求安装如下包(注意:Ansible端和docker端都安装,这一点和其它模块不一样,如报错,请自行在两端安装、卸载、升级调试):
8 [& N6 `* Q$ Y1 GShell
3 L5 v: O; j1 J8 `) ]0 m0 Y- I9 [2 D0 E& k3 S& S$ }3 B: {5 L: K
# 两端都安装,如果已经安装了,则在报错的情况下按需更新
8 O4 w/ q' u& x# 此外,根据Ansible使用的python解释器版本,按需决定使用pip还是pip3,+ l( S) e# f' J# F `
# 如果需要的是pip,则yum install python-pip
3 o" N/ h0 T* t$ pip3 install docker requests5 Z2 Y4 p9 m6 }+ {6 ~4 C8 [# Y( ]; V
如下是其中两次报错信息,注意其中的结尾:No module named ‘XXX’。) x% w/ n: m! R/ _1 x. Q
Code
M* s; ]; J V% X$ h& J) w/ J1 @- ]/ |
fatal: [192.168.8.65]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.75)) on controller's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.75). The error was: No module named 'requests'"}0 D7 F( H* }- z- M& d
fatal: [192.168.8.65]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.75)) on controller's Python /usr/bin/python3. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` or `pip install docker-py` (Python 2.75). The error was: No module named 'docker'"}
2 o0 a7 c L( U! r0 C5 d2 Ansible构建并运行Docker镜像
6 W+ h) J3 _; Q4 U. Q% Q" \通Ansible提供的docker_image模块可管理Docker镜像(比如构建、移除、pull镜像),使用docker_container模块可管理容器,比如将镜像运行起来成为容器。8 o! M" v+ `6 V1 G; I: H1 u
对我们而言,一般都是在已有镜像的基础上通过Dockerfile来定义新的操作,然后构建出自己的Docker镜像。所以需要提供两个文件:一个基础镜像和一个Dockerfile文件(基础镜像不存在时会自动下载)。如果使用Ansible来构建镜像,那么这个Dockerfile文件需要能够被Ansible读取,比如可以放在Ansible playbook文件的同目录下。
+ }( c, `/ U, a0 V4 g2 f& x% W为了演示以下Ansible构建Docker镜像,此处已经写好了一个非常简单的Dockerfile,该Docker镜像是在CentOS 7镜像的基础上添加nginx,然后让nginx运行起来并提供cowsay页面。. s+ C0 b9 ^ `
Dockerfile内容如下:
3 _: Z& e1 y: i) [ t6 uDockerfile
8 P' J" m) g. S3 d
' K% M- K) }. BFROM centos:centos7
8 E5 L& |$ W5 K9 D% H4 G8 YLABEL maintainer="test.com"
8 ^8 r: ~+ ~2 C- Z0 I3 qRUN rm -rf /etc/yum.repos.d/*.repo && \
3 c; o; I) {. [/ N0 ? echo -e ' \
1 V9 T6 q; l2 G3 E8 a0 o0 I& U) M4 G[base] \n\
L( c$ K! Q) [# T" m Gname=os \n\* y; b" w' G( I0 C/ @3 F
baseurl=https://mirrors.163.com/centos/$releasever/os/$basearch/ \n\9 |; K! j$ ^+ c) q+ m
enable=1 \n\
0 w) ?4 ` x6 e8 a3 p2 R$ E5 M, E1 hgpgcheck=0 \n\( k" \# V& {8 w* h6 t
[epel] \n\2 ] {# a( k* B* T# s C; w' p( x
name=epel \n\& U0 ~ s( i# _7 ^ O
baseurl=https://mirrors.163.com/epel/7Server/$basearch/ \n\
; S* p# i' [, k/ }$ Denable=1 \n\& L; \4 R6 l- K: a
gpgcheck=0 \n\7 d. i% b4 J }# {" Z
' >/etc/yum.repos.d/base.repo && \% A7 [/ H* v/ g5 J) ^
yum -y install cowsay nginx && \+ B7 _7 P! I3 i# C) s
rm -rf /usr/share/nginx/html/index.html && \" t+ M$ |. q; |8 i
cowsay test >/usr/share/nginx/html/index.html && \7 H5 I9 e6 l) j+ M
echo 'daemon off;' >>/etc/nginx/nginx.conf && \
; u8 z6 |% |4 n+ ^ yum clean all w8 I3 \' c A- _
i! Y( i2 n2 q- U) S( {8 T( \EXPOSE 801 u( O% v! D2 p6 ]7 e8 q/ m! O
CMD /usr/sbin/nginx
9 E# |3 V/ J6 s E. E/ x2 r然后写一个Ansible任务文件,假设名为build_and_run_image.yaml,内容如下:
- b" n8 C8 L" IYaml
" L- H% M; Q3 n2 v6 i$ Y2 h+ L; \3 z7 D7 S0 f2 T/ ], u
- hosts: docker
" _0 l o6 {; G% h- |4 V gather_facts: no
# I( b0 i- A/ H tasks: 9 x) ~1 f9 S2 |5 K! @$ W+ B6 O
- name: scp Dockerfile
5 ^+ {2 ?2 f. l& x+ Z' A& Y( ~ copy: 5 `4 ?$ j( q) P/ A* `
src: Dockerfile
0 k; u6 L" u' q4 ]5 ]8 b1 M5 b dest: /tmp/Dockerfile
2 p5 K/ E2 Y! Z1 L$ j; ~ - name: build Docker image centos_nginx:v0.1.18 f/ e& q' i j1 U- X
docker_image:
% l6 w( |& t1 N7 N) f2 w name: centos_nginx
# {! F+ E1 P6 y5 l8 s2 `6 i6 ? A7 n source: build6 H" ] c' h( n9 f6 F4 G
tag: v0.1.10 h6 E Y Y" h" e9 c; h! @
build:& {! a+ k) ]. b. o5 Z( i# b) k
path: /tmp2 v1 i1 r+ h) K4 \9 N$ p
pull: yes
2 Y. K- q, [. d8 b. H - name: start centos_nginx# a) k0 F$ q% ^& d
docker_container: ; {" B! e6 X( ^3 z
name: cng* o- X" P P' A) A, V8 S! c
image: centos_nginx:v0.1.1" q. G& Z/ |3 E1 e0 D- |
ports: 8080:80# ~ k1 Y1 r: J3 i( }+ @5 P, ^
state: started6 ~/ H; J4 l3 L. V4 H+ P E5 m
Ansible执行完成上述任务后,可直接在Ansible端使用curl来测试页面是否可获取:
* d, {5 I5 }* U/ z7 rShell; p& Z, L5 \0 J! T
, z# Z4 l( @: m5 I6 @+ ?
$ curl 192.168.8.65:8080. g$ G( d, T! M5 o
______________
8 j* L/ u5 \6 {, _: W) f< Junmajinlong >
( ^# c& _( c# V% h: i --------------
y, |& R h3 R; h9 o \ ^__^. c. h# K S" j3 s. I
\ (oo)\_______+ z9 E6 ~5 f- o% l# Y' V& Y7 [! b
(__)\ )\/\4 o8 [, n# v5 p; _
||----w |0 Q7 K- `5 m9 z Z, b5 x
|| ||
9 w) F ~4 X7 B3 Q这里对上述两个模块稍作解释。& }& @; V6 m- I/ l# t# s
对于docker_image任务来说,通过sourec: build指令来表示这是一次镜像构建操作,source指令可包含如下值:; \6 L2 B& S' X( ~
build:根据build指令的path参数指定的Dockerfile构建镜像% a* ^" k# G% P5 D# y. U* V
load:从镜像tar文件中提取镜像# n7 D' O) I, a* h5 f4 y
pull: 从registry中拉取镜像1 v& w6 @; Y+ s6 P
local:保证docker端已经具备指定的镜像% _1 v9 V- n! `8 m" Z
对于build指令来说:, A9 {* g3 M& A) Z1 b: |
path:指定构建时的上下文目录,该目录要求包含Dockerfile文件/ F1 w* p$ S# V+ P
dockerfile:明确指定使用哪个Dockerfile文件,而不是默认的上下文目录中的Dockerfile文件4 P" g- ~4 Q% q. r. J
pull:构建时是否从registry中拉取基础和中间镜像镜像- @* n9 O9 S% P: x, W
cache_from:构建时使用缓存的中间镜像) R& O. d' [; q$ H& }
args:可按照key:value方式指定镜像的参数,对应于Dockerfile中的ARG指令,例如listen_port: 80802 e I0 q' f% ~$ T
其它指令应该都通俗易懂。. u# W4 m8 X% O' g r9 d
构建镜像完成后,可使用docker_container模块启动该镜像。该模块指令非常非常多,几乎包含了docker container命令的所有选项功能,但是熟悉docker命令,这些指令的用法也通俗易懂。比如,指定卷和环境变量:. f& y# s8 h6 a
Yaml! a1 `( `2 u% ^5 S; n. x
$ x& u d A& N2 @
& Y- ]- `9 b+ j; W; E& o3 a8 {+ k+ |
- name: Create a data container. c' }" X. K n) [+ Y* n1 _
docker_container:7 k! C' @; ?2 ^) j6 [( B5 D- Y) O
name: mydata' T+ A; y3 u0 [. c% I1 g
image: busybox% \+ C H% e& b) r( s$ t7 Y
volumes:
6 z( u4 k( o s, i B$ Q* m4 W6 p) A - /data
# N9 O4 X% @+ s( ]* L- name: Restart a container
% s, i% h; O# t- i6 ] docker_container:
O. X( e( \' a1 C9 `( v* K name: myapplication
; ~" H. \3 E: \5 }8 L image: someuser/appimage, P) U$ P0 J4 x" q6 s' v
state: started# g9 n7 Z- W3 K- f! q7 [
restart: yes
- F y, H" w; i* r, _ devices:
# N4 M' B+ f1 Q) Z4 p0 O - "/dev/sda:/dev/xvda:rwm"9 n+ @2 K* i4 `& l) }4 h
ports:
6 _5 S2 n Q7 |, v - "8080:9000"/ O) C4 v* L% m8 w
- "127.0.0.1:8081:9001/udp". G9 V8 b& ?3 Z J6 w8 W
env:
3 k5 K, }1 s) \$ i+ u; s SECRET_KEY: "ssssh"& d* M7 x# N& g4 `. [4 V
BOOLEAN_KEY: "yes": S' Y$ U) N* R% j5 a& O
3 无Dockerfile启动镜像并连接容器
7 r1 \6 l+ |* x: F6 z在上面的playbook中明确使用了Dockerfile来构建镜像并启动镜像提供服务,但因为Dockerfile自身也是基于基础镜像构建的,所以可以省略这个构建过程,而是直接启动基础镜像并连接到启动的容器进行操作。; w( l. w# B. v
下面实现与上述示例相同的效果,只是不使用Dockerfile构建。7 o6 U# t, N1 y8 _0 s3 v
playbook文件内容如下:
* w- ^5 j$ }1 O/ dYaml- \$ J) u0 p% w( Y. ~% O. {
5 R1 W2 |7 k2 w0 l2 b3 k--- J2 I& r) v% `6 }* j: n
- name: start image * V0 E/ V: Q# i9 S; J7 d
hosts: docker0 `' g h; S0 g4 s
gather_facts: no) }9 f7 J# q7 l( z6 ~
vars:
: G( o( P% M4 x& O6 g _5 ~ container_name: "centos7"/ @( \5 |! S" G3 [0 w# c% ~4 i# g
tasks:+ k' B9 P& s' C* E+ f
- name: start basic container "centos7"& D3 T4 D. ?2 `/ R$ D, n. O Y9 f7 z
docker_container: ! ~6 u: D: r0 f5 m; v
name: "{{container_name}}"7 `" g ?. E) m5 x
hostname: "{{container_name}}"1 X: e4 h# N( b O
image: centos:centos7- [. c$ n+ ^( }% n, L& d: e4 F
ports: 8080:80
0 P0 k& }) R7 q- p7 L9 f state: started d2 ?6 \( n. q
auto_remove: yes/ ?* ]; ]/ W/ a. D% A. L$ ]# p
command: bash
! a0 H, u8 a. G tty: yes4 ~- q L/ t, h5 h
0 _% i e9 K$ I
- name: add container to inventory
2 M# y! k6 Y5 P9 P/ X5 k* T add_host:
& Q* j) C) k; i$ R' \ name: "{{container_name}}"
" r) ~, q. \9 T; H4 g5 P! J ansible_connection: docker" D( I& Y Y, H/ p
ansible_host: "{{container_name}}"" e0 X3 @7 W6 K$ K/ o8 u6 I
ansible_user: root
/ a7 R& P8 |3 I& a; V ~: Q/ D- w groups: containers
. D0 @/ y! T! X; K6 j+ o5 o0 ?5 F7 j / I- Q3 Y- ?: S% m9 _
- name: do something in container
: s7 e' L, T& S hosts: containers7 S* F; ^# q+ \5 v8 W" L h7 u% @
gather_facts: no
4 q9 t5 Y0 r" A" `" J' d4 z tasks: D& q5 q3 g Z6 b
- name: install python if needed
$ j1 o! I. D5 B7 J7 N raw: yum install -y python$ O2 V, {! Y( _" m$ H: H- E4 ]3 u
- name: remove all repos exists
$ j7 e! ~6 U' \) i b shell: rm -rf /etc/yum.repos.d/*/ d$ N: H8 w8 y' S& {
2 C* h: J% D# j( @2 Q+ M - name: add os repo and epel repo
" W3 y/ s- \) J" A yum_repository: 4 T n) W V+ I6 O* g" i
name: "{{item.name}}"1 j% j9 u. _; T
description: "{{item.name}} repo"
6 Y( k9 v6 L$ H6 n% T4 q baseurl: "{{item.baseurl}}") k/ d* y# Y& m4 L7 [2 A; w3 t# K
file: "{{item.name}}"/ X+ I) `' _: `/ c. G! s
enabled: 1
8 u0 O' b( \$ G. F gpgcheck: 0( [" g# W+ O& {, I4 E* c6 F X
reposdir: /etc/yum.repos.d' @, F, a- s1 M/ [
loop:' ?5 Y4 L6 \2 A, [* b2 R, Q& d1 ^
- name: os/ B6 r: @# f& {9 ~5 o5 |8 r
baseurl: "https://mirrors.163.com/centos/$releasever/os/$basearch"# _2 X1 w! M2 e1 s
- name: epel
' [! v) A5 i4 F1 z4 ^6 V: S1 O baseurl: "https://mirrors.163.com/epel/$releasever/$basearch" 4 |/ Q; S* d w4 b( i8 s% m
- name: install nginx and cowsay * |! ^3 r. F4 G0 @0 G2 }0 C
shell: yum -y install nginx cowsay
% W: L( K: m& F; _0 D3 s) {( | $ g H; m( }6 \* V7 }
- name: configure nginx
( y5 J8 S9 W7 M# k lineinfile:
7 I. Z# ~: H* ?) L line: "daemon off;"# h: z: K! o% v0 `) k4 \( ^
dest: /etc/nginx/nginx.conf$ j/ i3 J6 {8 i4 f1 q
' j: {% D0 J5 m; V/ I
- name: change index page
* O4 f: o/ T- x0 i) r( L/ l' G8 | o block: . b7 j6 Q0 F2 \
# 先移除index.html,因为它可能是一个软链接" u5 z. u( [7 m- [: k: ^6 k! M
- name: remove old index.html page
9 f8 }- Z' n9 [6 v7 p shell: |( v& q+ X8 i) x0 ^& E6 L( u" ?1 E
rm -rf /usr/share/nginx/html/index.html. d2 o! Y- p: b3 p
cowsay test >/usr/share/nginx/html/index.html2 I- Q$ W. H4 U2 p1 E! A
- name: run nginx' G) x9 X, Z9 x, S) W
shell: nginx &2 @2 ^1 _) i9 v+ [. m
在执行上述playbook之前,需要先在Ansible端导出连接方式docker所在主机的环境变量,比如Ansible连接到远程docker时,使用tcp:
1 `8 u2 E+ w8 z/ KShell7 k" n* q4 |7 z) z+ f
7 m! X5 ~' b/ e; N4 C8 @+ ?! u1 Z; \- V& S- F0 m; ~; s1 a9 D
$ export DOCKER_HOST=tcp://192.168.8.65:2376
! ^; ~+ d9 g2 m, t! a) g$ ansible-playbook -i inventoryname playbookname.yaml% J8 O2 B! U( U: A# k
上面的playbook任务中,首先启动docker容器,然后使用了connection: docker连接器添加该容器到containers主机组中,以便后续的连接。之后在第二个play中连接到containers组,安装Python(CentOS系统都会带有python),配置yum源,安装Nginx和cowsay,并启动nginx。任务流程比较简单。" Y, e1 Q. d5 r l
唯一需要关注的是add_hosts添加容器节点到inventory时指定的docker连接方式,这里不能使用默认的ssh连接方式,因为目标容器不一定开启了ssh服务,也不一定能和外界通信,而使用connection: docker连接方式,Ansible将会先ssh连接到docker服务所在主机,然后通过docker container exec的方式连接到容器内部。- N/ A. R( M c9 }7 M7 b
docker inventory
' k C, S* g# S' |, ?Ansible为Docker提供了动态inventory的脚本。可下载该脚本:, f6 K9 Y9 I) s. u* u% a
Shell
2 G/ S9 _! Y; | n! T' P
9 j0 K* _- _* J1 C
" }" v* N8 b0 b7 u7 z0 H" Q0 w4 Hwget https://raw.githubusercontent.com/ansible/ansible/stable-2.9/contrib/inventory/docker.py: G0 k( d! ?$ P7 q2 ` L
chmod +x docker.py( J. T- E) e8 p0 D
执行该脚本测试:( w( g8 \( q3 x9 ^" o Z
Shell5 B# J- c, M* V( ^$ T, [
3 a% _4 X( P- Y4 I
DOCKER_HOST=tcp://192.168.8.65:2376 ./docker.py --pretty
, i7 K2 @3 f+ V1 n% B4 ~3 t( S/ {或者直接在ansible命令或ansible-playbook命令中使用-i选项指定:& ?3 i4 U S$ w
Shell
6 X! T7 D3 M9 X& k- J" _5 r% |
_, v: H* Z: I) r, C! {( Oansible-playbook -i docker.py docker_containers.yml3 g+ A. ? d5 Y' t% k2 W: }
4 其它Ansible容器管理工具: [/ T7 Q' [+ v( T
Ansible除了在官方提供了docker相关的模块外,还有一些第三方的工具可用来管理容器。8 c8 t! a4 D9 d v N
比如ansible-container、ansible-bender、Ansible Operator,它们需要单独安装,对于ansible-container来说,在之前几年比较知名,但是作者现在已经将该项目废弃,据作者本人所说,ansible-bender和Ansible Operator更好。' F# y9 _( f9 J
ansible-bender:6 G/ P9 ]4 K9 u% J% q9 e
简化Ansible Playbook构建容器(注:此容器是符合OCI标准的容器,docker所构建的底层容器也是OCI容器)
* D. _* [% F7 w3 `$ z; O, `4 n地址:https://github.com/ansible-community/ansible-bender
: N; T! h) p" N# H# ^Ansible Operator:
$ z' g2 z1 a Y是Red Hat Ansible Automation和Red Hat OpenShift团队联合开发的用来将容器部署到K8s上的工具
6 p+ c! e' H, U2 e6 `& b! n地址:https://learn.openshift.com/ansibleop
9 @! p2 w( _$ z5 Ansible管理OpenStack& _0 f0 z0 _1 {# I1 I
OpenStack可整合一台或多台物理计算机的资源来按需创建、管理、配置、删除虚拟机(在OpenStack中,虚拟机对应的术语是”计算实例”,但后文都以虚拟机来描述),对OpenStack提供者来说,提高了硬件资源的利用率;对受益用户来说,可按自己的需求申请带有各种性能、各种资源配置的操作系统,比如公有云的模式,就像去网吧上网一样,想上多久、想体验什么配置的主机都按需付费来享用。7 W# p! j# C) _( p1 e5 ?
对于OpenStack来说,Ansible几乎是全程参与其发展的,因为从OpenStack很早的版本开始,就已经逐步支持通过Ansible来配置管理OpenStack,而Ansible管理OpenStack相关的模块也随着OpenStack的版本迭代在不断更新。目前为止,Ansible官方提供的关于OpenStack的模块已经有五十多个,下面是Ansible官方目前提供的模块列表信息简介:# s" E# I1 Y2 \4 { K+ g: S
Code
/ P6 X! s% t4 _9 {# R2 v
9 A& @8 J' \6 O/ U, Q. x, i6 Sos_auth – Retrieve an auth token) z6 M2 b* B4 I" U |5 d
os_client_config – Get OpenStack Client config
2 w; |% A$ W' K" z% e% _os_coe_cluster – Add/Remove COE cluster from OpenStack Cloud$ [* f4 |: f+ D% z
os_coe_cluster_template – Add/Remove COE cluster template from OpenStack Cloud+ I; j- n9 r8 W5 K' Q
os_flavor_info – Retrieve information about one or more flavors5 D6 q2 _" \, o9 B) z/ {
os_floating_ip – Add/Remove floating IP from an instance
4 g* u$ V( D) z3 Nos_group – Manage OpenStack Identity Groups0 |! ^& m$ Y- j
os_group_info – Retrieve info about one or more OpenStack groups6 C: m, D5 S4 U/ d/ P4 z
os_image – Add/Delete images from OpenStack Cloud
2 H% ]2 L( A1 Z1 v1 \- Q8 A* D- H/ Q) yos_image_info – Retrieve information about an image within OpenStack# i3 |1 @+ |5 t9 K Z" S* ~
os_ironic – Create/Delete Bare Metal Resources from OpenStack! g1 b1 v+ B4 Z, }
os_ironic_inspect – Explicitly triggers baremetal node introspection in ironic' G# T' p1 x7 A* r
os_ironic_node – Activate/Deactivate Bare Metal Resources from OpenStack
- Z/ d/ q z& e1 Bos_keypair – Add/Delete a keypair from OpenStack
& y' k, b9 ~' y% T3 t, `os_keystone_domain – Manage OpenStack Identity Domains
/ f% g. x) c; P% \/ x( \; Wos_keystone_domain_info – Retrieve information about one or more OpenStack domains+ ^$ `( ?( J" G2 [( p, A& s
os_keystone_endpoint – Manage OpenStack Identity service endpoints
9 S7 U- L/ m3 B/ ^1 Q2 F- w0 [: X% gos_keystone_role – Manage OpenStack Identity Roles/ F' M% h8 V% ]7 {' E& h
os_keystone_service – Manage OpenStack Identity services
7 d" q! p2 H0 e9 h; eos_listener – Add/Delete a listener for a load balancer from OpenStack Cloud( P" N7 I6 x4 {
os_loadbalancer – Add/Delete load balancer from OpenStack Cloud6 ^% o0 D3 U* }; {
os_member – Add/Delete a member for a pool in load balancer from OpenStack Cloud
: Q' [! x2 B) D @' Y/ v# tos_network – Creates/removes networks from OpenStack
/ s" A$ R( M f7 Oos_networks_info – Retrieve information about one or more OpenStack networks
9 d: `# T& \- A7 {, k! {5 `os_nova_flavor – Manage OpenStack compute flavors' g$ W7 B; E+ w5 y9 W# w {' X+ Y
os_nova_host_aggregate – Manage OpenStack host aggregates5 x2 V6 y; J, }* ?5 t
os_object – Create or Delete objects and containers from OpenStack) E8 s W6 I7 ~; |. r5 \
os_pool – Add/Delete a pool in the load balancing service from OpenStack Cloud
+ L) z/ D6 v& {os_port – Add/Update/Delete ports from an OpenStack cloud G& T# i. O, \( \. [: J/ O# _
os_port_info – Retrieve information about ports within OpenStack- T5 [9 {$ y% o, l4 I
os_project – Manage OpenStack Projects8 P [# m: a" C8 h& R. R
os_project_access – Manage OpenStack compute flavors access
% `: l5 h4 j7 f( @os_project_info – Retrieve information about one or more OpenStack projects
9 I" f% I7 m, g; I& S9 J5 \/ a9 ios_quota – Manage OpenStack Quotas
5 C* U8 T" W% l) @* x' xos_recordset – Manage OpenStack DNS recordsets/ J6 ^* r5 W1 i; @0 k; {
os_router – Create or delete routers from OpenStack
v9 u. p9 `3 h3 B& O% M% q; Tos_security_group – Add/Delete security groups from an OpenStack cloud: R% S1 b. ?7 m% t- @' O( E% y* b+ h
os_security_group_rule – Add/Delete rule from an existing security group
% {4 j! S& D6 \2 n' ^$ pos_server – Create/Delete Compute Instances from OpenStack9 v, _; f5 a8 q( C0 ]
os_server_action – Perform actions on Compute Instances from OpenStack) `( U$ {1 E3 W
os_server_group – Manage OpenStack server groups0 Q" x5 E2 y$ m9 b$ [$ V/ ~
os_server_info – Retrieve information about one or more compute instances
: }( F+ I# f4 W) Oos_server_metadata – Add/Update/Delete Metadata in Compute Instances from OpenStack w6 H9 ]" r: e b* K8 z
os_server_volume – Attach/Detach Volumes from OpenStack VM’s: i/ S4 q1 J% a, E
os_stack – Add/Remove Heat Stack
0 p7 `2 u# j' B: m1 P% }; r: g2 @os_subnet – Add/Remove subnet to an OpenStack network
% F. A7 [ W: D, o3 kos_subnets_info – Retrieve information about one or more OpenStack subnets: b/ \1 E2 Z- o* e
os_user – Manage OpenStack Identity Users" P5 V. R2 }' l) m' }0 a: f
os_user_group – Associate OpenStack Identity users and groups3 Z/ V2 W6 A. |+ e2 y6 h
os_user_info – Retrieve information about one or more OpenStack users+ o5 n. [' `. C# E7 h W% A1 D
os_user_role – Associate OpenStack Identity users and roles$ H2 n5 v# J# t% ~
os_volume – Create/Delete Cinder Volumes
: _+ a5 U5 b; B9 @/ C' q7 U7 X' [( _os_volume_snapshot – Create/Delete Cinder Volume Snapshots e. Z3 t4 b4 Z m* W5 q+ I
os_zone – Manage OpenStack DNS zones
/ ?1 f, W2 O) Y' _; v2 j4 o) D6 d# l虽然看上去很多,但大致可总结为Ansible可对以下资源做管理:6 G& Q4 ~" o9 a
(1).计算资源
1 N3 R; F. {2 ]- j' N1 b! y, S; e(2).镜像管理
( O' M( F' ~" a7 ]7 a(3).账户管理和账户认证
! `" k& G5 c9 u" D/ U1 X(4).网络管理
) s7 m8 e; S' n4 u" R0 r(5).对象存储管理0 \% D2 ~+ {' K: S! Y( E
(6).块存储管理
1 Z1 ]3 w# E7 Q% x0 c4 x对每种资源的管理可分为四类操作:0 r* s( P- I2 T/ _: Z
(1).获取管理目标的信息1 t8 |* K# f5 ~* }2 ]# x
(2).添加管理目标
; A( a' p8 F/ R( ](3).修改管理目标的属性2 q9 G) V6 m* y, I% P5 A W* L8 G6 Q
(4).删除管理目标
' o ]5 [ x' N; Z# w% P即增、删、改、查。) R" M# w) ?9 K9 f
此外,由于OpenStack自身已经跟踪了其创建的每个虚拟机的信息,所以Ansible还可以直接从OpenStack中获取这些虚拟机的信息,比如从OpenStack取得某些虚拟机信息来构建动态inventory,这样就免去了手动提供虚拟机inventory的麻烦。
1 r! C% n. q, n6 n+ N& T本文不会介绍Ansible如何操作OpenStack自身(比如添加网络、上传镜像等),这和管理普通服务做的一些基本操作没任何区别,不同的仅仅只是做不同操作而已。本文会介绍Ansible管理OpenStack虚拟机时最常见的两种场景需求:
6 {+ B7 Q0 D" h. ?/ `, _0 P(1).使用Ansible创建虚拟机,然后像平时管理远程主机一样管理这些虚拟机,最后删除这些虚拟机
9 o0 s! z; y8 _ B7 x6 [(2).从OpenStack生成动态inventory2 k- f. L8 ~5 y/ X6 q$ a
14.2.1 创建虚拟机8 d3 \. w' \% o) i4 w" l# X: U0 `
OpenStack管理虚拟机相关的模块都以os_server开头,目前包括如下6个模块:本文大概只会用到os_server模块
6 U( D2 _0 R) n! S( bCode
. i7 B3 S. c. I9 I0 K8 `' A) K5 d$ d
os_server – 创建或删除虚拟机6 P+ H+ E- e$ f' F
os_server_action – 对虚拟机做一些操作,比如虚拟机的关机、开机、重启、暂停、恢复等操作2 {1 ~8 T) V/ y
os_server_group – 管理OpenStack虚拟机分组,比如测试环境的虚拟机可属于test组,生成环境的虚拟机可属于prod组
- B4 j- F' b0 H' p0 X( }' Uos_server_info – 检索一或多个虚拟机信息,在Ansible2.9之前,该模块名称为os_server_facts,用于检索虚拟机facts信息
8 G V; e- d" h! B* v2 z5 ros_server_metadata – 增、删、改虚拟机的元数据信息,比如设置虚拟机的主机名、虚拟机设备信息,如网卡配置、磁盘路径/dev/sda
4 b3 }* j: E% pos_server_volume – 附加、剥离虚拟机的卷" P( q" A1 O. P4 T
这6个模块都要求先安装好版本高于0.12的openstacksdk包,在CentOS 7中只需执行如下命令即可:$ u) c, z2 G; [8 H2 h1 _
Shell
2 r# H$ x% Z, S) ^ k" ?( K! d
$ pip3 install openstacksdk' t7 x: C. y0 e8 N1 Q1 l- v. v
为了让Ansible连接到Controller进行管理,需要添加Controller的inventory信息。假如OpenStack的Controller的IP地址为192.168.8.65,可inventory文件openstack中添加如下内容:, g) A7 z* \0 O1 R
Undefined
# t" ?1 s! v$ j9 h- y
4 X: l) U1 d& m5 f) e7 S. F[openstack_controller]
, c5 T& N" c/ C7 ?9 Z) t192.168.8.65
0 [0 A ?3 m2 T' Q配置Ansible段和controller的ssh认证互信可自行配置,此处不赘述。
) V& [- `: S2 |9 Q/ r) A, ^4 O然后就可以编写playbook来创建虚拟机,假如playbook文件名为create_vm.yml,其内容如下:& r% l3 Y/ ?3 l: t
Yaml) \5 C5 H2 x; H! ]5 [
3 _: Z5 \; ~" `4 w- name: create vm7 y9 G5 r; o. E
hosts: openstack_controller4 D) G: j7 t1 \! s
gather_facts: no
6 p# w5 Z, q8 h2 u. H% W9 p tasks:
6 b; o' d3 ]+ w; |, }2 [; }- B7 n - name: Create a new instance: }! | Z" d) m
os_server:
' h/ \) {. W6 m2 S" F% m. k( ~ state: present
1 P Q; t8 x" I3 b2 n S/ B& N auth:: \, l" H- k9 G/ n
auth_url: http://192.168.8.65:5000/v3
. x! w* B+ y9 } M3 T4 ] username: admin
5 |6 I* Q& T* f. o' } password: admin123" @" J( k4 _0 E6 Z8 P0 ^
project_name: admin
1 _) f1 A2 K- Q; \& d! s project_domain_name: "Default"8 g' Y& @ S& H7 T' {4 R
user_domain_name: "Default". a* I& ?) H* x/ `4 X% T: g" t
name: vm1
0 V" Y9 w2 B, Q# c, @ image: "CentOS-7.9-x86_64"
1 ~, T; @6 @# o8 K key_name: ansible_key
2 J$ ?/ Z" \' M# U timeout: 200
* C, {% H8 O* N4 H flavor: m1.small
' s T" p% Z2 C( ^ network: 'ext_net'/ U% Q5 x" n& s9 R
wait: yes# O ~; v9 z. F/ ]7 o+ N
meta:
+ b+ G- O4 j+ D# ^ hostname: test1
% f; W# v# I4 M- t V' o group: test_group } Y4 d x, r( ~; k; _7 P X
userdata: |/ r1 q5 w; m- W' y |4 u0 b r! T
{%- raw -%}#!/bin/bash
1 J) {8 Q5 W$ S2 \; `& @/ \, \+ I2 V cp /home/centos/.ssh/authorized_keys /root/.ssh/
' d6 J! k& W* [9 k' z: g6 x {% endraw %} j$ d# E1 a4 V, R
auth部分是认证相关信息,name1表示创建一个名为vm1的虚拟机,image、key_name、falvor、network都是OpenStack中已经配置好的。这里还同时设置了虚拟机创建出来后的主机名为test1,并加入到了test_group主机组。
! ^1 k. L; L& m" g+ e( h) @' A M因为是CentOS镜像系统,该虚拟机创建出来后,默认登录用户名为”centos”,而且默认不支持root登录,为了后续可以使用root登录,上面使用userdata定义了该虚拟机创建后自定义的操作,即将保存的公钥信息拷贝到/root/.ssh目录下。( X0 s: C3 G$ C" h( O
注意os_server创建虚拟机任务中的一项wait: yes,它是默认选项,表示Ansible会等待虚拟机创建完成才会继续执行下面的任务。/ e% d$ N1 K0 ]/ ^" x8 {1 A. Y1 N
因为所有的模块在连接OpenStack时都需要进行身份认证,为了简化playbook中的认证内容,将上面的auth选项段落的内容保存到OpenStack Controller的~/.config/openstack/clouds.yaml文件中。例如:7 b* G) f) c4 S4 y' A7 a
Yaml
) f7 l: y4 }, T+ k/ H6 G. ^/ P( ~- s& J6 T
clouds:
7 B7 ?* t7 ~# {1 @' [5 c8 J# m$ M* k3 y mycloud:
" K5 ~6 h6 ^2 X0 X: i auth:* Y) b! |! y& q# p8 ^( k
auth_url: http://192.168.8.65:5000/v33 ~5 Q- ]& K8 S4 `" m- u
username: admin) U+ P2 D* z# \& b
password: admin123
6 X5 \# U# b1 I8 Z: A( N O! Z) W; Q project_name: admin6 e# ^/ e1 w0 ?9 z: n5 z
project_domain_name: "Default"
u& i9 ^; Q8 ~2 n$ t user_domain_name: "Default"
* T5 x' k# i. s以后在模块选项中就可以省略auth,而使用一个cloud: CLOUDNAME即可:' @/ _ j4 }: p5 i
Yaml. y f- E" `- _: @
7 T. i( z1 j( E* w, u" N J, u
tasks: - Z% q) Z4 O* U4 x( ]( \2 i
- name: Create a new instance7 F. v0 ?) B7 V$ W7 r
os_server:8 t" d0 ~2 ~" W
clouds: mycloud4 c# i& H- b# s/ _( H" t n
state: present7 g) M0 J# Z4 v, U7 n0 J/ _. }
name: vm1
5 g6 n' e4 D/ g) [. @) K7 S8 s( v# F image: "CentOS-7-x86_64"+ a# ~& G0 R. W# e# {" ?8 F# q
key_name: ansible_key
" J# a- s. _- z# } u timeout: 200
5 |% W1 Q( _* n; ~' s% S2 B& Z flavor: m1.small& K% m' a9 ]+ K, q2 t: g# D( B
network: 'ext_net'8 g9 M7 h( R5 I0 e
wait: yes
! s( D* W D, X/ q9 s meta:
" Q8 L: ~0 o( A9 V2 S2 ? hostname: test1% f+ {' K% K8 ] h2 i7 [* A+ }2 T
group: test_group
6 ^" D+ f* t+ a8 D. n但是要注意,将认证信息以明文方式写入文件是不安全的,可以使用Ansible的Valut加密。不过OpenStack的dashboard中也已经提供了一个环境配置文件,可以先按照如图所示的位置下载:# k! F1 S7 K. |
) l& I4 U' a( z. T$ g
然后以source的方式执行下载到的admin-openrc.sh脚本文件:
! Y8 d7 |' P5 x! EShell
( g9 R" O- F9 P' \6 X7 }, q
3 j s" u' ]) g# e1 b! h/ dsource admin-openrc.sh) k$ Z' B$ I' M( V! M
执行完后,Ansible的OpenStack相关模块执行时,auth和cloud指令都可以省略。0 L, q) O# S6 A! u& D
上面只是创建一个虚拟机实例,但很多时候可能需要一次性创建多个虚拟机。可以将每个虚拟机相关信息定义到一个变量文件中,然后去遍历想要创建的虚拟机实例。例如:
3 B3 Z! l) ]3 ` nCode
+ R2 U) n) p2 p' {1 H. z9 ~* |' s+ i; d* H+ _% L+ w. r _; ^
---' F. d& g; V' [1 G) y; a
servers:8 q! z: g1 g& Q- w8 g
- name: vm1" ]: N5 @5 _+ Q8 Y/ W* r
image: ) {* M& s: I/ o/ w9 t
flavor:
4 {. X2 N' A0 U* ~5 |' X( P key:
* h5 U A0 t @% t; x0 N! Q nics: / X6 S4 T( R N0 S# @4 v
meta: ' n, t- a# _" s6 W
hostname:
$ f- {6 l' l2 `5 g group: 1 C# D3 I2 I3 m5 K7 F/ ]0 z
- name: vm2
V" ^5 r; O$ x V image: - K7 ?! q! \/ Z4 X0 _5 D
flavor:
4 F6 f( S& ~; H$ B# X5 V: A0 O key:
* r6 n2 J. x. a! N+ Z nics:
# a* i1 u6 L. x3 X' d/ E' i. w% K- M meta:
7 a* N% U* H( j( a* o hostname: - Z y) y O/ W* u8 L, f
group:0 o( D' M2 M, T `
有了前面的Ansible基础后,此处批量创建虚拟机应该毫无难度。8 V2 h' ^$ _+ Z1 L* X' Y
创建虚拟机后,可以将os_server的任务注册到一个变量,从而可以获取该虚拟机的信息,包括该虚拟机的IP地址:
' z. R- N9 { XYml
5 a$ o3 K. h- b: Q2 z
. ]- p/ y" P0 v3 A0 \' vtasks: $ Z* p" @" `/ [
- name: Create a new instance" |' q+ z- e! p/ k7 z
os_server:; {7 |: N* _/ D9 Z
cloud: mycloud3 e' P! Q- E+ ?8 T
state: present
% H6 ^7 o0 [4 ^* b) n+ | name: vm1
+ ]5 Z4 n6 J( s& N3 l image: "CentOS-7.9-x86_64"
% B5 L2 T. y% {' O% o key_name: ansible_key
+ @5 p" W {7 u' a8 G( d' f) ` timeout: 2006 C3 }3 M, x7 O$ O, Y0 C& r
flavor: m1.small
) H# n! w& n6 k network: 'ext_net'
4 c2 g3 Z) b6 R9 R# B. j6 h wait: yes
" X- m8 a/ M- p/ b8 g: h meta:! ^* K3 q' A1 Z6 B. Y
hostname: test1$ r! F2 j5 b% v5 t$ [
group: test_group; a' u8 O) u. B5 x9 r7 R. j
register: newserver
$ i6 @: Z+ n" f
" n2 @4 [: L3 L0 f; ]7 V. t1 z$ { - name: get instance ip, L) b% g- w2 P: ~5 I
debug:
2 z% S/ H- K b5 A' \' M var: newserver.openstack.accessIPv4; s& u9 S' i% q: i
有了IP地址,对Ansible来说就获得了最关键的信息,因为只要将新虚拟机添加到Ansible inventory中,新虚拟机便像普通节点一样可接受Ansible的控制。
0 p% U+ I4 F4 ~. e$ d# Y' L6、将新虚拟机动态添加到inventory
& k: m. F( @5 E& N获取到IP地址后,可以将该节点通过add_host模块动态加入到inventory中:- }! m9 V& ?! Q- J9 E7 L8 \
Yaml$ K& B$ E( U5 ^( m! B4 t1 D) g9 }
& T9 v% o' q z# k6 J- name: add new vm to inventory! J$ z! Z- v- X. |1 i
add_host:1 g- X* ]6 i3 o* F0 t# e" p
name: "{{ newserver.openstack.accessIPv4 }}"5 o. Y& p- X+ r% ]# _6 D
ansible_host: "{{ newserver.openstack.accessIPv4 }}"
3 C6 L& O6 {0 ?, ^7 m. ^% Y( ^+ v ansible_user: "centos"; q& E" T+ q" s9 x+ u. Z
ansible_port: 22
+ i3 C! X$ P6 j3 f groups:
# f, d7 w+ F5 T) I - vm_hosts* L( Q" ]4 x* S) D
似乎这里的逻辑不太良好?如果虚拟机启动了但是不可连接呢?对于OpenStack创建的虚拟机来说,完全不用担心,因为os_server创建虚拟机成功后会等待该虚拟机可连接才真正返回。但对于非OpenStack的其它云主机实例则不一定,这时应当使用wait或wait_for_connection模块定义一个等待任务。尽管OpenStack中可以省略该步骤,但在脑海中应当要知道有这个步骤。
: p9 n( Z% J: D H此外,OpenStack安装的镜像系统可能是比较精简的系统,甚至没有安装Python,所以为了能管理这些虚拟主机,应先使用raw模块安装Python。4 s$ p0 B7 @7 \. f
Yaml: L6 |! u1 z( q# f! g& A
- name: for new vm host
; o# n( S& L' H hosts: vm_hosts% L+ w5 k! Z) Z, ] J
gather_facts: no
" G! K5 c+ ]) w" `5 l# J9 p, K tasks:
' I- x0 H9 K8 z$ Y# u* V - name: install python if needed& ?9 Q% n6 i; M G
raw: "sudo yum install -y python"
' O1 z/ {; b6 G/ g V如此,Ansible便可以像管理普通主机一样管理OpenStack虚拟机。
8 t3 J# ^0 t: [* k7、收集OpenStack虚拟机的动态inventory
4 h' O* j; `' s% d, e& S, w; U/ K动态inventory一般需要写脚本(几乎是Python脚本)来收集,但即使不会Python也不用担心,因为对于OpenStack来说,官方已经提供了openstack插件,还提供了openstack_inventory.py脚本,该脚本位于Ansible官方github仓库的contrib/inventory目录下,查看文件时记得先选择对应Ansible版本的分支。* a: B, n+ u/ w: [. j! V
下载openstack_inventory.py并设置可执行权限:3 e$ S4 g! G* x |, [9 G& o+ @
Shell
! n2 u7 f7 d" x" F9 h" j8 q% i+ S$ `! v% b7 F
wget https://raw.githubusercontent.com/ansible/ansible/stable-2.9/contrib/inventory/openstack_inventory.py8 H5 F( L+ v9 i! p
chmod +x openstack_inventory.py
4 s) s; } A4 {9 M4 l8 [+ {然后source以下admin-openrc.sh脚本,再执行:
' d" E3 f' [: L7 `8 b) n. k CShell
: W9 u* Q2 S4 R: y3 }. S9 ~
9 Z4 J- E( S' D6 v/ }source admin-openrc.sh$ d) U) Y' {6 T+ r' l I
./openstack_inventory.py --list
+ Q! ~& r- l( u( p可查看inventory信息。
4 e8 J6 L5 L! [! e! G2 I: B1 m9 Z之后要将该脚本在ansible或ansible-playbook中使用,使用-i选项指定即可:/ i. P, z( t) }9 F/ k0 Y1 I6 `
Shell
2 _. r% G' C+ j) t8 P- O- a1 v H! }3 _! P! m- A6 I1 h
ansible-playbook -i openstack_inventory.py -m ping
2 s1 h2 q8 K: \8 a6 Q5 q7 l除了使用openstack_inventory.py脚本动态生成inventory外,还可以使用名为”openstack”的inventory插件。要使用openstack inventory插件,首先要在/etc/ansible.cfg中的[inventory]段的enable_plugins中开启script功能:0 g- v% f- ~2 y6 n
Shell
8 s/ L7 n* {8 b# q9 @# m. q! A4 ~# w" p5 O& S
$ grep 'enable_plugins' /etc/ansible/ansible.cfg 0 K5 f* |; r9 M q/ t
#enable_plugins = host_list, virtualbox, yaml, constructed+ k0 X* a6 ^: B" V. e7 _, S
取消上面的注释,加上script:
2 A7 g* l- c* `$ FIni+ V% `7 Z. v7 u( k8 O
; I( k" F0 D$ K/ ?+ |5 u
[inventory]
- Z) ^9 W, | _, Jenable_plugins = host_list, script, ini, virtualbox, yaml, constructed/ f1 |2 @* w2 P' q6 M
以后只需在yml文件中加上如下plugin指令即可:- E6 q _8 Z5 b8 r1 U3 u1 E( A
Code( j: d7 |2 |- G5 M* C* T& Q
9 n/ Y* M" p0 M" K
plugin: openstack
9 u4 B$ R6 Q0 p: A3 J# ^0 R6 N& |% O8 i: K6 j
|
|