config drive on openstack subnet 没有开 DHCP

admin · 发表于 2021-11-23 11:14:39

上一节最后问了大家一个问题：如果 subnet 没有开 DHCP，会是怎样一个情况？
在其他条件不变的情况下，cloud-init 依然会完成那 3 个步骤，也就是说网卡还是会被配置成 dhcp 模式，只是最后网卡没办法获得 IP 而已。
不开 DHCP 也是一个常见的场景，为了让 instance 的网卡在这种情况下也能够被正确配置，我们需要借助 config drive。

在计算节点 /etc/nova/nova.conf 中需要添加一个配置，然后重启 nova-compute 服务

[DEFAULT]

flat_injected = True

flat_injected 的作用是让 config drive 能够在 instance 启动时将网络配置信息动态注入到操作系统中。

当前网络的 DHCP 已经关闭。

instance 部署时指定使用 config drive

Neutron 为 instance 分配的 IP 为 18.18.18.5。

instance 启动后登录系统，ip a 验证 IP 已经成功配置，说明 config drive 起作用了

重要的是弄懂 18.18.18.5 这个 IP 是如何配置上去的。打开 /var/log/cloud-init.log，分析如下

① 扫描出 instance 中的所有网卡，这一步与不使用 config drive 的情况完全一样。

② 获取该网卡的配置信息。日志显示配置信息是从 ds 获取。ds 是 datasource 的缩写，在这里指的就是 config drive。
在不使用 config drive 的情况下采用的是 fallback 配置。
网卡配置信息记录在 config drive openstack/latest/network_data.json 文件里，内容如下：

③ 将配置信息写入 /etc/network/interfaces.d/50-cloud-init.cfg，内容为：

可以看到 IP 以 static 方式配置。

总结

1. 在没有使用 config drive 的情况下，cloud-init 只会配置第一块网卡，且设置为 dhcp 模式，所以：

① 如果 instance 只有一块网卡，且启用了 DHCP，网卡能够被正常拉起。

② 如果 instance 有多块网卡，第一块会尝试以 dhcp 方式拉起，其他网卡不作处理。

2. 使用 config drive 的情况下，无论是否启用 DHCP，所有网卡都能被正确配置且成功拉起（如果 dhcp 网卡 >= 2，CentOS 还是有问题，可能跟目前所用的 cloud-init 版本较低有关）。

3. 如果可能，尽量使用 config drive。

讨论完网络配置这个最重要的主题，下一节我们来看看 cloud-init 的其他典型应用：设置 hostanme，设置用户初始密码，安装软件等。

admin · 发表于 2021-11-23 11:19:07

环境：

openstack版本	pike
控制节点主机	openstack-controller（ubuntu 16.04.5） 172.27.34.373 L4 q( N: M4 X- ?' K$ u
计算节点主机	openstack-computer（ubuntu 16.04.5） 172.27.34.38
vxlan1006 _8 n5 v, V: H P0 R+ u$ s	centos7-init（172.27.100.2）、ubuntu1604（172.27.100.20）
centos7镜像 $ P/ j* M) X+ C' ~/ {- Z	CentOS-7-x86_64-GenericCloud-1809.qcow2c
ubuntu16.04镜像. C/ u3 Q3 i( J! @' _; C/ a	xenial-server-cloudimg-amd64-disk1.img
cloud-init 4 C' x% U1 M) \7 H; n7 e	cloud-init 0.7.9

ubuntu安装详见：Ubuntu16.04.5以lvm方式安装全记录

openstack安装详见：OpenStack实践(一):Ubuntu16.04下DevStack方式搭建p版OpenStack

浮动ip搭建详见：OpenStack实践(九):Open vSwitch方式实现floating IP

cloud-init简介

cloud-init是linux的一个工具,当系统启动时,cloud-init可从nova metadata服务或者config drive中获取metadata,完成包括但不限于下面的定制化工作：
1.设置 default locale
2.设置 hostname
3.添加 ssh keys到 .ssh/authorized_keys
4.设置用户密码
5.配置网络

为了实现instance定制工作,cloud-init会按5个阶段执行任务：
1.Generator (cloud-config.target)
2.Local (cloud-init-local.service)
3.Network (cloud-init.service)
4.Config (cloud-config.service)
5.Final (cloud-final.service)

如下服务：

[root@ltestwithoutdhcp ~]#  systemctl  list-units|grep cloud-
  cloud-config.service                                                       loaded active exited Apply the settings specified in cloud-config
  cloud-final.service                                                       loaded active exited Execute cloud user/final scripts
  cloud-init-local.service                                                 loaded active exited Initial cloud-init job (pre-networking)
  cloud-init.service                                                       loaded active exited Initial cloud-init job (metadata service crawler)
  cloud-config.target                                                       loaded active active Cloud-config availability
  cloud-init.target                                                          loaded active active Cloud-init target

各阶段作用

Generator：读取配置文件cloud.cfg；

Local：定位“本地”数据源和配置网络；

Network：读取cloud_init_modules模块的指定配置；

Config：读取cloud_config_modules模块的指定配置

Final ：分别读取cloud_final_modules模块的指定配置

cloud_init_modules:
- disk_setup
- migrator
- bootcmd
- write-files
- growpart
- resizefs
- set_hostname
# - update_hostname
- update_etc_hosts
- rsyslog
- users-groups
- ssh

cloud_config_modules:
- mounts
- locale
- set-passwords
- rh_subscription
- yum-add-repo
- package-update-upgrade-install
- timezone
- puppet
- chef
- salt-minion
- mcollective
- disable-ec2-metadata
- runcmd

cloud_final_modules:
- rightscale_userdata
- scripts-per-once
- scripts-per-boot
- scripts-per-instance
- scripts-user
- ssh-authkey-fingerprints
- keys-to-console
- phone-home
- final-message
- power-state-change

cloud image
ubuntu镜像：http://cloud-images.ubuntu.com/
centos7镜像：http://cloud.centos.org/centos/7/images/
这些镜像已经预装cloud-init

config drive

当无dhcp服务时，可以通过config drive获得metadata

配置config driver

[root@compute01 nova-compute]# view /etc/kolla/nova-compute/nova.conf
[DEFAULT]

flat_injected = True

该配置是为了关闭DHCP服务时实例网卡也能被正确配置，重启计算服务后配置生效。

stack@openstack-controller:~$ sudo systemctl restart openstack-nova-compute.service

关闭dhcp

为确保实例通过config driver获取的metadata，这里关闭dhcp服务

企业微信截图_163763823247 ...

启动实例

root@openstack-controller:~# nova boot --flavor m1.small --image centos7 --availability-zone nova:openstack-controller --nic net-name=vxlan100 --key-name centos7 --security-groups centos7 --user-data /tmp/centos.config --config-drive true centos7-init

通过--config-drive true启用config-driver,通过--user-data /tmp/centos.config加载配置,ubuntu的配置文件为/tmp/ubuntu.config

打印的日志，传入的user_data

绑定浮动IP

user-data：centos.config

#cloud-config
chpasswd:
list: |
      root:rootroot
      centos:centos
expire: false
ssh_pwauth: yes
hostname: loong576
yum_repos:
epel-163:
      baseurl: http://mirrors.163.com/centos/$releasever/os/$basearch/
      name: Centos-7
      enabled: true
resolv_conf:
nameservers: ['218.104.1xx.1xx', '8.8.8.8']
searchdomains:
      - localdomain
domain: localdomain
options:
      rotate: true
      timeout: 1
manage_resolv_conf: true

packages:
- vim
- wget
- httpd
timezone: 'Asia/Shanghai'
runcmd:
- [ sed, -i, "s/^ *SELINUX=enforcing/SELINUX=disabled/g", /etc/selinux/config ]
- [ mkdir, /dropme ]
- [ touch, /root/abc.txt ]
- [ sed, -i, "s/^ *nameserver.*/nameserver 114.114.1xx.1xx/g", /etc/resolv.conf ]
- [ rpm, --import, /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 ]
power_state:
delay: now
mode: reboot
message: reboot now
timeout: 30
condition: true

验证是否生效

验证定制的配置文件centos.config是否生效

16376386143172.png

实例可以直接root登陆（默认只能通过创建密钥对方式登陆）成功获取到ip，配置了yum源、时区，runcmd运行正常，关闭了selinux（power-state-change正常），产生了测试文件，修改了[url=]dns[/url]信息

user-data：ubuntu.config

root@openstack-controller:~# more /tmp/ubuntu.config
#cloud-config
chpasswd:
list: |
      root:rootroot
      ubuntu:ubuntu
expire: false
ssh_pwauth: yes

hostname: ubuntu1604
apt:
primary:
      - arches: [default]
      uri: "http://mirrors.aliyun.com/ubuntu/"
      search:
         - "http://mirrors.aliyun.com/ubuntu/"

resolv_conf:
nameservers: ['218.104.1xx.1xx', '8.8.8.8']
searchdomains:
      - localdomain
domain: localdomain
options:
      rotate: true
      timeout: 1
manage_resolv_conf: true
packages:
- apache2
timezone: 'Asia/Shanghai'
runcmd:
- [ mkdir, /dropme ]
- [ sed, -i, "$a nameserver 114.114.1xx.xxx", /etc/resolv.conf ]

验证是否生效

验证定制的配置文件ubuntu.config是否生效

16376386143172.png

实例可以直接ubuntu登陆（默认只能通过创建密钥对方式登陆）成功获取到ip，配置了hostname、apt源、时区，runcmd运行正常，产生了测试文件，修改了dns信息

查看config drive

sr0就是config driver，挂载并查看

[root@centos7-init ~]# lsblk

[root@centos7-init ~]# mount /dev/sr0 /mnt
[root@centos7-init ~]# cd /mnt/openstack/latest/ && ll

[root@centos7-init latest]# more meta_data.json
[root@centos7-init latest]# more user_data

meta_data.json中存放了public_keys,hostname等信息，user_data对应centos.config。

cloud-init.log日志分析

[root@centos7-init ~]# view /var/log/cloud-init.log

第一阶段服务

https://cache.yisu.com/upload/information/20200309/33/67807.jpg

第二阶段服务

https://cache.yisu.com/upload/information/20200309/33/67808.jpg

第三阶段服务

https://cache.yisu.com/upload/information/20200309/33/67809.jpg

https://cache.yisu.com/upload/information/20200309/33/67810.jpg

在第三阶段init-network服务会依次读取cloud_init_modules模块中以下配置：check-cache、consume-user-data、consume-vendor-data、config-migrator、config-bootcmd、config-write-files、config-growpart、config-resizefs、config-set_hostname、config-update_etc_hosts、config-rsyslog、config-users-groups、config-ssh

第四阶段服务

https://cache.yisu.com/upload/information/20200309/33/67811.jpg

在第四阶段modules-config服务会依次读取cloud_config_modules模块中以下配置：check-cache、consume-user-data、consume-vendor-data、config-migrator、config-bootcmd、config-write-files、config-growpart、config-resizefs、config-set_hostname、config-update_etc_hosts、config-disable-ec2-metadata、config-runcmd

第五阶段服务

https://cache.yisu.com/upload/information/20200309/33/67812.jpg

在第五阶段modules-final服务会依次读取cloud_final_modules模块中以下配置：config-rightscale_userdata、config-scripts-per-boot、config-scripts-per-instance、config-scripts-user、config-ssh-authkey-fingerprints、config-keys-to-console、config-phone-home、config-final-message、config-power-state-chang

https://cache.yisu.com/upload/information/20200309/33/67813.jpg

cloud init的modules里面提供了丰富的定制信息，详情可以参考Cloud-Init官网：https://cloudinit.readthedocs.io/en/latest/index.html#

cloud-init调试

各服务单独调试

init-local:cloud-init init --localcloud_init_modules:cloud-init initcloud_config_modules:cloud-init modules --mode=configcloud_final_modules:cloud-init modules --mode=final

[root@centos7-init ~]# cloud-init init --local

https://cache.yisu.com/upload/information/20200309/33/67814.jpg

https://cache.yisu.com/upload/information/20200309/33/67815.jpg

调试某模块的某个配置

[root@centos7-init ~]# cloud-init single --name timezone

https://cache.yisu.com/upload/information/20200309/33/67816.jpg

调试cloud_config_modules模块的timezone配置

实践总结：

1.各模块的各配置项依次读取生效

在定制实例时，会依次读取配置文件cloud.cfg配置项。

实验环境的主机需配DNS才能访问外网，通过cloud_config_modules模块的runcmd配置项配置DNS，由于配置项package-update-upgrade-install在runcmd之前，cloud-init会先安装软件，这时外网是不通的，所以报错。

https://cache.yisu.com/upload/information/20200309/33/67817.jpg

2.centos的hostname设置不生效

尝试了很多参数组合，都没有到hostname，原因待查

尝试过的参数

hostname: loong576manage_etc_hosts: truepreserve_hostname: truefqdn: loong576

ubuntu没有这个问题，只需设置hostname即可，重启系统也任然生效。

3.DNS配置不生效

配置项resolv_conf不生效。centos和ubuntu都不生效，centos貌似是个bug，参考：https://bugzilla.redhat.com/show_bug.cgi?id=1489270，ubuntu建议将dns信息写入网卡/etc文件/network/interfaces。

4.runcmd执行命令是应该是双引号

这个有点坑，在用sed命令执行关闭selinux和配置DNS时，正常的单引号''需替换为"",否则执行报错。

https://cache.yisu.com/upload/information/20200309/33/67818.jpg

5.某个配置项不合法则整个配置不生效

runmcd有个配置非法

https://cache.yisu.com/upload/information/20200309/33/67819.jpg

报错日志

https://cache.yisu.com/upload/information/20200309/33/67820.jpg

此时整个centos.config配置文件不生效

admin · 发表于 2021-11-23 12:56:32

At the end of the previous section, I asked you a question: What would happen if DHCP was not enabled on the subnet?
If other conditions remain unchanged, cloud-init will still complete the 3 steps, which means that the network card will still be configured in dhcp mode, but the network card will not be able to obtain an IP in the end.
It is also a common scenario not to enable DHCP. In order to allow the instance's network card to be correctly configured in this case, we need to use config drive. Let's start to practice.
You need to add a configuration in the compute node/etc/nova/nova.conf, and then restart the nova-compute service.
[DEFAULT]
flat_injected = True
The function of flat_injected is to allow config drive to dynamically inject network configuration information into the operating system when the instance starts.
The DHCP of the current network is closed.

The config drive is specified when the instance is deployed.

The IP assigned by Neutron for the instance is
18.18.18.5
.
Log in to the system after the instance is started,
ip a
Verify that the IP has been successfully configured, indicating that the config drive is working.
The important thing is to understand
18.18.18.5
How is this IP configured? Open/var/log/cloud-init.log and analyze as follows:
① Scan out all the network cards in the instance. This step is exactly the same as the case without config drive.
② Obtain the configuration information of the network card. The log shows that the configuration information is from
ds
Obtain. ds is the abbreviation of datasource, here it refers to config drive. In the case of not using config drive, the
fallback
Configuration. The network card configuration information is recorded in the config drive openstack/latest/network_data.json file, and the content is as follows:
③ Write the configuration information into/etc/network/interfaces.d/50-cloud-init.cfg, the content is:

You can see the IP to
static
Mode configuration.
Conclusion:
1. Without using config drive, cloud-init will only configure the first network card and set it to dhcp mode, so:
① If the instance has only one network card and DHCP is enabled, the network card can be pulled up normally.
② If the instance has multiple network cards, the first one will try to be pulled up in dhcp mode, and other network cards will not be processed.
2. In the case of using config drive, no matter whether DHCP is enabled, all network cards can be correctly configured and successfully pulled up (if dhcp network card>= 2, CentOS still has problems, which may be related to the lower version of cloud-init currently used ).
3. If possible, try to use config drive. After discussing the most important topic of network configuration, in the next section we will look at other typical applications of cloud-init: setting hostanme, setting user initial password, installing software, etc.

admin · 发表于 2021-11-23 16:47:01

配置nova.conf文件：
      flat_injected = True  #使能network信息注入功能
      injected_network_template = ********  #配置产生网络metadata数据的模板位置。
      默认在nova安装目录下，存在interfaces.template模板;
      nova/virt/interfaces.template

      实验时，copy上述文件到/etc/nova目录下。设置injected_network_template为：
      injected_network_template = /etc/nova/interfaces.template
启动虚机时，nova-compute服务会按照interfaces.template的格式，生成content/0000文件。

admin · 发表于 2021-11-23 16:59:47

cd /usr/share/nova/
cat interfaces.template
DEVICE="{{ name }}"
NM_CONTROLLED="no"
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=static
IPADDR={{ address }}
NETMASK={{ netmask }}
BROADCAST={{ broadcast }}
GATEWAY={{ gateway }}
DNS1={{ dns }}

#if $use_ipv6
IPV6INIT=yes
IPV6ADDR={{ address_v6 }}
#end if

admin · 发表于 2021-11-23 17:03:03

接下来修改/etc/nova/nova.conf配置文件。由于要修改的太多我直接附上我的配置文件。并加上注释
[DEFAULT]
#debug=True
#基本设置
logdir = /var/log/nova
state_path = /var/lib/nova
lock_path = /var/lib/nova/tmp
verbose=True
libvirt_type = kvm
#volumes_dir = /etc/nova/volumes
#iscsi_helper = tgtadm
#connect to mysql
#设置数据库连接可以访问数据库
sql_connection = mysql://novaUser:novaPass@192.168.1.102:3306/nova
#rpc_backend = nova.openstack.common.rpc.impl_qpid
rootwrap_config = /etc/nova/rootwrap.conf
#my_ip=192.168.137.104
#instances_path=/state/partition1/openstack/instance
#Image Service
#用来管理镜像服务器
glance_api_servers=192.168.1.105:9292
image_server=nova.image.glance.GlanceImageService

#RabbitMQ
#通讯节点
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler
rabbit_host=192.168.1.103
rabbit_port=5672
rabbit_userid=guest
rabbit_password=myrabbitmqpassword
api_paste_config=/etc/nova/api-paste.ini
#NETWORK
#网络节点
dhcpbridge = /usr/bin/nova-dhcpbridge
dhcpbridge_flagfile = /etc/nova/nova.conf
force_dhcp_release = True
#libvirt_inject_partition = -1
#injected_network_template = /usr/share/nova/interfaces.template
#libvirt_nonblocking = True
#防火墙组件
firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver
#网络管理组件
network_manager = nova.network.manager.FlatDHCPManager
#网段设置
fixed_range=192.168.100.0/24
#计算节点的网桥
flat_network_bridge = br100
public_interface=eth3
#计算节点的内部网络接口
flat_interface=eth2
#网段启始ip
flat_network_dhcp_start=192.168.100.2
#
floating_range=60.12.206.32/27
network_size = 1
multi_host = true
#enabled_apis=metadata

#VNC
#设置vnc服务
novnc_enabled=true
novncproxy_base_url=http://192.168.1.106:6080/vnc_auto.html
novncproxy_port = 6080
vncserver_proxyclient_address = 192.168.137.106
vncserver_listen = 0.0.0.0
#AUTH
auth_strategy = keystone
#Must Have
#这个我试过用来认证如果没有将没法认证Keystone服务也无法启动
[keystone_authtoken]
admin_tenant_name = serviceTenant
admin_user = nova
admin_password = Nova_Pass
auth_host = 192.168.1.104
auth_port = 35357
auth_protocol = http
signing_dir = /tmp/keystone-signing-nova

#Compute
compute_driver = libvirt.LibvirtDriver
接下来进行数据库同步
nova-manage db sync
确保nova用户对/etc/nova /var/lib/nova有完全的权限
chown -R nova:nova /etc/nova
chown -R nova:nova /var/lib/nova
————————————————
版权声明：本文为CSDN博主「hengrjgc」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/hengrjgc/article/details/42063623

		自动登录	找回密码
密码			注册

config drive on openstack subnet 没有开 DHCP

浏览过的版块