问题情况
8 k% y/ m) Q& S4 H( `openstack xina版本创建虚机后，虚机在dashboard上获取到ip地址了，但打开虚机控制台之后，使用ip add 检查网络状态时，虚机内部并未获取到ip地址：' w7 v9 Q) Q% Z% T5 u
( S, j9 I' ~& g[td][tr][/tr]| 正在显示 1 项 |
# R& A: V8 K0 F% ^9 } G3 V | Instance Name | Image Name | IP Address | Flavor | Key Pair | Status |
: O0 _0 H+ Q9 @) ~$ g5 E | Availability Zone | Task | Power State | Age | Actions |
( c8 l e( \2 ^ z4 J | m2 | CentOS-7.9 |
- y* E. m, w9 {1 S- I o# S. _) F9 l9 U0 ^7 v" b
9 b" e# E- i1 O, |4 |5 V: G& M5 B
: a8 Q) y& w; j9 Q( _" p7 a9 R, Q% u3 z! g3 } O# O$ n9 b0 K9 g5 m
172.168.10.101
| m2 | - | 运行 | | nova | 无 | 运行中 | 12 小时，14 分钟 |
# k! S' ~* @$ D6 |; n |
6 G2 Q5 \6 U( h- K7 i6 G r1 \6 V* R
 5 s. w" u( {8 [9 c c! |8 f
8 b) O2 s* }4 Y4 W
分析排查思路：) G2 G; C7 W. H/ Y- j- s4 N3 }" h7 ?
( _9 a6 Z0 @/ e（1）检查neutron服务状态，确保dhcp服务正常运行：
: H. I; x, r$ P! X$ q+ I
6 q6 }' \9 h1 Z8 D$ v[root@controller ~]# neutron agent-list " e: _ C: t" N, F0 L, A
neutron CLI is deprecated and will be removed in the Z cycle. Use openstack CLI instead.
1 t" s% Y: W: J; o ~4 ]7 e: Y+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+1 m- ~/ D% L5 c) E
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |0 b+ Q! n$ X. i. x
+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+
( e. S2 E0 o2 R A| 133d6414-7d3c-42f5-8422-90ab1c7f3721 | L3 agent | controller | nova | :-) | True | neutron-l3-agent |+ t/ ~/ P0 N' D1 y3 ? m
| 2bfc7c83-94aa-4fdc-b7e2-055bb8db0f10 | Open vSwitch agent | compute01 | | :-) | True | neutron-openvswitch-agent |' o5 L- [5 h; k# W: O
| 4164d4b2-04f8-4d78-b514-351b1205d3ce | Metadata agent | controller | | :-) | True | neutron-metadata-agent |) ^5 l& G& I7 x' U
| 53fa495d-8039-4580-b1cc-20414ef1303d | Open vSwitch agent | controller | | :-) | True | neutron-openvswitch-agent |% a1 Y( g8 g# A" C9 P
| ef59abb4-35d0-48c6-876e-983ed713e2d4 | DHCP agent | controller | nova | :-) | True | neutron-dhcp-agent |
0 X" E; G0 N( ?+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+
! U6 G" S& c D& O% X( q6 Y q- ]' R& v4 ^( Q1 F& s7 \: X
. B) U' G4 X" U$ v7 r7 q% G) u
（2）查看dnsmsp进程：" [8 t8 D5 H q9 h- U/ g$ w
9 D8 ]0 D0 r1 B& @
[root@controller ~]# ps -ef |grep dnsmasq
+ \' N2 P G/ o0 ?3 Jdnsmasq 3548 1 0 07:52 ? 00:00:00 dnsmasq --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/host --addn-hosts=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/opts --dhcp-leasefile=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-ab92c638-b52e-4c32-8675-38b24f608b55,172.168.16.0,static,255.255.252.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=1024 --conf-file=/dev/null --domain=openstacklocal( W9 Q- H( m8 B; P9 a
dnsmasq 3553 1 0 07:52 ? 00:00:00 dnsmasq --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/host --addn-hosts=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/opts --dhcp-leasefile=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-e7722a92-a4ab-439c-b7af-129133c310b2,172.168.8.0,static,255.255.248.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=2048 --conf-file=/dev/null --domain=openstacklocal
; f4 K+ N- b/ p8 k! yroot 5024 2518 0 08:15 pts/0 00:00:00 grep --color=auto dnsmasq
2 E( @- Z8 y8 `$ G7 H3 Y* U5 i0 u ~* ]$ B
(3)检查ovs网桥中的 br-int 集成网桥是否有 tap口设备 连接到了dchp-agent 的 namesapce上 7 _1 @: X: C p: m
# @( y8 Z2 W M* w {& f4 k
" w( Z' D) \( m% P9 c* `0 k" y[root@controller ~]# ovs-vsctl show
2 n O7 J- d5 l: Y5 y$ Z' h04659b20-7658-4782-abe5-84ee5f33282f
' y* s+ z; Q3 ]- p: R Manager "ptcp:6640:0.0.0.0"! t3 ? |: I& z1 s" S
is_connected: true
U. H% P7 q& S Manager "ptcp:6640:127.0.0.1"7 Q$ D1 l _4 C: ?2 q" ^2 j6 A. Z& b
Bridge br-tun
; ~9 y9 n' J" q Controller "tcp:127.0.0.1:6633"
+ K. W7 {% @2 v8 h1 j8 I3 W) N" x is_connected: true
) V7 \% c/ h1 i. I1 Y fail_mode: secure7 A' @" y; M Y
datapath_type: system
3 n* k$ {' a$ G! u) ?4 x; s Port br-tun
' @0 N% k. r. G: N [* p( n8 v Interface br-tun& h/ m8 X2 B. V7 A1 I* @( b. j' n
type: internal
7 }9 E* v2 \1 U7 r# W1 k Port patch-int
y; P- g- O( s+ L Interface patch-int, z. B5 b! ^: k3 A/ z
type: patch$ b; w% M! g# H( b6 X! @
options: {peer=patch-tun}
/ m) Y0 e T8 Q. Z, p" u; f6 z; ` Bridge br-int, ~ y- M* r, ] r6 |. O- Q: U* [
Controller "tcp:127.0.0.1:6633"
* y8 P5 P* X: u1 x0 p is_connected: true
2 N) p. B' X" J! ?, Z3 _ fail_mode: secure5 f# u9 ?. s0 x
datapath_type: system
" b9 Z- o! {4 i) @( Y; m Port patch-tun
% a# r k" m9 V) o Interface patch-tun4 J1 z/ u7 `2 s' H" [. ^, K: w9 p2 n
type: patch
8 K9 `( D2 M' _! \0 g options: {peer=patch-int}
3 o' Y6 e! Z. u h8 A Port tapd2a5f73d-5b; n2 \8 G) ~& b# o: w4 J
tag: 2
# I/ x. j+ ~: ]' ?) X Interface tapd2a5f73d-5b# i, e& _: j; a* @" O
type: internal/ v \( A4 G2 b& ^# b
Port tapcee79ebe-a5
" I+ H2 Z# W/ a1 ` l- C# u tag: 1# b% t) f0 ?( T i5 j6 V& m
Interface tapcee79ebe-a5# ?9 ^1 j O7 U+ ~# x% k
type: internal
3 [# d3 q+ a+ A& | Port br-int
1 L3 L. T: P7 p$ |7 W b% Z* } Interface br-int
. W- B9 X$ j+ O* M" X4 d+ ] type: internal: V! y: l" t+ ]) l3 C
Port int-br-ex
* s' V: g# L8 ]9 B3 J5 V: H8 h! s Interface int-br-ex
( C3 v$ h/ s" p5 t) H type: patch
8 Q1 Z6 h3 c$ y, u- G" K/ @, Q2 d% K options: {peer=phy-br-ex}. g) J- h8 V6 M1 U
Bridge br-ex8 N, R8 [: k3 J0 r* B( y7 G. T8 z
Controller "tcp:127.0.0.1:6633"
0 d& ^( _& ?2 K9 d3 ^) j4 C' q is_connected: true
( [/ ~! k% k B2 N0 u" a5 P7 R fail_mode: secure
; j8 |' n& D N/ L datapath_type: system6 R" D' g8 A! U- o& H4 W# p+ q$ v
Port phy-br-ex
: I. S( O) r, q3 K Interface phy-br-ex0 B# K% F3 ?" k4 v( h7 r6 B
type: patch
" w9 z8 T# e9 F6 L) L4 @9 t options: {peer=int-br-ex}' k5 {9 @3 x$ X" X: z. c
Port enp7s0f05 V( M7 i2 P0 S! N% ?: K
Interface enp7s0f0+ x0 K# j: d0 U$ W
Port br-ex
4 W1 L* {3 r5 y Interface br-ex/ h' K4 \" {5 j% s5 r8 d7 l; I5 F
type: internal% l B$ I/ i- C$ V
ovs_version: "2.15.4"4 p( R/ Q" a2 Z, m O
# W: T; c- P# I5 I
) x o" r2 l& ?. N% r. f在dhcp命名空间中找到对应网络的 namespace 中找到 br-int 网桥上对应的 tap 设备，然后查看 ip 配置：
3 X. o5 N X" [* Z: n. T, q, ?' |- S* J) g% D2 y% H0 N
( v8 v6 \( P* P* U6 t I
[root@controller ~]# ip netns show
0 O, [! ^. ^2 `+ ~- [$ ?* V+ wqdhcp-ef99d400-71e0-468f-a969-e5d63fd79dc3 (id: 1). f) R7 s( S [# ~' q3 D
qdhcp-b3fdf316-0089-4ef3-9674-bd8fd8d6edaa (id: 0)1 \2 t. x; ~3 [& X7 A9 F$ M
( v P& d! k$ ~! |[root@controller ~]# ip netns exec qdhcp-b3fdf316-0089-4ef3-9674-bd8fd8d6edaa ip a
* Z9 |7 i8 Q% Y, K1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
% ~1 j& |% u6 l4 f link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:008 ] S2 p/ _0 J
inet 127.0.0.1/8 scope host lo! j3 O( r* k- ~$ H' u
valid_lft forever preferred_lft forever
) Y0 u9 V0 Y& W/ R" b( N! f inet6 ::1/128 scope host + j; O# S: Y* Q# b& N" c- l
valid_lft forever preferred_lft forever
( j3 s/ B4 e3 |6 B14: tapcee79ebe-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
8 q, y! P- f8 V( r7 T3 d link/ether fa:16:3e:0e:1b:80 brd ff:ff:ff:ff:ff:ff
6 ?" S) i- ? A+ U% ~. \$ F/ q inet 172.168.9.2/21 brd 172.168.15.255 scope global tapcee79ebe-a53 l1 f8 O) [; K- T+ p. f' K: L
valid_lft forever preferred_lft forever* Y* ?$ y" V+ k" m
inet 169.254.169.254/32 brd 169.254.169.254 scope global tapcee79ebe-a5
! n# U+ ?# j* ^# j" T7 K; V valid_lft forever preferred_lft forever
5 F( e4 \5 s" B( `2 q inet6 fe80::a9fe:a9fe/64 scope link $ P B( v6 A3 q U
valid_lft forever preferred_lft forever
S- ` B! g- O d" s inet6 fe80::f816:3eff:fe0e:1b80/64 scope link 9 ^& \# |* g \8 \. S5 D, h! h
valid_lft forever preferred_lft forever) n* e1 i1 H! s6 B' i
S6 M9 q3 J, I& |# i2 L& p% p
. [' K' H' o9 s7 a4 s! t8 q) v" d
定位问题：9 O7 @7 f$ R& M7 B/ l
通过上面排查，发现br-int 上是有tap口设备的，也已经连接到dhcp-namespace中，暂时没有找到问题的原因
) f9 l/ N( L" ^ U& G
( h8 S) L9 e0 ~; I' ^' d+ Z
& D* J: K* n) j2 ^7 g- W5 S. B$ J8 K
[root@controller ~]# ip netns exec qdhcp-ef99d400-71e0-468f-a969-e5d63fd79dc3 ip a4 m& s. e% ~5 }) b" _+ }7 |
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
8 y: n" T/ ^6 ]4 B! ]5 ^ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3 M4 x) Z$ x0 J, n8 a inet 127.0.0.1/8 scope host lo
( J* ^0 K* T6 p; W valid_lft forever preferred_lft forever
7 P. B4 y/ w8 ~( L; U inet6 ::1/128 scope host & @& X" `6 n, z9 O
valid_lft forever preferred_lft forever
! c5 P2 y- g8 Q15: tapd2a5f73d-5b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
. a: d* m( x: ^% ^ link/ether fa:16:3e:22:dc:dd brd ff:ff:ff:ff:ff:ff
* H w& I7 O& H A @0 |! m inet 172.168.16.1/22 brd 172.168.19.255 scope global tapd2a5f73d-5b
8 n) t4 D3 u( Q9 H5 I' a+ k valid_lft forever preferred_lft forever
' L) Z, m& ]5 R2 c- T d3 R inet 169.254.169.254/32 brd 169.254.169.254 scope global tapd2a5f73d-5b4 g! Y- {) {) B" P
valid_lft forever preferred_lft forever
$ e! P6 K& c, y* C$ R9 D inet6 fe80::a9fe:a9fe/64 scope link , b# O- d0 W- t/ R
valid_lft forever preferred_lft forever
* a* ?' Q& Y6 I S inet6 fe80::f816:3eff:fe22:dcdd/64 scope link
+ A2 D3 g, b! o- u3 p valid_lft forever preferred_lft forever
6 W) A6 H+ j, ?[root@controller ~]# ip netns exec qdhcp-ef99d400-71e0-468f-a969-e5d63fd79dc3 ip a# h% H0 _1 I, H
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
- _ D( S3 B3 ]- B- a, i link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
) \. E7 u5 B+ d5 X. X inet 127.0.0.1/8 scope host lo+ o5 ? R$ S6 t' g; d8 ?# `
valid_lft forever preferred_lft forever
) q2 Q3 k$ ]7 i; L: j+ f inet6 ::1/128 scope host 9 S% u+ @7 g% p% G- t3 m
valid_lft forever preferred_lft forever
3 f) R E0 d" |& x15: tapd2a5f73d-5b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
# o: g* z3 ^3 ~1 |; R6 T link/ether fa:16:3e:22:dc:dd brd ff:ff:ff:ff:ff:ff
4 s( x: g( d1 G+ F5 U2 i( l inet 172.168.16.1/22 brd 172.168.19.255 scope global tapd2a5f73d-5b/ J; `' t0 F+ C% _8 u' f) B8 L, h7 U
valid_lft forever preferred_lft forever
4 I0 K, V1 v5 v8 s9 J& a; e. p inet 169.254.169.254/32 brd 169.254.169.254 scope global tapd2a5f73d-5b
& T+ ~/ B8 I) P1 h7 I4 ]6 b" N valid_lft forever preferred_lft forever q' j- J& _5 d/ S8 J2 l& y2 Q' v# c
inet6 fe80::a9fe:a9fe/64 scope link ; I N5 J6 d4 l5 h1 r1 h( {
valid_lft forever preferred_lft forever
6 a; [4 `( R- C# n( K6 Q5 j7 A inet6 fe80::f816:3eff:fe22:dcdd/64 scope link + z" F/ q3 z2 a6 v8 I
valid_lft forever preferred_lft forever
+ ?' O: @ {* v5 K[root@controller ~]# ip netns show
) i9 l. a/ o3 C- M" mqdhcp-b3fdf316-0089-4ef3-9674-bd8fd8d6edaa (id: 0)
2 x- m$ e, J2 \8 _( A* O7 f- Mqdhcp-ef99d400-71e0-468f-a969-e5d63fd79dc3 (id: 1)
" E3 J/ `3 w: S9 k[root@controller ~]# ip netns exec qdhcp-b3fdf316-0089-4ef3-9674-bd8fd8d6edaa ip a
" b1 l3 }" n h. n: K% W1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2 h. {# k" k; M) A( Z/ f; r/ u. u7 w link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3 f/ Q# K+ ]# ? inet 127.0.0.1/8 scope host lo; s6 b/ L0 u/ k4 k/ B% H5 }
valid_lft forever preferred_lft forever
, V0 ^3 s% S1 a0 t inet6 ::1/128 scope host
# ]$ a7 Y2 `) ^% Y8 l) y3 {# A valid_lft forever preferred_lft forever7 S% ]* B5 q; _! Y
16: tapca61a844-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 s3 i( M- n1 U* R$ m3 `* C
link/ether fa:16:3e:3f:e4:a4 brd ff:ff:ff:ff:ff:ff# [' G1 u3 E- a, ]
inet 169.254.169.254/32 brd 169.254.169.254 scope global tapca61a844-c4) V1 k8 @. o n
valid_lft forever preferred_lft forever/ R/ L9 j, [8 e0 J% L8 x, t: D
inet 172.168.8.1/21 brd 172.168.15.255 scope global tapca61a844-c4
$ `. d+ o+ |$ W. X) \+ E: s valid_lft forever preferred_lft forever5 r/ j' b* I3 u9 X7 z8 b: i+ b
inet6 fe80::a9fe:a9fe/64 scope link
& Q. ^) p) D" r' x valid_lft forever preferred_lft forever, m) z+ A( @5 C& z
inet6 fe80::f816:3eff:fe3f:e4a4/64 scope link
- m3 x. b. X0 S9 s9 x) z9 `: W valid_lft forever preferred_lft forever+ A5 l! }6 _' P: ~0 [ l
% S% }! r6 I+ y/ [/ a) {! F% @& \* y5 j2 A% _0 U: U
+ b, A4 W% ?- u" J S' A) w1 y
$ a9 y" d9 I2 R+ D
重启虚机，之后依然没有办法获取到IP地址。% [2 f1 F* _3 j4 _" K, z* M. K
1 R: A8 d- c% @2 N. R; W; z8 O$ w9 t4 a$ ^
! M Y. s# L; @
在创建虚拟机下发请求后，dnsmasq进程会给虚拟机分配好mac地址和ip地址，并写入到/var/lib/neutron/dhcp/network-id 目录下的host文件中。虚拟机在内网中发送广播来获取ip的过程中，dnsmasq 会监听到然后将host文件中的对应ip通过dchp-namespace分配给虚拟机。 所以，在虚拟机获取ip过程中，必须虚拟机发出的包可以到达dhcp-namespace 经过的虚拟网络设备都存在且正常工作。 如果没有在subnet中开启上述的dhcp功能，那就少了一个对应网络的name-sapce dhcp服务了，所以虚拟机获取不到 ip。 : m) c2 N& h3 E6 }6 _, G
& u* }- @1 J$ A; D# n; B m |
发表于 2022-5-23 08:10:18