马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。
您需要 登录 才可以下载或查看,没有账号?开始注册
x
问题情况
. I" e+ y2 }5 [" A3 k9 [openstack xina版本创建虚机后,虚机在dashboard上获取到ip地址了,但打开虚机控制台之后,使用ip add 检查网络状态时,虚机内部并未获取到ip地址:5 y# J5 }6 }$ b
$ x* s$ M( j4 F5 t, F3 m- n
[td][tr][/tr]| 正在显示 1 项 |
5 i& b, E/ Q: z! j1 \ | Instance Name | Image Name | IP Address | Flavor | Key Pair | Status | ; n6 U& _! J& s
| Availability Zone | Task | Power State | Age | Actions | - Q; K. C, Z) T5 ^
| m2 | CentOS-7.9 |
" E8 S: G( Z( v& N# `% V
# d% ~2 z8 _/ N5 _
) C8 l4 i4 U; e U+ }5 U' H
0 r6 G: l' Z- G/ k* X, R
, d; }0 {: R1 _2 |, u5 r3 F172.168.10.101
| m2 | - | 运行 | | nova | 无 | 运行中 | 12 小时,14 分钟 | % [7 a4 ?# A m$ ]* ]! x6 k
|
K- K& X- r7 j4 |: a U# _6 B! I" p: r0 r6 e `# V& R4 i
/ D4 _2 `! c6 A B9 D, b3 [ Y* o
分析排查思路:
; T7 N |; X; G
( K' X3 \% X& f' S$ t(1)检查neutron服务状态,确保dhcp服务正常运行:6 P% R7 ` N$ ^. ]! A0 \+ S7 q
) l$ h$ y1 y- U: |( K
[root@controller ~]# neutron agent-list ( [/ u6 w( Q' L. L
neutron CLI is deprecated and will be removed in the Z cycle. Use openstack CLI instead." [; i$ N9 }- S+ j! t
+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+
; N& h6 \' m( E# \; f% ?# s q6 G! l| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
$ |3 o+ v- F& g4 s7 r+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+
& [9 z, O6 U; p Q% {# D| 133d6414-7d3c-42f5-8422-90ab1c7f3721 | L3 agent | controller | nova | :-) | True | neutron-l3-agent |
1 x0 m( ]# x, K: P6 O1 U| 2bfc7c83-94aa-4fdc-b7e2-055bb8db0f10 | Open vSwitch agent | compute01 | | :-) | True | neutron-openvswitch-agent | q6 |9 M, ^1 z4 _! |* L
| 4164d4b2-04f8-4d78-b514-351b1205d3ce | Metadata agent | controller | | :-) | True | neutron-metadata-agent |+ `3 n4 R0 r) }* U
| 53fa495d-8039-4580-b1cc-20414ef1303d | Open vSwitch agent | controller | | :-) | True | neutron-openvswitch-agent |
- C& w& v+ q/ b6 Y: \" t4 s| ef59abb4-35d0-48c6-876e-983ed713e2d4 | DHCP agent | controller | nova | :-) | True | neutron-dhcp-agent |) p! b* G, ~/ \, L. U: S- O1 A
+--------------------------------------+--------------------+------------+-------------------+-------+----------------+---------------------------+
* u& k% Y$ ], {4 \7 a/ R# B( u! p @& L' l- F0 [
# c, K( D" Q. O7 u: Z0 o
(2)查看dnsmsp进程:
5 a3 A8 [' Z. H7 j- v8 `7 T4 P; @4 Q. h; o
[root@controller ~]# ps -ef |grep dnsmasq# z2 W; c3 u% a- A3 Q9 U0 i) p6 I
dnsmasq 3548 1 0 07:52 ? 00:00:00 dnsmasq --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/host --addn-hosts=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/opts --dhcp-leasefile=/var/lib/neutron/dhcp/ef99d400-71e0-468f-a969-e5d63fd79dc3/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-ab92c638-b52e-4c32-8675-38b24f608b55,172.168.16.0,static,255.255.252.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=1024 --conf-file=/dev/null --domain=openstacklocal
6 H( W6 l+ G' {3 G p Ydnsmasq 3553 1 0 07:52 ? 00:00:00 dnsmasq --no-hosts --no-resolv --pid-file=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/host --addn-hosts=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/opts --dhcp-leasefile=/var/lib/neutron/dhcp/b3fdf316-0089-4ef3-9674-bd8fd8d6edaa/leases --dhcp-match=set:ipxe,175 --dhcp-userclass=set:ipxe6,iPXE --local-service --bind-dynamic --dhcp-range=set:subnet-e7722a92-a4ab-439c-b7af-129133c310b2,172.168.8.0,static,255.255.248.0,86400s --dhcp-option-force=option:mtu,1500 --dhcp-lease-max=2048 --conf-file=/dev/null --domain=openstacklocal! |* _/ h$ j2 p/ K7 A! l( ^
root 5024 2518 0 08:15 pts/0 00:00:00 grep --color=auto dnsmasq% h" \5 L) O4 K% d
$ |% @0 a6 z0 W& z(3)检查ovs网桥中的 br-int 集成网桥是否有 tap口设备 连接到了dchp-agent 的 namesapce上 }. X5 F* s5 s+ X$ @( s; E4 N' r. O% |
+ \ |0 ^9 q; O1 I. M# O% F
# E; m: J) J0 y2 p$ u/ P! d6 j- D6 J[root@controller ~]# ovs-vsctl show % f7 s4 W# p3 }! n' |
04659b20-7658-4782-abe5-84ee5f33282f
1 A* {1 ]6 Z; g Manager "ptcp:6640:0.0.0.0"
- h: C; F% [. n* `; } is_connected: true
" f0 [0 E2 R) J1 Q3 y! f( L$ i Manager "ptcp:6640:127.0.0.1"
1 T; O$ C. l& v% ~ Bridge br-tun
- n( M* P; B* n/ \: @ Controller "tcp:127.0.0.1:6633"
. P( A+ ^% j" {4 @0 p is_connected: true
- K- ^0 p; ]& Y9 d/ n/ } z, l fail_mode: secure
# F2 J7 U8 ?2 x datapath_type: system4 _4 Q: c9 o4 \. R0 H
Port br-tun9 x6 d$ U; I8 d
Interface br-tun
2 ~" {8 h, L% K type: internal- l) P* F! s( W7 e
Port patch-int
1 o! n* w/ T4 w& o+ ?: i' b Interface patch-int
% A4 z9 S6 _2 D, {. {! I: J* A type: patch5 c4 \- S% n. F. n$ |( y( ^; L
options: {peer=patch-tun}) d: F# _) C% L& p4 M6 f6 W! I+ j j1 W+ @
Bridge br-int6 w- J6 Z( J& S+ e
Controller "tcp:127.0.0.1:6633"
3 w& V+ {5 p+ r% X is_connected: true
5 |0 `0 |5 ]; a7 Y/ N- W+ p, h fail_mode: secure
! F$ b- Q) Z5 G! U+ E- i* r$ j datapath_type: system8 Y% n8 b3 h) ~
Port patch-tun
% T1 w) s$ U. b' C1 w: }$ U Interface patch-tun
) G. l5 p7 }9 T! t/ i8 K6 ?# w type: patch0 s- x+ }: _8 `$ a8 q
options: {peer=patch-int}+ V8 ~" ~& n: m
Port tapd2a5f73d-5b: V- ^3 S9 D9 ]! I7 M" L$ m3 [
tag: 26 R& ]7 B) \5 Y2 e; o( |
Interface tapd2a5f73d-5b
# y& w) r( P: w0 X type: internal$ n+ N8 v+ B. E( g* z
Port tapcee79ebe-a5
5 i8 I ~% i- V. t5 k3 b+ f tag: 10 Z- v. H5 L8 y( Q' \
Interface tapcee79ebe-a5( L, a$ y% j" `4 U
type: internal
7 P' b e& _( g4 Z# T6 a Port br-int% [: q' P. ], @) S& Q1 ]1 q0 y
Interface br-int
9 H2 e& T) A+ l b7 y type: internal% w( I; P+ C. Y/ _2 I7 @5 l) |
Port int-br-ex
: W4 @% w, P; \2 O! } Interface int-br-ex
& P3 s8 x+ O9 T C k2 U0 c. f type: patch% ^4 o9 t. \0 i ?5 a. N
options: {peer=phy-br-ex}
( X/ m6 J1 r' L/ @$ { Bridge br-ex
2 s# W" @7 L+ Y. V9 K Controller "tcp:127.0.0.1:6633"& w4 ]8 V% ]' U$ ~; Z% L9 f
is_connected: true3 K4 S, L: c4 w+ I6 U
fail_mode: secure% m; A, C7 ~% R# M7 }. l" l
datapath_type: system8 |/ i; _( @4 R, T H) O
Port phy-br-ex7 }* x1 [% _) U' P
Interface phy-br-ex+ Q4 q9 a8 Z0 O! c% g2 F2 K7 f
type: patch
9 s( d( I/ B% G/ [/ O7 a# |3 | options: {peer=int-br-ex}
& u v6 B, k% V5 m# _/ O; Z Port enp7s0f0
- b7 X: m5 S* O1 Y Interface enp7s0f0
2 o, u; q9 \9 a Port br-ex$ z( _/ X3 G6 z
Interface br-ex% c3 C7 h8 S6 \/ L q" q' l) ^9 t! R
type: internal
9 v% ]* n) Z/ v5 R. d( i0 Q& w ] ovs_version: "2.15.4"
x# V' L, @; p+ b) k! _6 H- z1 ^* h" k1 Z/ D) ~7 C& B, l& C1 v
$ [2 A' R) \: z( h9 G在dhcp命名空间中找到对应网络的 namespace 中找到 br-int 网桥上对应的 tap 设备,然后查看 ip 配置:
! l7 r9 y, A: G* D9 E5 c0 g4 Y5 D! l- a: b
: I: ?/ W8 Q* \6 U
[root@controller ~]# ip netns show
. i) S3 } d) |% Sqdhcp-ef99d400-71e0-468f-a969-e5d63fd79dc3 (id: 1)
2 ` u# m, G+ a# I& A) ~qdhcp-b3fdf316-0089-4ef3-9674-bd8fd8d6edaa (id: 0)4 C' J% `$ g7 f# m3 P3 n) ~
' \$ Y/ r( i T z
[root@controller ~]# ip netns exec qdhcp-b3fdf316-0089-4ef3-9674-bd8fd8d6edaa ip a
: [2 a4 z: e5 J* \0 v$ }- g* q# J1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 10004 ]/ ^& ^5 o5 G! H: F
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00* Z5 [. {' r& p2 Q
inet 127.0.0.1/8 scope host lo
5 H1 _+ [" }" R+ q" h valid_lft forever preferred_lft forever5 N# ^! L+ l; z/ L8 O( |2 v
inet6 ::1/128 scope host ( u' Y0 N2 \/ ?$ j7 z
valid_lft forever preferred_lft forever8 N' U9 y( `/ m
14: tapcee79ebe-a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
/ w8 I4 F w2 N- @/ q link/ether fa:16:3e:0e:1b:80 brd ff:ff:ff:ff:ff:ff
; N6 S5 S" d3 ?' b# z inet 172.168.9.2/21 brd 172.168.15.255 scope global tapcee79ebe-a5# b7 v$ N, d* p; e# |
valid_lft forever preferred_lft forever" ^4 a9 ?1 F0 T0 |* a" U
inet 169.254.169.254/32 brd 169.254.169.254 scope global tapcee79ebe-a5
# t. P" Z( A# i) ~ valid_lft forever preferred_lft forever
9 ]* M. E6 F& b2 b! l/ X inet6 fe80::a9fe:a9fe/64 scope link ! ^/ e) q( n( W9 u% g
valid_lft forever preferred_lft forever
6 [5 v7 O, X% L! z3 f- w inet6 fe80::f816:3eff:fe0e:1b80/64 scope link 5 W1 c# i* Z/ @# T' Y1 W
valid_lft forever preferred_lft forever
. W2 F) j2 }% d: b1 V! e! c( F; ^ I K2 R3 m
. u3 @" J8 T( d+ W+ O3 x
定位问题:3 z; Y9 x c, c3 x, f& q$ d, [
通过上面排查,发现br-int 上是有tap口设备的,也已经连接到dhcp-namespace中,暂时没有找到问题的原因8 U7 D3 u+ S( {, O; ~
) y* k7 N2 l$ J5 [8 T/ o7 Q3 i9 m5 d, l
% _5 H2 E& _8 F2 a, S2 G8 q* i
[root@controller ~]# ip netns exec qdhcp-ef99d400-71e0-468f-a969-e5d63fd79dc3 ip a
; k" s/ f, U- t1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
* ^# k4 y2 H6 n/ W+ n link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
7 A# Q% R2 ^$ d5 f. b0 k inet 127.0.0.1/8 scope host lo) O9 c' T; k2 `
valid_lft forever preferred_lft forever
/ p# b3 N' u, ?# ]% e) ^5 S) R inet6 ::1/128 scope host * P6 G, T4 ~- R- z- H
valid_lft forever preferred_lft forever
4 `1 w7 x0 i6 k9 F5 L15: tapd2a5f73d-5b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000# g3 N/ l5 n3 U1 j2 @, R
link/ether fa:16:3e:22:dc:dd brd ff:ff:ff:ff:ff:ff
/ E" c0 J" ~6 t E# }/ u9 O6 j inet 172.168.16.1/22 brd 172.168.19.255 scope global tapd2a5f73d-5b7 E, ]2 o7 v' F2 {
valid_lft forever preferred_lft forever
* s# f8 ? x2 R) h" m* U inet 169.254.169.254/32 brd 169.254.169.254 scope global tapd2a5f73d-5b
0 X/ n' V: d: A/ ^ valid_lft forever preferred_lft forever
5 ^2 {1 G5 Y% l0 O1 y inet6 fe80::a9fe:a9fe/64 scope link 9 X# K6 n, Y% W: w
valid_lft forever preferred_lft forever
# ^- b! J6 f2 Y: A inet6 fe80::f816:3eff:fe22:dcdd/64 scope link 8 W; b+ }* P1 ~2 S- D r' f
valid_lft forever preferred_lft forever* U! n, N% U( D; V6 }
[root@controller ~]# ip netns exec qdhcp-ef99d400-71e0-468f-a969-e5d63fd79dc3 ip a5 b! G3 ~" E( u* X1 y- |
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
0 |4 `: q. P3 a9 c2 ] link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+ g: e( h8 z5 q2 i inet 127.0.0.1/8 scope host lo
+ W- P3 A; f% i! N3 M# \ valid_lft forever preferred_lft forever
# f' O9 i/ D9 {6 T+ z inet6 ::1/128 scope host
) R! ]! V" @1 C1 O3 |7 V0 A# _( o valid_lft forever preferred_lft forever
& p1 L) ]4 v6 ^, C/ M6 O5 c/ q15: tapd2a5f73d-5b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 10001 R& F3 `+ ^1 Y% N& _1 {: k
link/ether fa:16:3e:22:dc:dd brd ff:ff:ff:ff:ff:ff
1 v8 E% i& H1 i' }# }. l inet 172.168.16.1/22 brd 172.168.19.255 scope global tapd2a5f73d-5b
4 E9 M4 j* ]& ]7 F1 X valid_lft forever preferred_lft forever
+ N9 a: u8 i [7 r: \! F inet 169.254.169.254/32 brd 169.254.169.254 scope global tapd2a5f73d-5b
6 e5 d4 U: k. M0 V% n valid_lft forever preferred_lft forever/ J6 j" ~: `# Q
inet6 fe80::a9fe:a9fe/64 scope link ' C5 m1 H3 ? V$ N; n; T, e, _* N
valid_lft forever preferred_lft forever( w% {' z X& Y4 B, E1 h' o
inet6 fe80::f816:3eff:fe22:dcdd/64 scope link
% B8 S- o8 s; _. A/ q; @6 A valid_lft forever preferred_lft forever
6 @2 e6 C; u) y" `+ d* R7 ?! {[root@controller ~]# ip netns show
4 D% P) l$ n' ]1 H& ~ W0 jqdhcp-b3fdf316-0089-4ef3-9674-bd8fd8d6edaa (id: 0)7 |1 W7 B% o/ e& ]
qdhcp-ef99d400-71e0-468f-a969-e5d63fd79dc3 (id: 1)8 }/ r1 W$ C+ D3 h& m9 V
[root@controller ~]# ip netns exec qdhcp-b3fdf316-0089-4ef3-9674-bd8fd8d6edaa ip a
4 a) P# q; n* \5 `- p% Q9 h$ h1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
* s- X) O: r! J& w) I6 v link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
7 q" Y9 l, ~) f" g1 T( R$ k inet 127.0.0.1/8 scope host lo5 `9 e& M; |, |6 L
valid_lft forever preferred_lft forever9 x- C; A* g0 T" C0 ^
inet6 ::1/128 scope host
- j) W8 Q0 ~7 A o. Z valid_lft forever preferred_lft forever S2 Z C5 v0 @
16: tapca61a844-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000: \" U( D& T+ q/ A5 s/ p
link/ether fa:16:3e:3f:e4:a4 brd ff:ff:ff:ff:ff:ff
/ p8 {0 h6 `; p inet 169.254.169.254/32 brd 169.254.169.254 scope global tapca61a844-c43 c; f$ w. {* o I; v4 P
valid_lft forever preferred_lft forever' O- N( C4 C5 k1 T8 K
inet 172.168.8.1/21 brd 172.168.15.255 scope global tapca61a844-c4+ u$ ^ L+ n; Q7 _
valid_lft forever preferred_lft forever A* D- u% f7 Z2 W* K, R8 q7 ?
inet6 fe80::a9fe:a9fe/64 scope link + K. [# G! U2 D4 _3 R4 Q
valid_lft forever preferred_lft forever9 g: ~8 w% ?0 X- h4 Q+ k
inet6 fe80::f816:3eff:fe3f:e4a4/64 scope link
3 S. e N) ~7 G; A4 E valid_lft forever preferred_lft forever+ k$ G& `* A; [6 C; n1 c
+ `. D; C" F7 Z( c
4 ?7 a& @8 R( y& E
4 ?$ F) V- [+ c- I# Q- B
; R& [. f$ i8 ]2 f重启虚机,之后依然没有办法获取到IP地址。
; M' c" z- ^& S, n: b' E( n4 q- y9 F- V+ T
( ^8 m" i4 Y/ c- Y
! |1 c5 H9 ?" R: {: a8 m% d在创建虚拟机下发请求后,dnsmasq进程会给虚拟机分配好mac地址和ip地址,并写入到/var/lib/neutron/dhcp/network-id 目录下的host文件中。虚拟机在内网中发送广播来获取ip的过程中,dnsmasq 会监听到然后将host文件中的对应ip通过dchp-namespace分配给虚拟机。 所以,在虚拟机获取ip过程中,必须虚拟机发出的包可以到达dhcp-namespace 经过的虚拟网络设备都存在且正常工作。 如果没有在subnet中开启上述的dhcp功能,那就少了一个对应网络的name-sapce dhcp服务了,所以虚拟机获取不到 ip。 2 Y" W1 Y" \( ^# ^
: e3 u+ ]9 ]( h% |# C- j5 Q
|