|
|
一. 安装环境:& H5 Y+ G2 m3 T
9 f. q2 d) j' U/ q1. Juniper EX2200 Switch
$ t4 e3 q% i8 }" B/ z1 w. p7 P! s+ g; E2 s5 e( ^
2. RHEL7.1 OS
' [5 x B2 P; `: X
/ }! I# s' ]% ?. u. B3 N( E- @3. 1Controller +1Compute
3 L# D, T3 b; r8 R2 m, o U; f6 B! c' a6 |& X
7 e9 i! P; M- S# y9 A& J9 C
5 n7 {1 s0 R/ u' H管理网:eth0 10.14.0.0/22" C. j7 h5 _& ?
+ n, o2 L; [3 R% ~
数据网:eth1 使用vlan-id 104创建网络192.168.104.0/24,使用vlan-id 105创建网络192.168.105.0/24
* ?, f) R" ^& j1 m! f$ t0 w
. @) d8 S# S6 W+ x
. a! `# x3 Y, H# @ t ^, E
. `0 F* N5 Q5 X; T- _7 B* oController eth1 连接Juniper port ge-0/0/27.0,Compute eth1连接Juniper port ge-0/0/28.0
4 P- h1 U) b3 V | M5 h2 N+ C- z( t; A9 I7 n$ T: `
" x+ w) J; |) k% O0 O1 ^ F. }$ b6 P& f \: W4 |
二. 交换机配置
/ b9 M$ ^* _5 d0 W' C8 f u1 E; ]! ?: z. x; I1 k$ ^) T5 I
1. 设置port为trunk模式:
* M5 y3 _- v5 Q% {2 k, X
- D; X% {: r! d
/ R [ Z- \6 K6 l
0 W2 B: Z0 }8 K: I; q: C- $ V3 _1 \( P5 n3 ~
4 ~. E9 ]1 ]2 ~& ^/ l4 J! Droot> show configuration |display set
* X( A& [/ R* B, M4 u( B' ?* I, S+ l+ A+ o- r+ F( u" Z% d
- + W# k4 Z, e9 R8 t6 Z
* A8 x1 B1 D; U) a S/ n$ Dset version 11.4R1.6+ ]5 \: y# Z. C$ w- H
5 T% B% s, b$ r9 Q7 f# m" x
0 c4 \, |$ { R) o6 @1 O! r4 m5 K: {& H- n
set system root-authentication encrypted-password "$1$jC9l6Bwy$ePVIDxgH.8iXqDfqrLzuJ1"* N2 \( d0 ^- w7 p5 r
! g3 f6 R2 S0 X4 A J+ k- - g9 S& n y+ ~6 e2 P
: {+ n( } J( a& r4 _3 Dset system services ssh
# u- ?$ y- t* d9 Q7 {- U0 P( k% X7 {6 {/ i* l
- 7 p& L' }4 L6 d, r
7 j7 `% p, F4 P9 o3 a8 |2 o
set system syslog user * any emergency5 a8 I# y6 z- o5 D5 {
4 T' ?$ b0 [# `& c - ' o+ |$ p$ x4 F; e9 y/ T- h
5 ~7 K0 m6 m X3 Eset system syslog file messages any notice k# G$ @) R2 T$ Q, o# l$ r' F
* U+ n- C# c! A+ R$ n3 ?
- 3 r2 t& }9 K2 S+ Y ~% [% I7 H. P
0 q( K' _" E; }4 o. H
set system syslog file messages authorization info
5 H4 z7 Z- z6 r$ X. y3 _. Q
& l! b3 x+ z7 K - 2 \5 ~( J2 Z+ K8 {0 ~ Q+ p; c8 d
4 e/ C" n$ s0 ?% P+ b1 B+ W vset system syslog file interactive-commands interactive-commands any
1 D$ E( ?! Z5 i. Q* r/ s3 w
" D$ f: ]3 o' \; c
- v2 X$ I( K4 R+ q5 h% v9 e0 B' c ?2 }8 m2 U8 v
set interfaces ge-0/0/0 unit 0 family ethernet-switching" [3 P+ C+ l0 R7 A! n% y ]* [
) h" ~5 t3 H$ B
$ [1 B: x$ H" p( S( A. Z& _$ [+ n" r
set interfaces ge-0/0/1 unit 0 family ethernet-switching
% P' Z1 T( J6 o: K- }& ?9 r; P8 W) S: T
- ( ^2 w0 w3 ~. W( \4 ~
; p7 b2 ]' \, c! [' y! M
set interfaces ge-0/0/2 unit 0 family ethernet-switching- ?7 V( O v0 B/ p: Y- m2 F
% Z6 f D8 \! j& y6 [; e1 E2 N
9 X/ S" P5 q) S' @4 H
# G( j3 U6 X- C( t xset interfaces ge-0/0/3 unit 0 family ethernet-switching8 A. z) a! N/ b
6 t$ `1 L/ }/ N- , Z) D2 u" U2 z
6 D5 ?+ i& Q) t+ m, _& f............... O8 l5 u6 z/ z% C: L. g) A; H
4 m7 P* Q y1 h+ z2 F
) _% m. B1 l+ ?+ H' l2 c* `1 w
0 P. z, g7 ~* J* t<strong>set interfaces ge-0/0/27 unit 0 family ethernet-switching port-mode trunk
, r8 B* Y4 u) z, _/ d
$ ]! x A/ D; h! s( j! H. P0 ~/ [) ~- + O. ?! X( _( ~" L8 u
6 o; k& l& E+ gset interfaces ge-0/0/27 unit 0 family ethernet-switching vlan members all
# C. | V' V7 s5 A
: O7 c+ Q3 T& l3 J1 r4 O, j6 h
8 j" \, w# P( X% K* k5 y6 C1 }
' x7 d; U- J7 l/ }' Sset interfaces ge-0/0/28 unit 0 family ethernet-switching port-mode trunk% M4 n/ P& k* e) n! I# X% w- Y+ p+ H
) ^% h3 q: A2 F8 I6 n8 ~
: _: H, m6 j+ `1 X8 m8 N/ q7 e0 @6 @" P. B7 _" f
set interfaces ge-0/0/28 unit 0 family ethernet-switching vlan members all</strong># [; b* \8 u' g2 R, R) |
! d/ ]9 }2 ]* a* |2 j
0 r% g; c# d8 C% C0 v
* @% r" M* e9 D9 I! m; t.....................4 b% m9 v# l2 O7 [$ t% t
$ O3 _* [. k* D' B3 C8 H+ m& x3 x j& B! r- z! o% `3 _( V! m7 d3 ?/ W
- e0 Z7 C. A2 M2 \3 Y7 @/ o
' c5 L9 O: \4 P% G9 u3 z- T0 J8 s2 ^
+ ^; e( D2 `9 s$ I$ s- h
. p( h. o4 Y' O2. 设置vlan id 104,1057 m) w5 m! R# }
2 {4 H0 }* n2 M- d" [2 X' C D o6 O/ P! W. g1 Q, `5 T( H
s2 y% z1 Q5 ~! O6 _
2 l; S, s, h) k+ d/ x W) c<strong>set vlans vlan104 vlan-id 1047 G. Y$ ]6 q0 z; K7 x( e8 f$ \
8 I9 r( g" h* C! ?+ J# Y2 c
' D- B! V1 o0 q
6 B- p" P) Y% _- mset vlans vlan105 vlan-id 105</strong>
6 b( d' Z, G$ \, b# [5 r$ X% P" ?2 j" p) A
/ E Q2 c8 T" C. d$ ?1 [& V) H2 y$ `$ |5 Z2 G7 n& M
set vlans vlan108 vlan-id 108$ u+ g5 z7 |1 p3 R
6 |! P# D( ^% g U2 Z0 d
- ( p* z, y3 K& C$ O3 V% w7 p5 f; v
w: l0 { x4 xset vlans vlan110 vlan-id 110
: ^* B& \- |- e/ o1 Z4 \7 M) E1 D1 M, I+ n9 I
/ o& C* J) T9 l4 h8 V1 D7 ^
) ]* D+ P+ Z6 T9 dset vlans vlan110 interface ge-0/0/29.02 i& F$ `( @+ j. |6 H
# e# N- V" m+ j
2 @+ H; a* a. A' }7 }+ _; V6 r
/ p9 Q! l; j y$ ~( y, `5 Dset vlans vlan111
- {1 ^! D h- |7 u! Q& T% G6 w: o8 Q! U# K/ P' n4 C. h% M- E+ f
" g& E' i% d0 }3 z5 Q8 E4 m) P& l" T5 [( i& X9 M3 H
..........................( {' O; @6 ~# A2 ~# X8 z
- y% F6 w |( H2 a/ K2 [4 p* C: r4 ?$ v! _6 P5 n6 n5 V3 F& X5 `+ ^% @
' o5 \3 S% ?% E0 |
% M9 g" q$ Q, u4 h0 ^9 g4 |9 K( P# ?
( o- E* d# E" a, K& F) q0 R, [1 t6 C9 V; v5 r. I
! a$ B& k0 H. x, _
ps:trunk模式下端口对带tag报文的收发处理' n1 K) v) b5 H
7 ^9 G4 H) D3 ]6 A1 @[img][/img] # Z5 f" V4 V0 w4 ~
6 u+ V1 L, |. a$ x
7 S% I0 ` P3 a$ S- V+ l
[img][/img]4 f% S' e4 G$ M# j/ V
/ b/ v' ]* o5 z5 i" ~. ~: K5 N! C9 \& t- N& _8 u9 a
5 V( q3 @$ V! s' E4 W
' k' a, G+ O( R( }$ U4 ^
三. Openstack中vlan的配置# o! X) a& P7 z1 G m) R1 H$ P" _
- H/ m8 d$ v- y& g6 _ j
1. 物理机上的网络配置
7 }# J- D% _+ ?! d! e5 j" X) h( a' @% P5 e2 r4 n& f" H* P
Controller:/ `% n% s$ A* Q4 U
% n' U f+ N1 R# i, p- o
# }/ ]7 ~3 P- j
6 f2 v+ v) k% ~- 3 w) N( u' K% p9 d1 \
7 b0 |; @6 D# U9 i+ Z
[root@controller ~]# ovs-vsctl show- A- H% {8 f g7 Z7 J
! ^; E, b) O& z
- ' R: @* W3 Z. w
! M1 m6 R3 c* S* I3 Wf2db5bf5-6591-47f7-a134-462f66a3fde5. n/ s9 h/ N. G2 K
, S5 K& j% |" x! V4 v4 D) b+ G
$ g( M1 G0 @8 d" }# p6 Z$ N" D
& e6 A' T/ m L0 Z8 Y# r( x Bridge br-ex( ^8 Q( ?, s" g1 t: M
! c/ v k- g( o3 G
5 u2 }: Q6 }$ Y1 p& S
1 Q, S; d" Z, q4 d* R Port br-ex. T* Y4 X0 J5 @: s
F) @4 T5 k' ~0 m; c+ V9 G- . f; O! u: e( g$ s) c3 a- M
3 }. r$ p5 `# D: Q; ?; x8 U
Interface br-ex
) @2 A; {" F+ G
0 Q/ P5 z1 I% Q- ?. [! d
1 h8 ^) b5 t* Y) n' M$ t( E9 p
$ C! n2 c# ~2 S6 { type: internal+ D+ j* r* K5 X; |: {% V
A# `; t, I. X
/ S; D( i- {/ h+ H* Z4 u
3 s$ t2 x/ {7 y# x# _ Port "eth0"
2 |/ \2 `, x& [: R5 g9 O" V9 I$ K# f- y& r5 A4 t
- * J! O* K' w9 q9 C4 w) Q, \8 r$ ~* g5 K4 H
8 N* |! B. D T. @* i
Interface "eth0"
. S, t4 V- ?4 |/ C( M/ p! N. J) w1 i9 M* T! A; a; o! r
- " q& }+ a% n8 `
. }; f* Y! ]# g" X$ i+ W( p9 Z6 p Bridge br-tun
( [, o# B# K/ L
! ?; v' a0 ` W( @6 Y - 5 ]: c1 `5 a* m& ~# ?$ b
0 K/ p" s$ A: w, } fail_mode: secure$ ]3 n/ Z8 E, _, k/ f$ S, h+ e6 d
# I" n- p/ N% L/ u1 m
) a# P8 m( _) y
. Q* ^" c. g& ^; c Port br-tun. A; ]' n! [, s+ [7 {8 {
$ V3 E2 K; f& d
- 7 V0 i3 X, h) o7 v5 y& C. c
6 z) N, j; ~ M Interface br-tun
2 X3 C% U# E1 ~# k
0 u. X; Q! t% m; g8 q
( _5 d- ~3 Y$ y3 K4 B
. W- l" b9 c! R. l type: internal# P, {4 N' R0 ^0 e* w; ^/ m
( G% T/ `! @6 y! q# d$ H
* S1 D; U+ }4 F8 a/ t3 @2 G5 p" @6 Y7 w5 H- a+ }
Port patch-int6 ]5 _) z$ Y- a
2 `$ Y" j8 t) `" x6 W0 G
6 r3 n1 E8 U. V2 _" D9 h y4 h5 r+ S1 m( Q/ F' O0 C0 P
Interface patch-int
$ V/ K2 u8 a+ s" }; l3 T+ H
- }5 c, ^" g+ \. M% v) c- y- 8 t6 m0 {. k- W: w: `
# o) ]' [1 p9 D, B- e% S
type: patch
, h3 E& Q. Q2 j t3 V" e0 \2 ^; M2 m) z! ~0 H9 Z) }! ~
- 7 R6 |# C6 d0 x7 F" ~, P
4 p% Y9 {9 e" M4 o' r0 \
options: {peer=patch-tun}
! w, o2 M7 e3 O/ N
% y8 ^& h& Y/ p& N, r - % j4 c$ p2 \: b8 `9 q O1 b
0 W! n1 l' G# x& F+ w! z: ?
<strong> Bridge "br-eth1"4 f! }5 V% k8 n* y5 m
' q( M* a% Y l8 |4 i6 r - + u5 ~+ P& C3 e. N e7 J
, b2 e0 e( U! A5 s* Y3 J
Port "eth1"/ @' ^5 y5 p+ o D
0 _/ j \1 ~; ^; W
# `. a4 Z1 L) x9 [
) F! j) ]0 e: K1 V! S9 k( k Interface "eth1"</strong>! a) o& B; {6 w1 t; r. V
' |; h$ v, v: |
- 9 n2 V3 A$ S# A
( {4 _7 x( M6 o) C9 L; v
Port "br-eth1"
; s5 L7 U* c5 O: g7 e" z& M0 L1 r
6 o9 `5 G4 \4 H! R - ! n7 c' F, K, _6 J. u. k
; \+ [* M9 a0 J1 z0 T Interface "br-eth1"( F. ^/ h0 b7 k: M
8 {+ R5 R# n# m& M: ~# B
. z4 s' c! k2 E b2 D& B
1 M7 B2 [4 L- b type: internal I7 P5 ?' `2 I1 e3 p& P! o
. n2 [; c% s- H
- 6 _: N5 L$ u% D* M
% Y* b: K: H2 D( E* T8 h
<strong> Port "phy-br-eth1"
- p1 R/ f; q7 B6 r0 `" C" C$ R" k( p% n& s( l! Z2 ^
- 4 w' t5 |+ U) L* f' y- Y
2 R: x* l1 b9 P' P& c' i* b
Interface "phy-br-eth1"
( G9 S4 V1 w3 U6 h- e4 l7 w" N4 s; o/ N- O
2 u+ f( Z" m4 e! o# n' L$ Z3 ^( w% J
type: patch
4 t% f# ?, y( g: C
% N' ]: {+ [' T8 v1 r: e; O$ o! N- - X$ E* l0 D2 z9 n4 w0 ^9 y
# S: A/ Q- K# l8 Z6 p1 K options: {peer="int-br-eth1"}; b- o: Z8 d- h# k1 Z
: }1 U S& r9 [6 |8 F$ P
4 H1 T9 F6 r% D/ D
x" f" D' y; X Bridge br-int ^+ u& J/ b9 a4 M7 \: L6 W4 J, j
! x: N, D6 x' H* T" N% T
- . t( ?; H/ Q) `3 Z
6 E. T- b# i- l6 U fail_mode: secure
% F0 ~$ A: |0 Y% |. g- a& V. C" _- P! N A8 G+ \7 R
- ) h5 H/ f+ U4 w. M" g
6 E- [0 H4 X" P7 t0 z! O
Port "int-br-eth1"$ O* t8 U; |& g, |) e6 Z8 o' [
0 m* T7 f1 d: I- x
1 W, D0 @8 G" }+ U& Y! ~/ M, _
6 g5 A5 O# E5 q3 w7 h Interface "int-br-eth1"
$ g6 M1 o9 [5 j& Y
- h. H# b& \8 L* l3 }$ j& N1 v% ~
+ `: \: z H- v2 |0 H8 a N3 ?* L9 n5 ]9 Z
type: patch
5 c) p& w" f* K8 p( T. }
. o3 w" u. z; Y4 [: P5 t1 Q' L$ C- 8 u' r% b7 }; p0 b
m3 |. }" s# P8 M4 k+ M- y options: {peer="phy-br-eth1"}</strong>
3 w% B% R% t& a
9 s) n9 c: y1 l( E6 ^
4 L$ Y% \( C, a1 f
- R% e1 o. A2 t8 f, B/ Q Port br-int) Q9 z9 H2 L) u. @4 b9 ^2 W
' `1 Q9 N3 u4 d7 Q' n) P
- - p0 u7 _9 }% \
, M0 r/ w6 z6 r: S. U
Interface br-int* L! h- a! ^& i( _
! Q& c. A B e; f! w
- 4 \* M( @ Z. C7 R# Y
2 c L8 x+ I" t/ \" v type: internal8 f" v9 g9 |! @/ r5 K/ j
0 }# b/ L4 C% h3 a6 s6 |* P3 v v- b: C% a$ I4 C: J' b; o5 S
& h: l/ u3 D4 ^/ S. b Z5 O8 e7 i7 S. Y( F' ]
Compute:
- k9 `5 F; I/ O5 B
$ ?; n* a' }9 {$ W8 J
6 G" |( N, a/ V% o, T0 q6 n: U5 V8 g# w
# V: q+ `% \+ d- . F0 ^; |4 }7 ^+ I( x: h/ `
- r% e, q9 r+ D% ]) s% E[root@compute ~]# ovs-vsctl show% o8 m; B E. g l
) H6 j7 B& r. {+ U8 z
) [5 h8 c2 j* h1 {0 e7 L6 g/ M* x! Y. k+ d) J. u
4c606c42-312d-491a-bf9f-04e558508115/ I/ J$ E6 u; u# J/ J* u
( R3 B' R9 J, u2 F7 T3 J; M, O/ Q
8 K! P7 }6 i- H5 H# s7 f6 J
: U7 q/ m; ^4 ^<strong> Bridge "br-eth1"
]- C# {4 m( w2 {
# _. t3 h5 _4 I
3 [( F0 Y# J2 y" `. ?
9 v" O7 V) @8 ^6 n Port "br-eth1"3 N' @: e' |' @1 Q8 L
^0 t" ]; X% d6 Z
1 i; w& {$ @; L; Y/ Q' F" ]8 M
3 [/ Q M- T; J/ N2 K' o Interface "br-eth1"
# J' F8 A# Z3 V: S; Z5 \6 L$ u/ D) ] b
0 S! C I% Q7 {- s; |9 \( p( w# V" I5 `. H: I l; z+ q' Z
type: internal
* k1 Y. n, `$ H7 Y( C! x/ `. I6 T
! y9 ~9 D! j/ v4 W' v
8 f" o, ]- e6 E3 h/ b: T
9 }8 D# Q* U: b2 A3 z( D8 ~% H Port "phy-br-eth1"
* X7 b; x m8 h- w* t5 H7 A
2 l( o' g3 h' [
7 \1 n5 S5 z }! l+ a( J
9 C' y" j. \4 Q- j* n* Y Interface "phy-br-eth1"
8 Z3 l4 U8 J( W+ }% H0 h
' }3 v: O/ [" C. y- * x; U( [+ t5 H
N0 }0 y; _$ o3 B9 V/ B1 y& y0 ]
type: patch# x9 y U4 N( L6 _" K
, t7 k* E0 a! w8 m5 g+ A - ( n1 R! C! n9 p' |2 g
& y. [. e ?: F& K
options: {peer="int-br-eth1"}</strong>
- x9 d7 p9 b2 d" w) E0 U) ^% o' M" r2 D; \: |
- 6 [" b+ F. G4 F% q+ u
; ?, r) @/ I: ?
Port "eth1"4 l0 l C4 ^7 W
; h! K- v- ?) o6 o/ K2 c
* {2 _# w1 i, I8 v4 {! s2 H c' A4 z& _+ @& P
Interface "eth1"
3 {- C i" _2 H" D; {- z R$ R$ Y8 I* O( u9 D+ |/ k8 S
- 8 O' e9 o: b0 \$ t. f% s
9 G/ l, L, V$ `$ E* d Bridge br-int
- `+ K U0 i/ e" g
2 F" ^. P3 O5 @7 q; W; {( @* c - 7 u& ?0 U1 Q% V0 m4 i. w
9 C* z! F% F- S6 \9 F fail_mode: secure& [8 o- H4 }5 q" k/ b) G( G
( w" t! Y8 T# c) t" V* v5 a; p* O
5 n. a# y4 O/ U# r# u( v7 k; [$ u* L) X6 {7 J+ B1 r
Port br-int' N+ p* c6 d* e4 W: k
- b4 i" f7 K6 }# Z8 x: E' D
- \8 c$ {* }7 O r+ U0 ^5 t
" ?7 h- L8 o8 B Interface br-int
0 J/ G9 `7 p/ p3 _# E7 U
; H6 A* A" ?+ i# @
2 ^+ P: Z7 K+ n+ T( C7 C8 [& G9 z: M& P0 H
type: internal
6 O2 z6 q/ a0 H) }4 i2 W9 a6 l/ b7 S( X, A, N, ]7 r. q
- , @& \- s( r% k- i- K) f# |
1 t& X7 l9 \+ g* X2 h5 D9 Q0 }, k<strong> Port "int-br-eth1"
7 r7 _: n5 e I8 ]! d4 C+ U9 c+ m
6 ~* r* c% { r2 U' L7 ` - % Y7 ~) x+ M+ f) [0 ^. \
/ c6 @; k" ^9 I6 J1 b5 _) w
Interface "int-br-eth1"1 f, p, J5 S7 b9 ^- ]9 D( z* f
; E1 @3 y) I8 _* |5 r( j
9 _- y' S+ o1 X$ E3 [
( J# d+ B4 u2 E& _( R) L type: patch
% y; @0 `, j. h% ^8 C) N `+ c" j- N. A( `+ ?& T2 n: ^
- ) g* Z; p# k0 W4 e! F+ @! r0 J
0 w- Z+ n( ?8 ^2 B. F options: {peer="phy-br-eth1"}</strong>
6 j- c8 [- E8 K3 z! t" I* q( i: W0 g3 }( x9 B
- # u% e4 q3 U0 V# @
1 B0 f9 o) @7 D Bridge br-tun
% O. b5 L* F, x% v6 R
$ S8 I* h7 z5 Q
7 H5 ^1 {6 \( Z9 S9 z# @
- S' x$ P2 i; ^' j! a' D2 W fail_mode: secure; M& P$ L& s0 f# a/ @; t: [5 E1 J
8 e- v. Z* g7 z# d" q f- ; P. O" r) u6 W# ^1 M# l9 G1 ?
4 l2 j+ ~3 o0 Y% l* C
Port br-tun8 _7 y4 Q. p/ `) B1 f/ A
8 M$ n$ ^# T$ t; v6 a9 s) J
- v1 o O! x8 {) a$ a: |7 G, c
) T3 @9 p/ b% d# P
Interface br-tun
k. U# m' N, e S5 r- K- ]: A/ O8 Q% \: u; |1 u. R
' L& n" N* Q4 Z8 {
. l' Q2 R& i1 e% w type: internal) [+ `$ k% k4 s9 b4 b
W2 X1 L% c" m/ |% \
J# Z6 I% a$ H% b4 j( t# y- e; `# t* n2 K
Port patch-int
) Z/ Y5 v" c; A5 K% {/ q5 e1 |; A7 d) z8 l9 P$ @) A% u' s3 C
' M; }) I; _4 r# c, i
! O$ Z6 d [9 y. s2 L: w Interface patch-int' V: x9 t4 F, W# T; A1 B; W$ B
' h: I& ~9 _! g5 i9 T/ q) T# i
9 h+ }4 l1 w3 F0 C$ [& |1 p' z; \) W& z
type: patch# I) S2 j8 L- e* Y
( u( L2 F" ? F( ^# X1 g+ _- ( f+ {( |4 z( ]- O1 f
" I! C0 a& `6 d, A; i3 z options: {peer=patch-tun} d" m: G+ A4 L5 G) {8 ~0 O
! n" b, C: _- o5 e) R - ( L; k, Q0 U9 |! M
: a& h9 e# l" Q- z! J& I ovs_version: "2.3.0"* G Q' u& a2 D
1 ] ]" w' _* |) ]5 M# |
% ~7 w6 D$ t* p+ N, x9 R3 x
& a; r) m$ W; F% w) ~) N. ~7 z5 N( A" b# |
br-eth1向下连接物理网卡eth1,br-int向上连接vm,它们之间通过phy-br-eth1和int-br-eth1这对pair交换数据包。$ _! u2 L! @4 D) @- a
$ d7 L- `8 r, C/ `$ S1 G
: G" }3 {/ {# M) ]" N
) K0 O( s8 q; x+ {( T1 ~; e' _- ]- Z. d9 |' h
2. Openvswitch 中vlan的配置
* k" A5 ^! p/ B, W/ k; w3 O5 k; Y( D( K2 O
, j% Q* [5 X) }* _+ L
( ?6 K2 ~& a" E9 L$ e/ D0 L9 j+ e- 3 P) y( J4 k) t+ J
, F+ }8 `9 r% ]) H* |6 g
cat /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini5 Y3 }( j, L: E( [6 H8 ^
/ @3 ~' O" u, C% A, z' \" p
# T% j4 B; A2 P% A; P2 `% @: ~( v/ q& U
. W* B6 s, |8 Y' y
$ _' j; M3 o5 X+ J0 d. F$ @
" U# g& _# h0 I0 K+ V7 q0 m9 i+ ~
& E/ H9 E+ h p, ?- % H4 c3 R* i" ?: A
- H5 p, ?7 R( A* h[OVS]& r1 y) R$ w! W3 `: e, C l3 A
1 @, o1 K; F* M9 Q X+ c - * Q7 t' l# v( i
$ l9 o/ N6 A( o, V) B& {tenant_network_type = vlan
* H# G0 N4 [1 a+ w' ^; h U# ]6 U
2 R" K/ }( d+ W' T
, \- Z4 i0 |* }; v( v3 u B" i
l4 k9 `7 C3 I8 Bnetwork_vlan_ranges = physnet1:101:1109 U; P0 e* A3 S5 `$ f+ M
/ b& T5 S; v# g) ~6 x8 L- ( T. s/ X7 s0 ~. {+ w# ~5 P
" U) ]5 v) m( W8 `( S- G5 m
enable_tunneling = False8 j( u* W- X7 m+ E5 y2 U4 p
- A# l# b$ L* _# {' c
! J1 n8 R6 e/ @4 N8 n4 x
J3 c7 U" {! R2 O$ }: Pintegration_bridge = br-int5 q5 m0 _0 ^( e$ j: J1 @5 I3 k
# Q4 F- Y* }; T: X
`- ~- K/ h4 T! [; M& F8 r
* B, Q7 }! X7 ^ Z( L* Ltunnel_bridge = br-tun
; A$ i% [- w A8 t* V; o+ o$ c/ L
$ A4 d1 a0 O1 T- ]- / A/ m+ m& L+ J( t& {( d2 n
$ r' i3 X- H; `1 a6 L0 elocal_ip = 10.14.0.102
9 G7 {. C k6 n( \6 i; Y0 h9 p
9 d4 T& H" C& v$ X, r: Q
4 a0 Q I* t- K2 ], o7 r, ?0 v% c, c6 ~ F7 l; c" {
bridge_mappings = physnet1:br-eth1
, p$ P u' k8 G5 ?# A1 l3 l
6 A5 g4 l p2 \- g0 ?8 b* a
/ V/ q; i. z/ B, `$ U% R: [5 Q% [; X0 A# ]7 |7 b. F% T O. m" k0 O. ~
........" ?5 I4 [* }% r$ Y3 M
3 e1 \7 Y1 c- d2 r
. w1 p n7 e% z! O- F + r" {7 v! V& H8 t% E* U$ q; u. ^
4 L O# y- l* y y重启服务:( J( W1 ]/ c3 l9 W- y5 O" I7 A
. n3 ]' |2 v; f+ `+ q6 L5 n6 X
. l' K N$ E7 D0 L/ P& e4 L
v% F( |. _' a# c2 s
4 _6 C4 y+ j* ^% ?& ~systemctl restart neutron-openvswitch-agent.service
: H3 V0 z8 S/ c5 E+ a
8 e% H* D0 t0 Z* m3 Q, I, P3. 创建vlan网络! z) s" E- z0 [
$ K/ U( K1 t8 T
5 G# J( L& y- O/ N3 V$ z
创建网络vlan104,指定使用vlan id104(segmentation_id) :
* R7 K; y$ W2 A
6 H# h5 s% J; C- K- d( R, ?: t& Y3 k K- z* u7 g
0 e1 O) ^( H& ?7 T& [neutron net-create --tenant-id 225e734464134ea79139eb6b0b202095 vlan104 --provider:network_type vlan --provider:physical_network physnet1 --provider:segmentation_id 104
5 v: u9 k' Y( j% R& V- k3 `7 u: m& P2 q s
+ ~9 \; g% J( B C+ h% K! ~" w) z8 P
3 w" T) N; x7 `5 h; {3 X( X- G" z( m, n% P( M2 m
- $ b c. `3 r# M, e
5 E d4 _; B( l, @5 T' z* h, o4 l
[root@controller ~]# neutron net-show vlan104! C( h6 v! f3 O% F7 q6 j- C
6 c/ H9 x ?) a5 t& c- e) L8 M
( O$ A; n2 v% ~- [: Y; _
% e5 h( {' Y* g6 a T# y+---------------------------+--------------------------------------+7 P( a" ^. D' _0 |! E$ c4 L
3 i0 `) w, Z# M* J# I
' ]. ]' q- u/ b. {& [. e4 f
5 c+ c+ `% ^! X; Q* S3 R. n- C% i| Field | Value |
' p9 W, g$ r/ y7 v% f$ f2 M& \! y* V: y) V6 |
- % q5 y: I" t$ D0 @+ b1 O& [/ }
% D! F# c! h. b+ P/ @5 g' g+ j2 {+---------------------------+--------------------------------------+; d, f, t4 Z+ X' m8 n/ }0 p" z4 U: J
% [$ X! I* H1 c+ Z5 I5 ~7 @ - 5 h4 w7 l& o8 K) h
! J: X& ~* g. K- j, z
| admin_state_up | True |7 J" J* f g! c
: E" _8 E- [6 H* c3 i# O/ u6 l
9 @* N" ~8 e& f5 e% i; q2 X' i
| id | 239d0325-5a65-4369-a05c-635b583f6185 |7 t! w, y0 A6 u( A, v8 z" \
$ N0 d0 ?2 k' c. D, f
" Q! \! ?7 z9 J( V, ]0 x+ M% U# P& ?: B5 V& x8 x
| mtu | 0 |
5 d: o7 m! c' t, o: \
, a" w: C: K9 B9 u5 B
. m/ n! x# D3 `& x4 @; \" T1 W5 W- r2 K, T. h
| name | vlan104 |
3 F( s* j( U+ M" G2 J n) d f. [7 M9 `5 s; ~& F
- ! K) A9 @4 t# m; u
7 |6 J( F0 F4 D" w; _ P) V0 W| provider:network_type | vlan |
' b( D! X1 Z O6 N9 `0 u/ u7 b" d) V) q# `
- 8 r7 ]; I: b$ c5 W7 }
% B0 Q$ W- G% j+ T| provider:physical_network | physnet1 |
9 ? n B1 H) q$ ~* g* O
' A6 [5 {1 P1 s% p. l: N7 K+ s
, l; [' q8 o. k( W
' p& {3 C" Y' [3 B4 @8 \4 W| provider:segmentation_id | 104 |
& k! _+ W3 j8 j5 v; M& Q# i" M8 S4 M+ v7 }& U* H
- " ^7 _7 \# g C0 t1 O, q5 j5 e
# c; m. T+ @# D# o: D r% y| router:external | False |
& I$ |2 R% g8 q# S* k8 S- J
' q2 S2 {+ i0 } - 0 p9 c8 I- N' @- ]6 J4 E" K1 k
* q/ ~& R$ r V8 w2 }3 P" `/ A| shared | False |, M4 i) P, g! U6 d# _6 Q; t% d3 u4 Z1 z
2 l/ |+ R0 D6 z
( {) d. p9 @ }3 s9 l: b5 N8 V1 [; c- A5 _2 ]5 n; a' `" E* H
| status | ACTIVE |
: r: ~0 R8 o6 y, A
5 L4 S! E4 j' L9 i9 A7 Q: {- ( {9 C9 s! p8 K/ t
! u5 u6 b _+ G7 e" D$ x9 m| subnets | 76f8bbb0-52af-4b43-a246-679d6c456956 | ]* h; V$ Y* V' `, {: \! M# o
9 d0 q: T! C0 I - 3 F( J% o. c) \! m/ ~3 o9 M
% m! J* h( T" q% c( {' `1 v
| tenant_id | 225e734464134ea79139eb6b0b202095 |8 b) d! }5 l! @' d
" v3 A: {8 ?+ f; h2 M
- ! {8 D a2 C, f+ x7 Z" E% X
$ I/ @( O( Q8 ^
+---------------------------+--------------------------------------+; \% @* k0 y0 H _) V
; k8 d/ D( [& M! s/ ~
G, H, x4 t! v' x: D
. ]2 W4 U$ I% J: C6 u( l5 m- a2 k2 N# p5 l
为vlan104创建子网192.168.104.0/24: [ ^9 P: O8 s! }5 s1 `, N
" Q3 h. J- D6 s- v% T
$ q1 W( u- l! p" sneutron subnet-create --tenant-id 225e734464134ea79139eb6b0b202095 vlan104 192.168.104.0/24 --gateway 192.168.104.1
: S# C& }: a7 e) O1 p5 l+ |. G5 b4 R/ s) F! p' e
+ p% I- n3 w+ n5 r1 u6 P, d8 H. E" N! u
neutron subnet-show 76f8bbb0-52af-4b43-a246-679d6c4569562 n D* Z6 i( Z6 Y- [
7 a( V1 ]& {' w; {' r! [+ n0 I
+ Y2 C% A' l+ @8 \3 X6 C4 E& w
* j0 u7 \# q! r W# W+-------------------+------------------------------------------------------+& x+ o. _$ l) V4 n
' c8 L5 k, c! H) j0 x# {
- " o6 d: M$ K2 b4 J T
+ @, {" U; v2 N| Field | Value |
0 ?( p$ I( f3 x2 B$ O6 z6 Y- x
9 j1 F! Y* \* I6 B( v - 6 [' Q ~2 H& _/ }+ a7 `
9 T. O4 }" D t' U6 O0 T* s! Q2 S. d+-------------------+------------------------------------------------------+! X2 X* s+ B; r* x% y$ e( u8 o
$ D* L7 S6 W' d4 \ C
- `/ Z2 s' u: H" W7 b# ^
* P! Q: p+ q% T9 ^$ M| allocation_pools | {"start": "192.168.104.2", "end": "192.168.104.254"} |
: s, k; F" B" Z$ ]! o. a
& S) ~2 a, \) G( w9 y1 D# [4 b- # f+ E, V$ W& U1 U6 g
. b9 m5 W) t6 k8 k# f& U) s
| cidr | 192.168.104.0/24 |* C7 Q. V2 P( T
9 G: W- b: l. z
- ( r* y* e/ s/ X. j' t
9 z0 Q; ]+ [/ c, a8 Y7 \; N& w5 a
| dns_nameservers | |
, V( B" ~- {; ^+ R% w. i
3 \$ T! a" g l& N - 3 i* _8 H$ H# N) O6 F' O0 g5 U* W
; `1 E- W5 S! s
| enable_dhcp | True |
( m+ e6 y# j, |7 C! K. B" u i: M2 l% n& O- L. W
# O+ q% f4 T" R0 X: G
& Z/ D! ?" @" d" l| gateway_ip | 192.168.104.1 |
& q- u- t( o; C( d* g6 O9 t* Q' Z! S3 r
5 R! O+ j& _8 \7 d0 ?: h. n6 e& T1 n# V
| host_routes | |
: G8 L( ]( h. k
9 K3 N5 ^6 }2 a4 k5 s
( N6 N: n3 _ X6 E
9 u7 l v8 R N3 x. r+ s; W) P| id | 76f8bbb0-52af-4b43-a246-679d6c456956 |
1 y( U% p3 G o v9 g5 q$ o7 H7 l8 _2 \* v' Q
/ C* [- ~* L8 X( \0 Q2 }4 K0 P5 _* j( G% m# L7 S% i* ` `
| ip_version | 4 |; f: i, B$ @" Z- n. V+ i8 Y& W
- F; ?, p$ |1 c! j* P9 u
4 ~( ~1 m, p5 x1 |& K5 G
6 T# x5 R+ w }" u' J m9 j9 p8 U| ipv6_address_mode | |
2 f# |7 F. R' L5 W! a# ?8 n
& s3 ~- c* P% k- z' t0 D ]3 I6 n& y, Y
- Q- R) s% U2 J9 C; i* \. `
| ipv6_ra_mode | |* o" t3 X+ w, z! b/ `5 }. d" B; V
; `5 I3 R6 G7 R T s2 `' J - & G% B# ~3 x# b* }
2 c# P4 U2 i6 I$ Y| name | |
2 u W" @ B* U. l$ t' _5 t5 N) E6 o+ H; Q3 G, f* i
- . V4 E) ]0 i! l9 Z- |' k
5 }2 E; d, s7 h4 S5 y
| network_id | 239d0325-5a65-4369-a05c-635b583f6185 |& A; K4 k w/ n: ?) c5 P/ N
! M8 i/ k1 ^5 t' I+ d/ ?( W- E - ( i" ~8 |6 N! _( _
; P. r, j& }( n+ P8 N
| subnetpool_id | |% D) z, g" c* V8 `& ?" j
! _6 M2 a5 N" K" ^0 H
8 d+ ~; Y8 ^; z
{, o$ r1 L+ S: H- G1 B* V| tenant_id | 225e734464134ea79139eb6b0b202095 |9 |/ j4 f( k$ w
9 I5 C" S, R! w2 ^7 q3 p$ k
. ]: Y4 T% |8 f, \0 {# y7 O% P6 _, J5 c( c& U
+-------------------+------------------------------------------------------+
# g9 y9 P, X8 E3 f9 P* `7 |0 e7 P. j- U, O: C
% _# b3 v9 S# R4 v 5 V% d# P0 g6 m( O' R( H
6 B* A1 W. N4 W6 b- y! [, T) _创建vm:
; O) B/ W/ I% J. n
9 n$ T0 }' N3 t0 j9 v1 C! s- u2 v' B; Z2 v3 G- `5 p l; B" r
" \) g* P5 h6 G" V$ i
! J2 {1 I0 U% y9 E! J8 @, v( U
nova boot test3 --flavor 3 --image 421f61ad-14d6-4054-a4bf-883fc1413834 --nic net-id=239d0325-5a65-4369-a05c-635b583f61851 P& X' b: F+ g- t# T! `4 n
( V( q3 }0 M! r! q; W
- ! E. T& b0 c+ ~. C# ?- [, ?7 L
9 y' ~* a+ E& M7 W- w N
nova show test3
* W! Z9 v. l) Q# @5 l. b
! N: Q/ x/ s2 L
# U" V% n7 e, P7 P$ V1 C! b$ z* N P7 D
+--------------------------------------+----------------------------------------------------------+/ q! z2 o3 B( U6 Z o" o+ G
" _% j% [( P3 m2 z9 [
5 F6 ^# p( |; E T7 |# p( [. w; I1 {- V: ]% W1 q' e: B- L8 i1 f
| Property | Value |
6 l* ?& F. H% v8 @, ^9 A! x, Q: ~( |; [: @
( E5 E0 K+ B& J% E. G! K
" x/ l% g4 r+ [7 l1 E& }+--------------------------------------+----------------------------------------------------------+
4 h7 d9 _# X. N& Z& n0 [7 G$ D3 f$ w; }$ e( p3 D! A
4 \, l- z0 W/ q O+ L" W4 B+ G* F' c+ y* w
| OS-DCF:diskConfig | MANUAL |8 O" a+ u+ R) ~2 _
, Q0 x* i. q% K& L
: M0 v' X2 x/ U ^
2 q0 ^+ r. b$ \5 M| OS-EXT-AZ:availability_zone | nova |( Q e! b9 g7 k- x$ W/ n0 J
2 g* C3 ^9 r& S& E
- * ?! o; `0 q( f/ J1 ?$ v: G
# Q4 c* f# o) g: [) e$ S
| OS-EXT-SRV-ATTR:host | compute.ibm.com |
9 v4 N- x. L$ F2 Y( @& E$ }) I4 e: T) Z5 G
- " j A$ V5 z" L* R, V
; h# x$ V9 ]5 s; k" F3 I& f
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute.ibm.com |
4 I3 X2 ]$ a: G- j+ B$ ]. Q' C/ Y- b
- ! l8 E8 j& o6 C% R' J9 I; d' n
4 u; B; R' j" Z: r R) c1 O7 Q X+ f' @| OS-EXT-SRV-ATTR:instance_name | instance-00000005 |
# f$ s0 t" \$ g4 ^) i
- @' f9 y# k! A1 ~ - % K& ]/ P$ M/ |( I8 F
$ @, j h+ {0 l) i
| OS-EXT-STS:power_state | 1 |
& b% K) k C8 H' E" U" P, Q" {, c+ W7 n! n) r( J1 d
- : r: r! Y$ ?8 N9 T0 \
1 U% L1 t$ l* A! V
| OS-EXT-STS:task_state | - |
# D- c! T3 H/ Z2 v& |8 X$ o4 k. L& q
- ) T7 t0 x# y8 e
5 I# v8 q1 N, m+ Y| OS-EXT-STS:vm_state | active |+ }+ k0 s/ |2 g% `8 B* A
" J& @. R2 n) b- ] - 7 h7 X3 _! L" n( h: O. L. Q
, K1 }. B2 |3 q' S- O3 e
| OS-SRV-USG:launched_at | 2015-08-25T02:40:51.000000 |9 s/ ~. A+ ?, n6 ]
. D& k$ a4 u% ?. a3 F& u$ M+ {8 O
F% Z( Y0 @* Y; [- \1 o
]/ z( D: Q8 y6 f+ `" m5 x2 y| OS-SRV-USG:terminated_at | - |
4 L$ t$ U. `+ ~0 B4 D7 F$ [/ \8 a4 A/ H" A0 W# w+ `0 i
- ) A( Q' q0 ?( l( X8 T+ G9 H
* T r; z/ K9 R& ?) p
| accessIPv4 | |& O$ h- ~. P% n0 \3 h) j j
2 b# w( ^2 d9 u4 T- A
0 h; B, i, y3 r# F3 j$ ]- Y& k! n4 a7 H0 z j! i& c+ {
| accessIPv6 | |3 c; I9 l$ F8 S: A
- @7 |3 M* u' q$ m- |0 r- $ `3 ?( v& `* M' |
# _6 _( {7 n& y7 c& I1 _
| config_drive | |
& F7 x' }0 l$ @% H& H6 W! M: B% O
4 J# F+ U- I0 ~9 l
, m2 t4 m) @" ]| created | 2015-08-25T02:42:49Z |, g7 G# o1 f7 { n `
4 L: ^7 } O. t {; F ?
5 q! N0 t1 T1 F
% `+ S4 H" Y5 P: x8 i: g| flavor | m1.medium (3) |
* {, M+ |5 _1 L& c# O. @; P) j
2 n2 h' R) ^6 s# A* G) o- % m: q0 |# E( `6 ^2 l1 q( x
J! w9 X0 s+ k4 y- o$ l& y| hostId | ac73486a08f6bc6d7333faf316cec310ed2b19da6a55adffcedc30cd |
4 V7 Z' `- W" ` W6 \0 B! N& O8 o
* ? @6 c( W& p! M- D* X/ c8 A
, G9 w0 @! D" k j! M _( A
2 j# ^& L8 F- I9 {| id | 4f91f8ca-78c0-49e6-a8be-96ab827315b7 |2 Y, I0 w) @/ _! `! h ~
+ k* `3 H% N( \5 {; H
- + B- C0 o$ C1 X8 R0 W$ V
/ j9 U: P: A* _| image | rhel65 (421f61ad-14d6-4054-a4bf-883fc1413834) |
: A" |; d4 U8 }) X) G: c5 v0 z$ x1 a9 _
- " |( o( U( \' n6 m. P
/ f" a) j* ]9 L2 _7 @2 E
| key_name | - | s0 V0 q- ^+ |# @0 M/ e0 N5 e
; c. e3 f6 L) f& i) B8 o# ?% a
- 1 Y' n: V7 Y) O/ ]
: v1 G5 B& b* {2 o! j: I| metadata | {} |8 p4 u s! n" D' S, W7 C
% L" N. K$ \. y
% S- }) y! |% r! l/ B
1 _. h3 T0 w8 e1 T n3 w7 Y3 P| name | test3 |" a# p3 w( N: d: X% s) a( j1 N6 W
0 _. j7 o9 H: j; H N# |
- * y( y, c+ {( v' u: S
3 K6 j- P8 u; d| os-extended-volumes:volumes_attached | [] |; X# [/ b: F2 ], Q: `, j
- J/ @) y" g6 ]3 l- }6 O
. T$ Q& c" d" G8 \- H6 a) I# C! g: K6 u3 c; W4 Q
| progress | 0 |) \! g* h3 m, {7 ^7 R. U- N
/ W4 F) x M; @0 R4 y2 r ]
- ! n: f4 z% b; s2 x/ V1 n+ t! O' h
2 D; p3 W r4 O
| security_groups | default |) p+ U/ R$ O" }6 U
4 a) V a; k) a' h) P3 s E# Q! |3 M' J
% B. M' \8 ] n2 i7 l* j g
& D3 h, @& x" V# y, l7 h| status | ACTIVE |& R0 h2 O; z. Y5 l& W. {+ J7 i3 e
) k* t4 n; E* b- `# o6 o" O# @- ( @: n7 a+ U( j' G! {
, i+ a2 I) h" A+ g; d6 _| tenant_id | 225e734464134ea79139eb6b0b202095 |
0 u' y+ `) V3 O- J" s( n) n, E
% Z5 A. o' R, n$ U4 f( y- C - 2 E" ?% a) n8 I( |4 Q- x$ n
2 Q* `7 ^: e8 O+ m$ E) e) Z- @
| updated | 2015-08-25T02:42:57Z |. |1 n! k' }( [7 C
' j) H/ _* O! `; U! ^7 s) Z L/ a
1 w! ^. n7 z9 F* p
$ o% J: c) L7 d7 [( q; w| user_id | 6454c58d76c245c194d744cdb245a0a8 |* K8 p. S1 |" t
, B0 w+ Q4 c3 B1 `" c, e, r
- 3 w& Z9 l, ]7 p$ G) Q9 U
+ z; q/ o* \3 M! d" T R
| vlan104 network | 192.168.104.5 |
$ a% r2 ] C! b0 [8 B4 C2 ?$ w+ E. l9 r% y- F' b1 P- o
! w. e B' q9 g' F3 d) s# q" R( A t% A/ r% B% t. {
+--------------------------------------+----------------------------------------------------------+% v" d: f" L O+ W, @6 z
, ^ W2 t2 |( ^9 a8 u
1 y/ k9 S# |7 d A9 |0 x
0 S2 h% c( z2 n8 ^% T# R3 d8 J$ `+ ^2 y! V
使用vnc查看vm:
L6 K/ H3 W# X9 s1 L" P8 N5 g9 Q& d: ^6 t- C
* C/ Q3 w+ `! g, C6 T & @0 g+ y2 M- M
% L3 q9 f% L$ {% S7 j- K% w: a/ A2 J1 ?% { `" n& y/ w$ H" U
$ H/ M* b: Q. }. ~5 B! C( w* Z9 y% M/ g& j8 J0 J
四. 抓包3 {: b( f6 S, l2 P
) E7 _* n8 P; j$ R7 }, [! K5 X
1. Compute 端+ V* v, r& v! q7 A/ u, x& i
! S4 Y9 M/ Z7 \3 s 8 ]* o- j5 w, t9 u% s! S
& P& e5 L4 q. s( }9 b
/ k+ T! h& p. {1 `3 C# m2.Juniper端- B( s# k' c% D& s
) Z- n2 H8 x: S, z
6 o j* h: n' l: t6 O$ O. a
8 U& j! b3 _+ M0 ^
! n6 J, L% r. |, v* f1 }0 f3.Controller端:. _0 L- m( h0 m' I
S3 O9 j$ {5 |2 x x9 F
[img][/img] 7 `7 }* A& R9 ^* t! ?# R+ u
# L, s# u5 U# U( H# N8 d* y |
|