|
|
一. 安装环境:
1 M( I( U9 t9 o! O; p5 n! q$ |4 Y& m. Q! }
1. Juniper EX2200 Switch
. x u3 q+ O* y+ ]7 N
8 D# \4 v! Q# ?% O2. RHEL7.1 OS) Y% y* ?1 X% ^& P6 d3 \9 \2 ?
. A" u* w$ ?4 H- f8 ?! N3. 1Controller +1Compute! q9 T" p8 \0 _+ X% ^
9 @8 b3 I& q: ]( d" r+ j6 [ R }$ k ^5 `: t% i
# v! z' ~! _8 Z- i
管理网:eth0 10.14.0.0/226 R J; _ B( r6 k
8 o/ ~2 r9 b$ ^# j$ h; Y* h
数据网:eth1 使用vlan-id 104创建网络192.168.104.0/24,使用vlan-id 105创建网络192.168.105.0/24
* e4 Q5 @( \: h$ E
& F" {$ V% s9 Q! ?4 E0 r" }3 @' D$ t" ~; u/ `
& {6 z* m3 l4 ?: p' h: M1 o4 k
Controller eth1 连接Juniper port ge-0/0/27.0,Compute eth1连接Juniper port ge-0/0/28.0
( F7 D' x) T! b7 a. @! k/ w b+ d. H1 E f! ^+ U' G& k
: H- H) e# M% `) N1 @' v
# N' c; A- _2 W u* |. m" `二. 交换机配置
* J2 c% ]9 d! S$ y6 N ^/ v2 k
* l4 y R" S% ~5 V& ?1. 设置port为trunk模式:0 {/ t0 Q# C) x; J9 P1 {& U
! u0 Z. i& y" z2 v2 ], s4 K
. [! i3 K& a+ F _) _5 Z. {& J9 E: J# i+ s4 C3 d
- $ O9 h9 t4 q( c; F0 l" f+ S
' V1 c1 ^7 I9 M9 @: w0 I; F
root> show configuration |display set; e/ v2 [' m3 ^3 J+ V# E
# ?& L# j& K) q8 F& O! ~ d
4 u) L* r$ U. i$ {/ K3 b
; z2 V9 u6 y9 x5 T% Hset version 11.4R1.6
8 G+ p9 H1 d+ t5 ^8 o/ F w; Q+ b& @1 o: C
- % |0 @4 z- R7 E) M' |
3 ]& ]: v. t8 pset system root-authentication encrypted-password "$1$jC9l6Bwy$ePVIDxgH.8iXqDfqrLzuJ1"
$ M& @' }" Z" L2 |# ?+ m5 F' s4 r# ]( x
- , c. u+ O# ]' o" Y8 \
- H l& w8 g3 T/ y0 O2 h3 x- B( T! {
set system services ssh
6 a7 u) b) l4 o0 e9 S' F9 m! R* \; e% C' X) \- S0 u
# x5 E& N$ ]/ C1 m2 ]6 n+ \- f' Z3 E1 s; q6 }3 `1 q, ~
set system syslog user * any emergency/ W# O0 k, [/ X
& K# g' I1 q& p- |$ z- * d1 \$ r4 P9 g8 e$ l
$ @. p7 v/ ?/ \! k% R2 g- Iset system syslog file messages any notice/ q& E( Z- W$ w
5 t. K1 n; c1 V" ]5 B1 P% f# y - - V( q% o r h1 z2 F9 }
9 |( v# ]$ L! J Z6 xset system syslog file messages authorization info
) U7 I# U% Z4 Z# F8 H9 X( N2 X% S
- `, D- P6 v9 U0 `) L- K2 ~0 X) [; e/ r! h
set system syslog file interactive-commands interactive-commands any
" m9 }6 z! c! _2 p$ ]$ B
* t: d) @ x6 Y
- z. {6 C3 G/ X7 r* ^: d, `2 b+ Q
+ V" f" t+ p$ H2 } A Nset interfaces ge-0/0/0 unit 0 family ethernet-switching
% i& b s9 S" U
0 J% G# |2 i, \! A2 c+ m+ _- 3 w' @8 s8 _* Q
, B2 r5 F! K' `: yset interfaces ge-0/0/1 unit 0 family ethernet-switching; S; p$ ]% W6 O$ K+ h% V
K7 ?* k* R- r! y4 Q7 k A, P: r+ Y6 J
8 v1 P+ I- z, R, e6 y/ B, G
7 u* X- I x' Z0 e w N8 cset interfaces ge-0/0/2 unit 0 family ethernet-switching
5 V/ |9 ]4 k8 r
% b, z2 v$ A$ M6 \1 Y( n% H
% Q3 O! j) ^9 X& {9 T$ s0 P2 W5 W, _+ a- D8 m. p
set interfaces ge-0/0/3 unit 0 family ethernet-switching: B2 M+ p" Y0 w5 G) n
c4 o. O3 f" O' G% s
0 T1 T0 _% }/ d4 O- I- r2 @0 n2 ~7 Z* C1 A, u: l
...............
' c2 h8 [( I3 ^( o# I
+ I& f. n0 C5 S3 N0 d- ; N5 ?& y5 X4 ?9 |6 U
8 s4 u4 d6 |! _8 {) c
<strong>set interfaces ge-0/0/27 unit 0 family ethernet-switching port-mode trunk
5 N8 f8 I* @5 |2 A9 `5 R3 Q+ [ R3 R6 ^$ y D
2 ^1 T, C9 P1 i6 B) e
" \' I# |3 ^7 }5 H( j% Kset interfaces ge-0/0/27 unit 0 family ethernet-switching vlan members all0 o I1 X9 t- V" `
2 S9 Y5 H2 q! |5 {
- 2 l% m" p6 X7 r
5 P# Q8 g1 Y# P+ B; a
set interfaces ge-0/0/28 unit 0 family ethernet-switching port-mode trunk
3 B- d+ c! Q4 V& ] A7 U- r
6 K" v# a; ^/ n5 ] - + L* I) J. v9 Z, ^. b( @
/ i3 C7 G; f1 v/ }4 K* j
set interfaces ge-0/0/28 unit 0 family ethernet-switching vlan members all</strong>
; Q$ X0 R4 R7 e8 X U& ]; I. E2 {% i A m; I) Q
- : J; S+ Y$ j# ]/ H9 O
* b0 J' F* Y u* }" k3 r0 t..................... ?* ]) M4 X+ h8 W- s0 K# D
8 ] [6 \1 Q5 s" [& _
( e; {1 a+ O6 V3 ~1 X * P! ^. N! W% R( N
2 J D. a5 P. h q1 R
0 D( w k8 C( |/ T
. D- Z3 f9 |0 m
2 Y% w2 z" J' y5 X m$ E" j
2. 设置vlan id 104,105
: Q) m! Q/ v& }9 x6 T$ L' |8 @! `( G$ B# V, {5 c9 Z M8 t- W( k+ w2 f
- M$ M1 a5 l, C7 o* z- D9 a0 k
& R! y2 W& y @. U, R6 ]4 k( f# ?3 j/ b7 n- O
<strong>set vlans vlan104 vlan-id 104! y2 [$ i5 ^7 B. V S- C' B9 K8 v# P
8 P, k/ @7 m( @+ W ]" j$ o
7 [9 N ]& h$ Y1 i, k3 m
9 X$ V0 ], e5 {( h" A% v* c/ I" Yset vlans vlan105 vlan-id 105</strong>
$ t& n+ A+ X h2 d+ P, c& U: Q b$ c9 D5 N1 t4 p5 {: g9 S/ w
, E. H( H2 S2 r. J8 B( {
- a% Z/ z" ?9 uset vlans vlan108 vlan-id 1088 }0 W S) b& o$ j1 }
- I, [# I( x6 f7 o q! s: x
1 r' e& w( i& ~3 {' H; N S+ r. f" _& Q# V# ?# M T- O: |
set vlans vlan110 vlan-id 110
& r. u8 T3 J5 M: H1 j- ~; m0 a. e4 M+ \; ?" n) c% G$ R# M
% c3 ?5 {" c" e" [" F v
# K# A: J6 E* `7 b9 O: e4 ?set vlans vlan110 interface ge-0/0/29.0( S/ B R: ?. g) e2 f) ~8 x/ n; i
9 f( ^+ D& _* W( V3 K" e- 4 ?9 `. J1 Q$ A3 I* B% P
* t$ i. b9 ]4 o: Z/ k. o; ~
set vlans vlan1117 r: D" D* n- T
/ I, d; [( l2 ]
- ' b; P2 \. s( W6 P
9 j, P; H( g( k7 r/ F- y1 X" m |$ R..........................
* J; ]7 F+ i3 M, y' X7 A# X& I$ B% E$ G- V2 ^: s3 m0 n
! ]- F Q8 {# t$ Y; f" Q* Y
e" T3 Y; [( [- u/ I$ M# i
9 y5 p8 P1 k$ P! z: c: p" G+ r9 K0 |4 n+ \
) L- x# L+ L- I2 D8 o [3 @% d
; r: I" {* [) P; ]" i# U' A- j" r( @$ s3 y: G5 A7 J
ps:trunk模式下端口对带tag报文的收发处理' A+ j4 |" t9 u' F
! b6 ]7 k: W+ u( D" @[img][/img]
# a' E1 u9 z( M2 |/ _3 n d+ r5 U
- M T6 N+ s2 ^& l7 e# l- }. E
9 s# _8 h( F5 v0 w2 C0 Q[img][/img]. E6 ~& e/ `# `8 N7 \
: S1 e+ h% F$ z" W0 O
, c/ z2 G. m5 C9 y0 ]8 K
. H) n) f/ X9 | b) B: G' w) l0 u: \+ N
三. Openstack中vlan的配置
3 p1 ?' |0 [- K
) h9 W! H3 D- v/ n% G1. 物理机上的网络配置9 j" ]6 |! j* N1 f. P$ w9 i
* M6 ?3 I3 y- v8 N
Controller:7 v' b) w6 D b) U: `9 L6 \" ^$ v
& q5 J b; P/ L1 t7 |/ \7 j& p. T9 {- g* f
& s. J+ A, ?9 m' n8 O
: [; C+ G' @6 M& g+ q0 \* m0 s
: U j. m2 L& F: v# Q- l" A6 |' x[root@controller ~]# ovs-vsctl show
, l: y7 H! H, u* }5 e& y( O/ T# g" O9 S$ E! z6 }# b# K
- & [* n; ~" B/ s [0 @# q
8 M- i2 z8 C) }f2db5bf5-6591-47f7-a134-462f66a3fde5- U d' ]7 K6 Z; e
: k. ]9 H$ M8 d - / P" ~5 ^ d; S6 s' u W( ]+ L4 ~
& T G* a$ d% I Bridge br-ex
8 |, T5 E }2 {5 Q7 M6 [0 T- g, K$ a' U4 D$ o
- 8 ?5 T" Q4 ?7 j1 X
0 F0 j# k* T7 }: m2 h. Z- l' B
Port br-ex. d! J- w1 X" {9 _
: R: {2 j; `; [7 d+ H/ S V
; g# q+ F/ b1 I. F! y# q! F8 ~* [6 T2 b" y* G M3 U3 b
Interface br-ex% X4 _* \1 u4 W% w
2 G( \" ]6 T* }' c+ W
Z( i% O7 z: i; W' o: p
2 z( z! U! ~" |/ \7 [9 p type: internal' L+ @9 u- y+ J6 B
. m5 h* Q& d# D/ V- O; t$ {0 {2 r
7 h( c# z3 [) c2 U: J& O2 V# C
Port "eth0"# ^ B. Q) Z! b! e
) v/ X) M- c E
) D$ M, O q, J& n! D
6 h1 b- o: U4 S: ]' n P/ D0 i: n* f Interface "eth0"' X% h& n8 t. z& r4 `
% d8 c9 D! X, A+ u, z6 H3 |
& g; k. g- K% w* G* a
t6 a9 r; [5 J' b: `. C Bridge br-tun
, u- U! v$ {- k1 I$ q# z, R9 [. l# P8 F- C" B$ D9 S. y
9 M; i7 W) s6 G5 H$ G5 k0 v6 z G' n9 S& E2 p" ]# @* |' H( l
fail_mode: secure
( Q {6 ~7 b, @1 W/ H0 y- a0 U* g! O% S1 B1 U+ Y
$ ^3 }' d* L+ s
8 d" g/ F( T- W! _ Port br-tun
6 s9 p- _6 S$ e9 O6 a0 K0 _+ @. q& l l; g
- ( P# `+ U% G* t/ W) e
8 Z1 I4 E# h* ^" N" i Interface br-tun
+ I D: o9 G' q8 v/ l0 a, t* n6 u/ ]" j; b" o
J% j/ r3 h m! u0 |8 x- K( h) `9 z+ j
type: internal
" k, a/ S! l+ c: n- W' N! L3 c$ O: y$ u9 X
- " f$ T' `% L7 W/ K) w6 q$ [
8 W8 F! |0 \+ D; k3 Z: n
Port patch-int' E7 {% w# { R5 {( i5 W
8 I7 h3 ^ [9 Y
+ Z& B y5 a+ c. q2 f# n& b6 y9 x- }) b) F9 H
Interface patch-int+ q k1 Z" v/ f3 O( J) l$ K
& {9 X+ C9 r- |5 P
; p* T+ V% ^7 l& [ O; i3 k8 U, z" w( K) A8 x5 S; D! i0 b3 t
type: patch m5 n3 A3 Y) I3 n1 x
9 D0 E# G2 x u1 W, T: r3 X
- & |3 p, j: M5 d8 b
! Z! D7 G& J. {' Y% {: P6 u+ Z* T: t
options: {peer=patch-tun}3 I+ r2 y! B& j" m8 O) I7 E& w. {
# F" @; F) a- P
- / `4 P& E* m, M; O' w
! C2 V! {. `6 u$ M# v: C$ r1 M<strong> Bridge "br-eth1"# o/ ^, H! E$ e
% k3 ~1 l0 s4 A8 P0 O0 `% N
4 k; F1 m$ V* D5 _( L% W
# o& w- T P: r) M3 w3 Q- h Port "eth1"
, m; s5 n% Q( t' P
a5 Z' t( ]& o9 t- 2 k3 L. ^5 B/ V U
Z+ p O: ]2 ?- C! e
Interface "eth1"</strong>
/ W4 k. k4 J2 G+ s0 i8 T3 L9 w2 F
" t; `' H: ?, i$ I8 e$ N3 F - " ? F. { ~; r# y
. c/ o8 E3 }9 z1 o g9 M0 d Port "br-eth1"# s' s; ]: t/ s( A6 S
: i3 l4 B6 f; y V+ S- u - / }+ K8 d9 @* J, ]; K9 u8 y- {: j
6 E. b- o# R6 B
Interface "br-eth1"
7 j+ Q3 E- X* G" o! B8 f% _! c0 m+ j( P! b) H5 `8 G) \
- - z7 P: {1 D4 c* R: J
! j# U t" K5 R* {# O type: internal
8 K& @# M M4 o8 X V. P5 a. Q5 H& y7 u: ]
, r* w4 l+ _, R2 ]
1 L4 o$ T0 {+ @9 t; x! W( n <strong> Port "phy-br-eth1"
p0 m6 |. E# Q7 ?; v, \
. O9 A& J. p0 u' F6 t5 e- 3 _5 N; x+ p6 V; [! t! U
" |7 L/ ^/ T9 r$ a7 B
Interface "phy-br-eth1"9 {5 X* A' s5 S: x
' o3 {/ d8 m6 S; L
7 w7 [" Q |5 i/ w+ A: p$ C5 l2 y# ]. u' O3 Y
type: patch
7 t3 j! s; g0 `/ n* K/ v' ^: b6 q
/ p' H7 [4 V% F9 n" P2 C1 f
' [$ Q, y% ?2 ^( Y options: {peer="int-br-eth1"}! v2 d6 i6 _, j; G$ T& g- H
! f' E% L5 l# e& v( |% y
- x4 S1 ~1 }. m7 m/ p& O6 Q8 g8 g; F
Bridge br-int
* X- S" a; g5 G/ d; a* A }( \$ V2 | I% f* V0 H' M4 D
- 3 R; ]% g' R9 t3 h
1 E1 l+ Z' n# @7 o fail_mode: secure
) n- Q; M8 ~$ G4 D; R! c* X( c+ R- z7 p+ A0 X
) }) g- I$ s- N; W! m" S N: A& h* u1 h5 ~2 G
Port "int-br-eth1"
/ x$ O7 e* Q1 @& [' f# m0 B5 I# p$ x0 F1 {, K0 K' x# ^7 W
& A V2 n! T9 S( Z1 L! g% l7 Z5 p! c
Interface "int-br-eth1"6 T: l3 A @* V1 w0 x. K" |" d
4 N) a$ p; q1 f8 O
/ e) `% b. f- ~, u
- Z9 Q$ @/ j& |1 r type: patch
0 k; a( o x2 |+ W) b `
6 g% [2 h% l$ j- C4 R3 W- ! e: N6 n0 q% m- x& t4 X
9 H9 b `$ M: ? options: {peer="phy-br-eth1"}</strong>. i9 J/ Y! j) |6 u7 A& a. h7 t- k
; x$ n6 t9 J2 j E+ A0 \+ W
0 b1 q3 R) f' k( R" P
" ^1 S! C j2 H$ h% W Port br-int! `9 v, G! v. C% j4 i
1 z, q5 ?2 G9 V1 u+ A
- 1 P& T! j5 U; |0 [6 [6 |$ ] P3 P
7 _0 [* h% T4 o! w" w Interface br-int- A5 v: X9 \7 i% i
# J$ G% a* S, s6 x5 g5 s+ I0 m1 C
8 p. \; M) _' t* q1 S n& _1 f0 K( F4 E% _
type: internal
' r* {5 ~+ u K2 J) ^) `0 T' X+ R0 V" S2 v
0 z. y/ `! u7 a$ l2 ` O 0 e0 J" b7 d& W, v2 Q- v
% ~8 J9 ^+ g" I% x+ `: w2 T, o$ L) WCompute:
8 J0 J5 Z5 c& T! m ]1 o- H1 Q& C( s
5 O5 q' |* X5 l x2 B/ y- a& J. N! E- f
: F- N& g i+ h6 W- 2 g0 r2 i( A. s, t$ a
4 [! G; x/ ]4 A[root@compute ~]# ovs-vsctl show
% [; X9 {1 W/ z: S
. j5 n- B. e" T7 D
2 N2 ]/ g2 U2 F# O, X
: @3 K$ A/ R3 d! k7 B" Y. f O! j4c606c42-312d-491a-bf9f-04e558508115% N" m2 f6 B/ Y* ^% U3 X! h1 F
6 d( ?, b: F# g1 T2 U# q
5 K) J; `2 R2 Y0 t" v- n
- E# t: c+ }/ b. x0 B, _; b<strong> Bridge "br-eth1"9 Y5 w1 |# N. e' y# S8 [
; @) U5 e$ Y! M4 D" u2 G+ M) ^
8 k. O- i+ y- @' k2 |3 W. L( x* S8 ]% Z4 T- ?, N
Port "br-eth1"
: ~$ N) y' J6 n Z! K5 l' a# c' J" d5 m; U t4 ^
; x1 g5 G( W+ T1 k' S
3 R( A; C6 p' X8 S9 |3 c Interface "br-eth1"+ D! F: Z6 i+ o& R7 D. [
+ I9 o7 @* v4 j! s4 _! N3 h
) D) W! G# T* J1 r) |2 _8 V
& F; a5 C# D2 E3 E type: internal# j" V( W Y1 [* S2 O
. n3 D! S. M) X
- 9 S m' T9 ^& _( O; y
% N% B* Q- t9 d, w
Port "phy-br-eth1"- B( B% P3 P8 |" Q3 f
" E& A) r& c6 T
- l. b) m J# e4 b( Z" E/ l1 ^$ O; `$ u* C
Interface "phy-br-eth1"
+ D# A$ C; @: _- |7 a. a0 w
8 C/ [' h. F5 v' b
$ A5 d) ]; _' N1 h. U5 T; q. X! r# {# |
type: patch9 h. Z5 W% T! _; l/ c6 J
8 q5 ^) \& v- Q* M5 ]1 ?- 0 O* S) E; Z. I2 r
# U! q- M; M8 z options: {peer="int-br-eth1"}</strong>6 j: D+ A, A- K" d, R
+ x0 ?# w# r) W* S
- ! v8 ^ y- Z6 u+ v; E
3 M( H' a9 C Q5 h
Port "eth1". L6 V$ I! v }0 h* \3 f- v4 `4 e: \
$ _1 u* U6 f1 x3 E/ V" P
- 5 V' {/ Z/ n8 k: B8 `: {( ^
' E4 M9 f+ e! \6 e4 i+ t0 [ Interface "eth1", Y/ C0 K( R7 L3 Y
9 f7 E- _2 b. T$ E2 a1 ]5 @
- 4 Q4 U! X, J( j, S; N
j' X/ F3 s3 Q6 l1 r/ G
Bridge br-int, Z/ F' j( J4 J4 O& C
+ D) M- X: k ^! I; z
% L @+ L9 G% N, o! _, `6 X0 H* s0 `: [7 e
fail_mode: secure* b# v; a% u4 Q# `5 L0 i. Z
" c# ~5 ^+ z/ r" G: h5 h7 @" a
- ' u7 I7 {! I# e# S
0 d: u6 y6 X; ] Port br-int
3 V$ Z @7 C9 e5 X! b( ], \1 g8 U3 ]( X5 J2 N! n8 ~1 F
# U8 x8 Z: Z- T: M- m/ }& @
& V# u5 p7 O( m2 O) U( @' ^+ \ Interface br-int4 |6 K& L+ G# Y% M, L2 \, a2 ]
, Y0 f3 S$ S/ w- 1 r3 [0 ^, W8 Y5 u/ g
# ~, T! I" T8 { type: internal
! s" b9 G1 a" t9 d' }9 A6 y; w* `
% ]" L& u: H5 B% x3 \, ?
, Z2 ?) ~3 W+ E' y- W" X2 E: c+ c5 D4 q
<strong> Port "int-br-eth1"
- l6 F, t+ D* U# {8 b, ~6 [4 A4 ~6 z/ W$ `, j! N; r
8 n y: n& |9 c; s. ?3 w+ r/ e: b Z2 x* D- S; ^0 n" }
Interface "int-br-eth1"4 r2 K" r. z6 K' W
! L5 J: K7 N% p: ~$ u q1 U8 a
; ?0 |: v/ r0 _3 s5 ]* r) K( x/ D: ^7 ~; Q6 j
type: patch# S0 K) Q) c$ ]6 o( \9 W
R2 w. m: z( M$ O( \) C# I; m
5 q9 }: j8 ~7 K% S
. e9 K( C3 e1 c4 U options: {peer="phy-br-eth1"}</strong>
3 p5 y. e& c( A. t8 ~. g9 ^ Y' ~! D% X3 G2 `
- 1 S x" k0 k9 b2 O
8 H( G& N9 ]( X2 I" z8 }
Bridge br-tun! G& ^( r# @5 ~! m
# {3 v& G- S8 Y: h. J! A! k
9 G3 U+ u3 V5 f# L: v" Z8 _/ |8 n# K( ?, b' V6 ^7 c8 d
fail_mode: secure
; ]/ v- C. T% }7 q$ Q
# e& `4 [" G! U
' T ?/ \( F5 O( k
7 X. L: c5 D; t9 b" s! ?- h Port br-tun; G' I2 S5 s% @" T4 x' S! o
0 ?+ v& v f* S- u+ a4 E: [
* f: q! M; a4 T
7 a) z; h! Z0 y3 ` ~ Interface br-tun
9 b' F# U, o. a% G2 K3 \( _" W; ]
' Z7 d8 Z' h' H' T0 d- 5 n9 m1 e1 [! O
# V2 P( G4 H5 Q1 ? type: internal
, W; o: E7 H- T' p, p, W6 r' H$ e1 V7 L* e E8 o
# z) H0 g5 `% O4 W6 C% c. A
8 J2 m: u# A9 ~9 R) J4 `. @. H) Q Port patch-int
! v# q4 n) L& g3 ?( p5 ?: b0 X* {
T5 W! B `( O) ]( r' ]4 D) y0 r$ `
Interface patch-int, k6 Z: X/ q3 Z& u6 W' l
2 d9 \. K* q2 @# x& y/ P) [
" {$ B/ ]( P$ {, m
. m5 }( h- _& ?. t# W' m type: patch
7 L. D4 R) k, |1 P3 A
4 J1 h5 a( X' L C0 z- . j; H- B7 j) ?1 [& L8 H
8 A5 |( f: T% q9 s' p
options: {peer=patch-tun}
5 \7 \0 a: T9 R9 C# z/ p# b, @% ]0 W1 n; l
% ^+ R8 I6 g* Y/ g' x8 \
6 M# T8 P0 i' ~2 J2 u' h ovs_version: "2.3.0". A( v7 _% v3 C4 x
$ X: S2 x0 R) @3 F( l2 A
7 i6 @: c5 }/ e$ z. Y# u ) Z* @* m, n0 {; f1 v' \3 h% r
! i4 ~) a, @% N3 W$ S
br-eth1向下连接物理网卡eth1,br-int向上连接vm,它们之间通过phy-br-eth1和int-br-eth1这对pair交换数据包。) z0 M9 A1 ]9 j0 v
# }2 N2 h4 w) { X+ y* l
% o% Q B: n/ Z1 t; t& _/ U! f4 P0 z3 C! b. v& j& C
7 T+ k, ~; }- E" R+ b0 |# q# W, ~2. Openvswitch 中vlan的配置/ i; V& L9 k5 S5 R7 k7 M) \
" I0 Z! Z4 E. [8 U
# L. q) u7 `9 }9 [/ E* t
4 u7 y5 }+ i5 e7 p, s( Y0 Q- % T2 M+ [3 M! O3 ?, s+ v: ~
. r0 R$ Q1 @( A8 ^/ [+ x cat /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
* S4 b. K" v( H8 p( Z, Z- O
" u T. v6 H5 B p$ N: `4 h - - ^: q% S+ F! E8 H0 F* K1 `
) M, v* N2 |0 x/ |3 p5 c' }8 n3 q5 ]; b& }6 D8 A
/ q$ r& C: E2 W# c; S
) h# M# f/ K9 H
( ~2 w/ l+ F1 {# W9 m* B
6 K! {! x/ ?( t; j/ i1 j* [
% W/ l3 P, U- a. H/ B, P[OVS]
* P5 k0 j8 [7 v4 l9 E; m& q
9 F- K) Q+ c8 R* R! s- l4 x
$ i7 z9 Q2 v) T6 B! l) w1 M
+ z. z6 x! q9 W8 P& R! Utenant_network_type = vlan
. ]5 n5 C; T& A! c( }+ f4 A8 }+ e( X3 Q2 ?
9 t: e' I7 \4 f6 _# s6 r* m |0 n; x/ L/ R6 Z* J, v0 Y! T1 {' X# D
network_vlan_ranges = physnet1:101:110
8 j# m5 v% C2 K+ x# S) Q2 i5 d7 r
' t J5 q2 T7 q2 V* D/ ]
1 q5 L# V4 ?( C% I8 I# I; _. Y$ e2 T8 ^% T/ b4 _, m3 H1 X
enable_tunneling = False( [# r* x$ b# C2 }4 W3 f
$ E- a' N: b! p5 z; d8 Q
! |7 E. p. r+ v R
( W9 `( A4 w9 L0 m; Aintegration_bridge = br-int" P) t$ ]7 i* x. H5 ?
, J, T. f1 m; E" @
- 3 o0 w7 x6 j: z- g
3 R2 g0 n2 \" u3 S5 btunnel_bridge = br-tun
6 O8 ]3 F1 J5 u9 B1 A6 D; w
* a% E' {" j# P8 Y/ X
4 ]+ o; O% {7 u+ \7 U1 j! x7 p0 ^8 N8 S; r7 ]$ E
local_ip = 10.14.0.102
2 f7 S$ P7 f" p. x; C) l' C2 c2 ~$ j( j1 m1 z3 K/ v2 c, \) K
- 0 d$ f) r/ Q( ~ ?+ N/ x
8 f+ ?$ ]+ l: `/ {bridge_mappings = physnet1:br-eth1
: d. F& C5 |" A# A& m( w
4 c# L; u' D+ z) ?& J- i4 a; E - 5 M& a$ z& Y' P
% ]) ]: U9 v) R4 t........
4 ^5 h) h/ l; B3 r
9 o1 J( p! }" K+ E* X# k% t
1 o! v" x/ j& Y9 L& s . }6 E' }+ e. w; ]" i$ T
# L. ]9 w" F0 w- ]& c9 v( |
重启服务:. E- s& Z0 w+ V& o( _9 w
c7 ]1 _' v2 F4 u# N9 r. M
u1 m' _7 W: p; s' A8 b3 r) V/ E. x8 J. P
. o+ k: c _# n( r6 S0 C
systemctl restart neutron-openvswitch-agent.service- U. b i% n* G! u/ J- H2 ?
6 Z3 Q6 e6 |* C3. 创建vlan网络
5 J \/ i' d0 n, @( D4 `8 P
. c/ b9 E# X" K$ h; a6 ]9 v3 U' ^7 H, a) z6 Q( W+ d
创建网络vlan104,指定使用vlan id104(segmentation_id) :
8 [4 [2 }. T0 T) ^' q) W# e3 B" ?- R8 c6 r j9 @
- s1 L1 M& z* v; h
# l. u# K) L) s( T) o Jneutron net-create --tenant-id 225e734464134ea79139eb6b0b202095 vlan104 --provider:network_type vlan --provider:physical_network physnet1 --provider:segmentation_id 104) [) t/ S. k: i! s Z$ G6 ^
+ |% Z8 j, R1 {$ v/ ^; k& S) ?( f. M# h4 b
+ h6 s5 e- N! g( A( g6 |+ J- |3 u5 K \: N/ S @: z$ T, L6 u- p
8 w# i i8 D/ T" N) n/ n5 z- . \8 m; y, _9 j- e7 N. R. B
9 V3 }! ^6 t X: M5 c1 K& _5 Z! i[root@controller ~]# neutron net-show vlan1041 R# P) Y/ }. C) M
5 @$ t5 |# u1 i9 M - - H. s/ U8 }) L* I; \# w! m0 _
7 S* n- Y6 e1 T% l2 M R, m
+---------------------------+--------------------------------------+& r( d" R; d, y- K7 A
& _3 c' s( H5 }) ?9 ? - 2 i$ g3 R& f! K* u5 P1 R
( Z1 i" q* ^ q| Field | Value |
+ a9 {! d% C+ f5 k% n2 V& D$ O$ \' B
- 4 B4 D' H* W0 [. z0 ~
. M3 b& U5 p" L; N3 Z! Z! ^0 B. l0 ]4 c+---------------------------+--------------------------------------+0 e, B" k1 \% T- b) v& T
* x4 }& V6 K7 I2 i9 j6 L. C! \ - + M# |# c9 ?7 c6 b; e. l+ d
' B3 O1 c- N, K9 [8 M+ T| admin_state_up | True |! P: Q, k: V. j7 ~
6 @$ z1 z V1 a' ` ^5 { F0 I. a
+ p2 H" x) M0 S6 T+ s* i. _% S
8 x- p7 x2 M( `/ {- l- F| id | 239d0325-5a65-4369-a05c-635b583f6185 |
! w( T" Y& I- W- V0 b. w, g' [' M" @8 r; A3 \& X
- 5 K8 S I& N, U4 e4 t
* _ ?( O0 J2 b3 Y4 f/ U
| mtu | 0 |
@: N0 |& [: A0 C$ m
$ j# S. C3 S0 y; A7 N
% u. X+ u# O: ^' @5 t: F7 e% W+ ]* [4 X( ?6 L `
| name | vlan104 |
2 e/ P; d/ [# d7 S! G! g; m0 h7 [' l! W5 a4 ]0 @3 w
- / `, \$ T8 n2 l
" g) P- ?( M4 o. r/ @
| provider:network_type | vlan |2 @# N& E; O: u$ H" M' _4 [
& Z B7 T/ h3 r, z! L9 m
! {5 J' j x1 h; p
+ Y0 s; S) D) P+ L0 w! _6 q/ q* [| provider:physical_network | physnet1 |( ^/ I; T" \ o: B7 [0 S2 S+ {
. T& g( s) ? J: L8 G
- ' Q7 r% r3 B1 J& u/ M8 x& |5 o# q# Y
' j1 d- B, L4 q1 y0 c
| provider:segmentation_id | 104 |
; \+ `- ?# m8 {* ]) x/ F/ a/ [9 Z
! m# J- ?/ J9 H# v3 X+ V - . N5 b" f- t2 F6 c2 n1 c% r" ]2 [
+ h* s" B; M9 h* l+ T1 n) T| router:external | False |! Y! q; ?6 m0 o N0 ~! @1 y
8 T8 o/ i( z0 V* v
/ G7 c. @: @ X' M. c1 h
1 W) A7 p5 P+ ^7 o( P| shared | False |0 Q( ?1 j9 {; j( {1 W8 Y! I
/ A$ L f1 z6 j8 L0 A* N- \
/ X# i; q+ L0 q! h, Y* l5 N
; Z* H8 X& W2 M9 l| status | ACTIVE |
, H" Z5 t0 [1 r/ i
5 l& M: D) W* K" T0 W* P# B
" ?- l* w2 H! L5 S
* W/ _' s# z% e2 h0 m3 Q| subnets | 76f8bbb0-52af-4b43-a246-679d6c456956 |
' p1 S7 P A6 H" V0 \' h* A4 \ A; K5 R# X4 }: t
- ( v: `! C; V, Y' A; P# O
6 W( w ]0 u/ v9 {| tenant_id | 225e734464134ea79139eb6b0b202095 |
# ?$ q+ d) O4 B
( v3 u. s5 [0 [, t! y( |, T: j
" Q9 F# y( b. X& C/ {2 Q7 O! L- }% I9 Z4 H, E
+---------------------------+--------------------------------------+
) D. O/ N! P0 U: z: a2 a
$ P8 j% \7 S% h
1 i! `) ]' L, ]8 h
* t0 y; s& O7 O
7 D6 r# \* S% b8 W8 X为vlan104创建子网192.168.104.0/24:
, {/ @0 e& q6 ?
* z$ R$ u; J: p5 Z- ]5 i% `# |/ x `) Y: G( x1 I+ N/ F
neutron subnet-create --tenant-id 225e734464134ea79139eb6b0b202095 vlan104 192.168.104.0/24 --gateway 192.168.104.1) T* P/ M3 @! h7 D3 d
+ F+ [+ O* v1 O
- : x. [6 P3 J# u8 p) Q1 I+ O; h
6 ?6 F( ]% P; K: O2 U) ?
neutron subnet-show 76f8bbb0-52af-4b43-a246-679d6c456956( s3 |/ O) u: Z1 X& v ~
4 ?" P" k g- y# x7 S/ k. v
1 f5 m8 H+ S5 [- F9 t/ }
+ ^5 D' w; O, A9 R+-------------------+------------------------------------------------------+8 y3 X) O6 `! J4 h7 z9 Y! f" m
9 k* Z* v2 N; Z _- n) s) Z
- . @& ~8 h: }7 b# J* k i4 {( m: f
$ R* `& Z/ Z# Q, |# S
| Field | Value |9 w( A" o# Z$ X
* j$ f# H8 z% U' {+ j9 t, b
- 5 B+ `/ U1 q! ~ @3 E/ [6 G
* k4 P% m5 _9 b3 W2 X, z n) q+-------------------+------------------------------------------------------+
; j* v* h/ @- c- B. p" N ~1 m0 Z4 l2 O9 S7 W
- , U9 ~4 \. x* \
' R" v+ l) N- _" P7 k| allocation_pools | {"start": "192.168.104.2", "end": "192.168.104.254"} |) e: z4 A4 J$ z% h. h% O& M$ s( r
$ L R& e/ M. y$ T
( i; k+ m0 L- h; z8 ^, b7 z+ I9 b! A5 g
| cidr | 192.168.104.0/24 |
/ X% |6 E: }$ R% v9 O" x M" s1 V2 q, u5 z) P& E
- 5 x$ U! s- d! t. u% D# h
1 V; G7 J- a) ^$ L. l1 M% y4 s7 Z) z| dns_nameservers | | l3 {5 L2 J \5 p, B
7 J; I+ L" h2 Q' W+ u; `) s4 h7 F; ^
/ f1 M& Y6 t, t" i6 @8 l e# j5 e$ \9 S4 j) n) h
| enable_dhcp | True |
4 T) G. N( `9 a; \; ]- w% t4 t6 J2 @; a- w4 Z
- ; M; h l: L5 k$ G% c: t
# H& I0 Z, |3 a- B' ?2 m
| gateway_ip | 192.168.104.1 |4 q C# j1 W' D0 A+ x8 J
$ K7 L) @) z' M" G8 S - ' _) F2 \/ C$ d3 s# T. D3 j
. M; k0 p3 _5 M& }% d$ E% _+ @
| host_routes | |/ c' |" a. u* H1 G) }, _
& y5 l. W+ j! }' Z% ?
- ' [* W; Z6 O5 ~7 n5 E ]
! j& H" |3 x) X' A' U9 r* N0 r| id | 76f8bbb0-52af-4b43-a246-679d6c456956 |
6 ]' S/ J( x' } f0 Y. U, q9 V* o) C: `: p- u. S
* k% i- W' y- ]' Y
% o6 Z/ A% `; j| ip_version | 4 |% ~: y7 b& E- u# z- q. i9 [( X9 ?
& C) e: j$ S* C K% L% D6 X
- # ]0 P' L$ H/ J3 E
6 j+ ?9 U+ d8 C; c) q4 o
| ipv6_address_mode | |4 u* m& H! X" g3 C& J7 D* c
* }3 i9 \0 k" P. U! L: P
7 W5 [' x3 i$ L) \' [& ^6 }1 }
: v, q( ?; f7 Q8 B: e| ipv6_ra_mode | |
0 ~ S, j( Q; y ^
5 Y: M! r# ]" S- ( k9 S J t* y4 H/ d6 O( Y
! x Q: m+ q4 F. ]! j5 P- V| name | |
5 O& Z8 W' l3 g9 P1 o: W! }+ V8 A" [; Z# s, c/ `( l$ d
- - ?' z6 i ^: V% E/ v! v m
8 U9 y/ l! k# X| network_id | 239d0325-5a65-4369-a05c-635b583f6185 |
/ a* t+ |! z8 X# Y" {9 B" r8 w3 A, M* M( y
- 0 E5 e- X' b; Z6 w1 B
1 } }$ I B3 E9 w. z+ m1 h- a; Z| subnetpool_id | |
% M, j* e6 F ^+ i; s; M, q1 R) g0 l
- # s2 j, r1 V' T1 ~
/ [0 U$ e+ g ?3 D| tenant_id | 225e734464134ea79139eb6b0b202095 |- L6 }# A9 j) _' U
* o' j# k K# Z9 G6 h* ^5 r
- ) z3 m* h+ x( W* B( m u. M' _
# D3 \ {) t8 P9 }
+-------------------+------------------------------------------------------+ ^0 p$ e, d4 E: z5 H( C6 v1 }
. } r# R6 Q9 o$ x% @6 o# B
* p9 T& s7 H9 C9 e$ y2 o: Y
& M$ r h3 n/ v' Y% f4 T% c d. R" |9 R' Z( z( V& c0 q7 q, z; }8 I
创建vm:
$ ]8 u5 ]; \; E$ t' M
3 D v6 X8 B9 O' a9 r: t K+ ?3 v, O. N7 g/ w; @* `% i& g: x5 c; l
( V9 j2 l7 w" c5 U2 `- X
/ G3 C) E# M/ h: \6 vnova boot test3 --flavor 3 --image 421f61ad-14d6-4054-a4bf-883fc1413834 --nic net-id=239d0325-5a65-4369-a05c-635b583f6185
" Y4 _1 V U& ^ Y! j2 z( G* w. {
9 `5 p. l3 Z# E- : Q- X" {3 ?/ t" N) L
( O- J1 L2 \7 K( e8 K& ] nova show test3
6 x- m5 U. H8 ?$ h+ O J- P
7 s! U" A- `( T7 m: ~3 K
9 U8 C8 Q$ W! G, Q% v- K& W# e$ O3 |; f. }, K$ R8 s
+--------------------------------------+----------------------------------------------------------+
2 W" p- [4 s! v& M
9 h: a( w' P. R
, ^, ]0 B+ j: r* a" c! @/ B4 i5 K. B8 o# A7 P! I6 i
| Property | Value |, U v5 [7 o4 H
' _# F: W% X& S# O
2 M+ [! V1 [6 H+ z* I' k$ w
4 j* `3 _* z* a8 C/ t+ |+--------------------------------------+----------------------------------------------------------+% R1 L1 O1 {4 |' j0 Z
$ K+ y7 n* Q6 K" t9 a
# \! V! }( u z7 x# d9 W, i7 B, y
3 }) B% k# ?% I0 @| OS-DCF:diskConfig | MANUAL |
. k" ]' T9 E; m& k T$ a' F* a1 C
8 U+ J3 j3 A# A7 T- & }$ e _. Z! e% z( F8 j( j
8 y/ P* q1 ?& Q; o| OS-EXT-AZ:availability_zone | nova |
& c% l5 y0 l6 X1 ]' o; Y0 r, r4 A; ^/ C
- $ ~: l ?; w: V: [# q# c, u
2 D$ o+ p. @$ h c" O. m( k| OS-EXT-SRV-ATTR:host | compute.ibm.com |9 s3 J: ~; x' g7 `& l
4 c: N, w, l' r
, l$ W9 t! ~9 Q! o" v$ L& t8 N4 Q! P+ R" Q) [& @2 h6 F: h
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute.ibm.com |' J4 g4 O# z- E' l; g
* R8 p' r+ s) d b% v2 o, I- * X9 a6 E5 ^+ l9 J9 I' s+ ?
- _5 c+ `( N" O
| OS-EXT-SRV-ATTR:instance_name | instance-00000005 |
0 V; l3 h- c9 f0 e% o( {
6 J7 g8 z+ t3 J( n - 9 R7 U' w& l8 v# O
( d7 }3 c& N3 [# ^/ O& d7 ~
| OS-EXT-STS:power_state | 1 |
7 N0 k. Z0 [; M0 X, \. |8 t0 \) N" `( C; s8 l# {
' l3 c- U( O5 K$ W. `- o
0 Z3 u7 r! m/ b7 ^! a1 X| OS-EXT-STS:task_state | - | E3 K2 U$ _) U* l
4 }0 G6 T8 H' c1 H! W0 e- 4 V+ _5 u2 H* ]4 ?
3 k3 m: g+ t# @
| OS-EXT-STS:vm_state | active |
7 S2 L+ y1 O$ A# N# Q; ~+ X6 \2 u6 l( K
8 `: w/ V6 x! H0 ], C
4 p6 W9 x: @6 h9 K, F| OS-SRV-USG:launched_at | 2015-08-25T02:40:51.000000 |
G, ~4 J; r& ?+ l
9 e: F9 h; R$ z
% X2 B# Q' g5 f2 n4 ^( Z7 Q8 C2 B$ @
| OS-SRV-USG:terminated_at | - |( o' ? _( V$ p& R7 H
3 Y* Z& n1 M! z( }- Q- ( N( z' f5 i" Y5 G5 k% P: u
2 ^* T$ z% J% {! p. U. D
| accessIPv4 | |
, a+ p5 `0 F) Q1 v8 ]; B" ^5 n: E( ?0 S" O4 [% @$ x5 n
- 6 }6 a! O& I1 {! z0 a+ [/ a1 w }9 p
) E8 _9 Y, O: f6 q+ I, c4 n* I
| accessIPv6 | |1 W) ~2 D X! B5 o5 Z
& d `/ N7 d9 E4 @3 R0 Y
* R; e& y/ N" R7 `4 L5 X' W1 W8 ~5 |) R
| config_drive | |
]; F$ u! `0 A% h( [
1 p) y5 l& I. z0 @1 ?- : X9 c U2 H3 I7 K8 k- {" d, l ]
0 z3 F( `& ~7 Y5 j5 M5 K| created | 2015-08-25T02:42:49Z |3 ~7 x8 x }+ s ^- P5 Y
- n( p4 {5 F% J( U3 `* C; ]
6 C+ L( Q- G' \; G+ W4 s2 o6 V
/ N8 p) e. o! ~3 Y| flavor | m1.medium (3) |0 t4 x; t: u9 m4 u3 \/ ~9 O( u; @
7 l% w, o' C2 X
- . H% u* ^" F1 i) N; Q
% ]7 a2 J# T u1 j- K; P
| hostId | ac73486a08f6bc6d7333faf316cec310ed2b19da6a55adffcedc30cd |
# N6 T1 a* X! s2 `0 N( S4 z( s7 N% A* H
3 G9 u9 ]" X( h- C
6 P9 @, y n' b$ A; }9 l: g" [3 a5 u| id | 4f91f8ca-78c0-49e6-a8be-96ab827315b7 |( ^8 w& q9 U0 y* O# h Y# R
5 M- t2 w" }8 i
- ' e+ U" r6 v* O5 l5 r/ O! B2 H
6 Z7 Y# h( I9 h# ]- P2 t7 N3 J
| image | rhel65 (421f61ad-14d6-4054-a4bf-883fc1413834) |8 S$ J+ [9 V1 \. }* T
5 I& R# w5 i, k! u6 T
- 3 u/ L% S& X; W1 x0 g. C
$ t% v/ b9 J0 j$ d
| key_name | - |
5 [# @/ w! j- E! u' m! {+ \& ^+ I+ P* E" Z* C/ p
- Y* t8 [( w& |9 m' u$ e
6 J0 D; R5 n# D. w+ z- i& _9 H& @* H| metadata | {} |9 I: F; E+ g# j
0 Q0 n- d/ V7 A% z" X- | `
+ Q; S5 m: |; n3 I9 S0 x$ e$ z# e/ i8 a- o$ z9 q' `
| name | test3 |& K; a$ S4 H5 U
( Q$ ^8 V% k2 K p6 o
5 F, b: Y9 _/ d/ ]% b/ Y
; i* x% q! I) }% W| os-extended-volumes:volumes_attached | [] |
3 p8 c, w. h, e$ A, D
( P) y2 C3 q+ d- O* ?$ r7 h7 w
: ^' q' L2 x- Z5 J% J) n- m- } }2 {
| progress | 0 |8 m4 t* [7 U! P# q2 w% y
R5 ?! Y8 g! R3 h9 A2 v+ J
; F. `! H+ ^ J* X
( y8 \ f0 \2 Z9 \9 \5 v7 f| security_groups | default |
; F4 }" j* s( o# t! j! C, y# f: R: i4 ]5 Z. g% x' A, b2 m( F
- ' B- {# u% D. J- J# Q4 h2 H0 x1 g
9 j9 _8 }1 G: m" h. O5 u% g
| status | ACTIVE |
( w G4 e! n. S- Q, C8 P) Y: I% G1 j" q
% ^" u6 A* v3 N0 l) j: h- |9 c. r p( g* `0 i+ i7 Y
| tenant_id | 225e734464134ea79139eb6b0b202095 |1 `9 H8 k6 E+ E& E% ~: H
# Q- q2 ]( v3 `, K8 K _6 r
4 e0 d0 m U* c; |
0 w2 C5 C; g% e2 `7 V1 V* @4 `; Z. N| updated | 2015-08-25T02:42:57Z |) [) O, I5 d0 ~3 v. a' e
+ g3 M& N: l# I* l
/ `4 `( C1 R! A& F Z& \- B; X) } \( w# |# m
| user_id | 6454c58d76c245c194d744cdb245a0a8 |
6 Y O; C: v* c K3 @$ l" F
1 z( W! ^. z r5 v
0 H* W! \! w4 O: n3 A% k+ {0 _3 L5 V% V
| vlan104 network | 192.168.104.5 |
0 b( n) x$ ?3 h. G) B/ B% H) a2 \$ w: r& n2 U) {1 U0 N
- ! K' R+ D6 }4 |9 O
, ]1 i- a3 J/ Q* |
+--------------------------------------+----------------------------------------------------------+
: A: U. U' d6 ?/ Z+ Q: I" i
- d" Q/ Z; o% z0 q) y/ m; v& Z/ \
8 n$ a1 L' J; v% G2 X" `# U
9 B# ~9 R' {$ ]6 \' t7 e4 |6 u5 n8 o) G; I
使用vnc查看vm:# M8 c8 S. u0 ]$ U* o
% k- G9 }, }' S, W) {
! K# J; I, F, n" P
7 q0 }! U, A1 @
0 m8 d3 g" e! [' c+ f& I" ~: v5 N6 d( ^0 r% e
) g5 c& S9 k) S5 R
+ U4 D m# u" t G: D1 a
四. 抓包
9 x' l8 J4 }8 S% \3 v3 c: l! Z" K
1. Compute 端6 o' |9 W# l" {& L( m
4 m/ d0 X; H/ \# l3 }
" Z2 }# M+ @" j
; V( h* }- D6 y s; r( _' ]+ a; Q
( F) f/ c6 j9 ~9 u2.Juniper端8 r3 m. M% S Q* b1 _
7 q4 u/ `3 [" B

( i4 {( |( u) C7 G! l" ]/ ]2 R! c, Z/ T* u) a7 N7 Q& q
; Y: b7 v- Y- Y( s3.Controller端:- ~( k; B2 Q4 n6 o/ n. x
" f9 [- c$ q+ d
[img][/img]
% Y6 z8 K1 N( z+ f
+ W! @+ d' W- \' M- B |
|