|
edis和docker这两个词语会自动被修改为首字母大写并链接到知识库,所以在这里先写一遍,后面就不会被改写了。 0、具体操作见 (vmware中搭建k8s),virtulbox也是相同的流程。* ~+ x t5 R% }! ^
1、学习k8s,读的是这本书《KUBERNETES权威指南 从DOCKET到KURBERNETES实践全接触.pdf》 2、这边书刚开始讲的是在单机上搭建一个k8s的hello world,用的是centos7. 于是我也在virtulbox中安装了centos7,并成功运行hello world。 3、然后,开始尝试集群了。在网上找了个教程,比较简单,很快就成功了,kubectl get nodes能看到各node了。 但是,应用跑的不正常,有的节点可以访问,有的节点不可以。而且从不同节点访问,查到的数据不相同,似乎是多个独立的系统。怀疑是iptables中cluster ip的规则有问题。
+ X+ E' p# Q% i+ Y( k2 i; U 接着,集群坏了,k8s的基础服务都启动不了。怀疑是不是因为我创建这些虚机时,用的是链接式拷贝,是不是原始的虚机安装了其它软件,导致k8s集群启动不了。 4、删掉重来,用完全拷贝的方式建立虚机。 一切正常,但是在启动redis-master-controller.yaml时,docker中无法建立容器, 用kubectl describe pod redis-master命令排查,发现下拉不了镜像。其实这些镜像已经存在于docker中了(搭建单机k8s时,自动pull的)。只是镜像名字前面多加了docker.io/ 我修改了redis-master-controller.yaml中镜像的名字,仍然pull失败。奇怪。 5、今天定位了网络问题(见virtualbox虚拟机无法上网),网络搞通后,hello world终于正常运行了。 之前的一些疑问,有答案了 1、随便访问哪个node的ip(比如http://192.168.56.251:30001/,http://192.168.56.252:30001/)(http://192.168.56.250:30001/是不行的),都可以访问到服务的(proxy自动转的) 2、在内部时,访问真实端口也是可以的。 [root@centm ~]# kubectl get svc
0 }% u' N/ P: l5 ?NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
& P0 T5 V! m1 ^; cfrontend 10.254.218.57 <nodes> 80/TCP 5m. u. d# `+ v+ C4 ^; ]8 F* I0 k
kubernetes 10.254.0.1 <none> 443/TCP 15d
; ~! p/ |# ?+ ?0 `4 predis-master 10.254.142.174 <none> 6379/TCP 8m
/ r3 [ N1 F* k. z+ n' }; u% p' lredis-slave 10.254.201.123 <none> 6379/TCP 6m
& J# p+ z* l! |8 f/ V9 S: [ curl 10.254.218.57:80 可以通。 ping 10.254.218.57 不通 3、node中多了一个127.0.0.1 ,不知道为什么
5 u4 u: A# o6 O2 p9 R [root@centm ~]# kubectl get nodes4 N, |* E$ t* }- T6 u8 m
NAME STATUS AGE0 J2 H; u3 E9 X# H G6 X8 A
127.0.0.1 NotReady 15d
- s$ C' |* L9 ?4 j% x* ycents1 Ready 1d9 V. {) a {4 V, o5 E D
cents2 Ready 1d0 c( p& k# W7 |+ ^: j
[root@centm ~]# ps -ef|grep kube2 h1 z8 O; I+ g
kube 578 1 0 Jan20 ? 00:15:55 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080
' O) C3 |5 c& |# n/ o S0 Tkube 588 1 0 Jan20 ? 00:01:09 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://127.0.0.1:8080
; L& V; Y# e0 \- X& [5 N! [8 Hkube 2079 1 0 Jan20 ? 00:08:11 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota
5 ^9 x+ ]( b1 M8 R; g7 u4、clusterip是一个虚ip,实际是iptables中的几个转发规则。 [root@cents2 ~]# ip a
9 V2 A9 N0 l* c# |6 e1 _2 k. h1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
4 Z: o* M$ O: t& q0 q2 u link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2 o' h Q) M7 p! k4 @6 g inet 127.0.0.1/8 scope host lo
+ A- c3 S7 g P2 x valid_lft forever preferred_lft forever5 n, @ C8 p2 G1 n3 q; B
inet6 ::1/128 scope host
6 q) Q6 q7 p# r6 o" u) r2 Q) ] valid_lft forever preferred_lft forever D3 ^/ ]3 D( T$ X) V% M9 c6 |5 O
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000: B0 q3 X; m+ m' \3 ^+ a2 H
link/ether 08:00:27:58:5d:6e brd ff:ff:ff:ff:ff:ff
- }# ^& V% u3 e( ~ inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
3 o# X# F: O- W4 ^$ S' g# ]( R" f valid_lft 82058sec preferred_lft 82058sec
5 L% v* f7 M; _' D; s, {# v inet6 fe80::b171:84d0:5173:de63/64 scope link3 A8 h& Y/ ?" }" k- {. f
valid_lft forever preferred_lft forever
" Z; v" @8 D; R3 B0 `6 V8 X c1 M3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000: _/ T3 z ?, X0 @
link/ether 08:00:27:7a:24:14 brd ff:ff:ff:ff:ff:ff
+ i4 q5 H0 R$ C# r) T inet 192.168.56.252/24 brd 192.168.56.255 scope global enp0s8
9 ?/ p- S5 e' q# d3 i1 q1 } valid_lft forever preferred_lft forever
8 f% L( i8 L) I' k+ ~# n inet6 fe80::a00:27ff:fe7a:2414/64 scope link
: H6 q' \9 M9 z* p2 r- s4 S( ~ valid_lft forever preferred_lft forever, z8 [% P; P" V5 |, [
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
) V7 ]8 p- Q. @ link/ether fa:5a:c7:c5:aa:e5 brd ff:ff:ff:ff:ff:ff3 V! H+ U( p+ ^7 L3 ?
inet 172.16.80.0/16 scope global flannel.11 \& z8 C9 q# b5 N& X
valid_lft forever preferred_lft forever
+ j e' K) ]* n9 ^5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP" k2 h% C* Q/ L# r6 @( i5 o/ z; j
link/ether 02:42:72:2f:1f:ae brd ff:ff:ff:ff:ff:ff7 W# {) Q }4 X6 u6 T8 M4 w) O9 D
inet 172.16.80.1/24 scope global docker0
: n% C3 m* P, s3 h8 t. i& Z valid_lft forever preferred_lft forever
: c3 J- O8 C0 b' o0 a' R- f5 |' B8 Q inet6 fe80::42:72ff:fe2f:1fae/64 scope link: k9 F/ o" [% x& ]. \9 _
valid_lft forever preferred_lft forever
( H+ y3 D9 G3 b, H+ \7: vethc56c1d4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
: p* k( Z$ w3 q+ |; T# ~ link/ether 92:c8:3d:3f:b9:49 brd ff:ff:ff:ff:ff:ff link-netnsid 0
F. |- q8 p3 { inet6 fe80::90c8:3dff:fe3f:b949/64 scope link f% X+ r8 E" A
valid_lft forever preferred_lft forever
B2 ?; Q7 |& y8 s- a& i9: vethf961994@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
" C4 L6 r. M/ B: p0 b/ O; V link/ether d6:be:4b:6e:26:81 brd ff:ff:ff:ff:ff:ff link-netnsid 1( F T$ t2 C, s( T# A
inet6 fe80::d4be:4bff:fe6e:2681/64 scope link0 f8 U' y5 h) ]6 f
valid_lft forever preferred_lft forever1 p2 k n1 S6 ~ @5 N5 I0 n6 S3 T" H
11: vethe4cd28e@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
9 c( @! @. |& ?5 L0 F, m* K) C link/ether ee:55:55:df:4e:50 brd ff:ff:ff:ff:ff:ff link-netnsid 2/ m1 S3 L2 ]; @3 t8 w
inet6 fe80::ec55:55ff:fedf:4e50/64 scope link
+ | }& B7 X6 t8 t valid_lft forever preferred_lft forever
9 c8 o$ z5 p/ R- z0 J6 M7 o1 v# F8 x: z- b- d8 z( A
[root@cents2 ~]# iptables-save
/ @/ a$ ?$ S( G- G2 u ~# R9 L) g9 m# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
: C+ @$ z; \9 @*filter
3 `! ?& b0 Y1 ~* d* k- a, d:INPUT ACCEPT [27:4324]9 A4 w5 m' }, W) u; W
:FORWARD ACCEPT [0:0]5 k% g6 P! Z( [0 D U2 }8 m
:OUTPUT ACCEPT [25:2821]
: G* k3 @. E. y. E# Z:DOCKER - [0:0]
3 `# w. O' V7 `0 d; x0 J& g& |; E:DOCKER-ISOLATION - [0:0]
6 u; W8 s0 u+ v" E- S, u3 j6 O:KUBE-SERVICES - [0:0]' T3 A5 B0 v" ^# O
-A FORWARD -j DOCKER-ISOLATION
) |. a* b! s" I4 ~) M; w-A FORWARD -o docker0 -j DOCKER) P3 N2 Q" G2 V% c6 ~, }' x
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
" e2 G* ]+ ?# y; s7 X) {-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
9 a/ r6 @0 O5 M6 M/ o! y-A FORWARD -i docker0 -o docker0 -j ACCEPT
6 ]. q( C5 ^+ ~: X/ Q* f2 I-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES/ _/ u& y C& I$ b0 Q* ^: ~
-A DOCKER-ISOLATION -j RETURN! r. h5 _: @! j% }. {$ e
COMMIT0 w9 [- B: {; X. z* c, C, m1 V
# Completed on Sun Jan 22 00:41:01 2017
5 C8 O# ?7 f" [( G7 B# A5 d" w# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017& K0 J3 o; q5 l: P ?; j" d' U, b a+ N
*nat
( W4 {! L4 m$ F- c- f+ F# d6 ?:PREROUTING ACCEPT [0:0] n+ i9 O$ k) i. c& B: F# v
:INPUT ACCEPT [0:0]* V B: W$ b1 R; L. _3 Z
:OUTPUT ACCEPT [2:119]: t5 x& o! r2 q5 b m7 r0 X
:POSTROUTING ACCEPT [2:119]
# I* o* n' y, N8 e6 j- h/ D/ R:DOCKER - [0:0]
2 W, Z* L, o) O; ^- R( P/ X$ A& v7 ~:KUBE-MARK-MASQ - [0:0]
- U5 y; Y& f4 `1 A6 U L. t:KUBE-NODEPORTS - [0:0]6 b7 ~4 u M, L6 d# c6 b1 T* S
:KUBE-POSTROUTING - [0:0]- i* x+ }6 Y4 K" H5 M$ F! x# J
:KUBE-SEP-63GTHXGNEQIFF6GY - [0:0]$ g! S* M: @0 K$ C% ^
:KUBE-SEP-77PLGVXVTAKNHL2K - [0:0]4 X) c1 q# Z, I/ K
:KUBE-SEP-7R2ESD4YYXMXFEFZ - [0:0]
4 v; v3 l4 j8 p3 d% G:KUBE-SEP-GIMIRAR4ZAKGMA2Q - [0:0]
4 [! o1 I. J( l:KUBE-SEP-LYGBYJFMWSAWPLXU - [0:0]* ?* L) R5 w; c, v4 G u! S8 H# P
:KUBE-SEP-Y7WMR7EBCL4N3QJX - [0:0]" S/ Y. p2 |( k# l3 J
:KUBE-SEP-ZDWRYP3AMCRYOGNR - [0:0]
. q! @. C! E, r7 S:KUBE-SERVICES - [0:0]
+ e. r+ P0 m8 Y4 t:KUBE-SVC-7GF4BJM3Z6CMNVML - [0:0]1 G/ u" M! V. n: k
:KUBE-SVC-AGR3D4D4FQNH4O33 - [0:0] ]! F' }* I8 D' v- M
:KUBE-SVC-GYQQTB6TY565JPRW - [0:0]! Z7 }: C7 O& q% g
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
9 P/ A' q, t, K4 p6 R-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES# P: s3 U! Q# U3 d7 l ~- x
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER2 P) o) _' E% Y3 v# q E1 `
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES; q2 c7 I0 Z2 g0 q0 @. G' A
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
6 @& ? ^2 Z* T-A POSTROUTING -s 172.16.80.0/24 ! -o docker0 -j MASQUERADE
3 l, H7 K$ v+ E G4 N8 x7 j/ Z-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING7 x K$ q4 r$ z3 b) H6 P/ w
-A DOCKER -i docker0 -j RETURN
( F3 t0 C6 u; M* m-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
# w& Q) X7 ^; G; z9 {# m" k-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-MARK-MASQ
" o, \9 a* W: U3 T( g( O-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-SVC-GYQQTB6TY565JPRW# D8 c! `/ L# C
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE! J8 u0 r$ ~8 B
-A KUBE-SEP-63GTHXGNEQIFF6GY -s 172.16.62.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
5 d" `# E, K* ^6 |$ F-A KUBE-SEP-63GTHXGNEQIFF6GY -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.62.4:80
2 W8 W; b5 U+ E, p/ o-A KUBE-SEP-77PLGVXVTAKNHL2K -s 172.16.80.3/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ) S( e. t) H2 z$ q9 v: y# g
-A KUBE-SEP-77PLGVXVTAKNHL2K -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.3:807 Q- T& G7 x1 G9 g# D
-A KUBE-SEP-7R2ESD4YYXMXFEFZ -s 172.16.80.2/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
5 D6 Q+ G( i [+ _4 |5 @& |-A KUBE-SEP-7R2ESD4YYXMXFEFZ -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.80.2:6379
! B: ] x, F( u4 Z; A-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -s 192.168.56.250/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
2 J- e! O" ]& k( a-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.56.250:6443
; z! [& R+ N" Q4 g* S-A KUBE-SEP-LYGBYJFMWSAWPLXU -s 172.16.62.3/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ1 w2 q! g' {7 z- X$ W
-A KUBE-SEP-LYGBYJFMWSAWPLXU -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.62.3:63794 X: k# y- k" F" w6 g
-A KUBE-SEP-Y7WMR7EBCL4N3QJX -s 172.16.62.2/32 -m comment --comment "default/redis-master:" -j KUBE-MARK-MASQ
% P) g% j9 u1 P. E( X7 ~+ H-A KUBE-SEP-Y7WMR7EBCL4N3QJX -p tcp -m comment --comment "default/redis-master:" -m tcp -j DNAT --to-destination 172.16.62.2:6379
6 L- |4 C, B2 r0 b0 x* G) Y; {" E-A KUBE-SEP-ZDWRYP3AMCRYOGNR -s 172.16.80.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ$ Q# g$ `3 P7 [4 R9 z
-A KUBE-SEP-ZDWRYP3AMCRYOGNR -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.4:802 N1 `6 e( ^% c; o1 P
-A KUBE-SERVICES -d 10.254.218.57/32 -p tcp -m comment --comment "default/frontend: cluster IP" -m tcp --dport 80 -j KUBE-SVC-GYQQTB6TY565JPRW
' y! @ _- G: o$ Q5 _ I-A KUBE-SERVICES -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y# T% q8 |" `; M4 l3 R+ n
-A KUBE-SERVICES -d 10.254.142.174/32 -p tcp -m comment --comment "default/redis-master: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-7GF4BJM3Z6CMNVML
; g# G; h0 s9 `+ q7 D9 k) p8 D8 W-A KUBE-SERVICES -d 10.254.201.123/32 -p tcp -m comment --comment "default/redis-slave: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-AGR3D4D4FQNH4O332 `4 @4 r0 e# f- u" h4 V- e
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS' q# B% `- X, ~4 s
-A KUBE-SVC-7GF4BJM3Z6CMNVML -m comment --comment "default/redis-master:" -j KUBE-SEP-Y7WMR7EBCL4N3QJX- T% N4 x! j. {' q) Q0 k3 C
-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-LYGBYJFMWSAWPLXU
# Q4 @+ \6 o. G( @6 q; m' C/ P-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -j KUBE-SEP-7R2ESD4YYXMXFEFZ4 C7 i% `) t4 Q/ H X
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-63GTHXGNEQIFF6GY
2 ? l" H0 K0 ^3 N-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-77PLGVXVTAKNHL2K4 p- R7 _4 ?$ O; R' z* R2 P" G
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -j KUBE-SEP-ZDWRYP3AMCRYOGNR
; [% _2 ?( w2 `-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
( l' N8 y) x+ l, B. Z-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-GIMIRAR4ZAKGMA2Q2 @6 U. h/ I+ R4 Q( q& b
COMMIT; O1 J$ W) o1 `9 U
# Completed on Sun Jan 22 00:41:01 2017
7 D% f0 e4 R& `9 Q: m+ A0 v) C' s
; F# m4 x. L" a$ c" g---------------------------------------------------- 尝试了本地卷 [root@centm ~]# cat redis-master-controller_with_volume.yaml
0 |" O0 x6 t8 X" ]" GapiVersion: v1
: g" v b h8 {; k; d7 Pkind: ReplicationController 0 p; Q4 g% q- M; W
metadata:
7 d5 C. h! A" Q$ W labels:edis-master
6 f* `: F, k: {0 R name: redis-master
- g" H0 F& z; S& r0 N# `2 Y6 sspec:
" J+ U1 d& M7 u: S& _ replicas: 1" X3 |3 n" j. m' M& m3 j1 N8 H
selector:
9 M2 K( t- V) a G name: redis-master
/ Z1 h" @0 e; J/ ~" N% N2 v template: $ ^6 z! h9 X3 ^# o/ J5 v* _
metadata:
8 J8 }) p1 t7 | labels:0 S( ]4 ]. e' J' v& R9 x
name: redis-master1 T% K& d8 {. E& U7 t
spec:. s" d; r: f' X0 O' I0 ~
volumes:, v, }7 U1 \& U( m9 S
- name: "gf-dir1"' C& v! Y4 M* x% M0 T' f& |
hostPath:
0 k* K T( b- G" w5 U, t# x8 m path: "/tmp", ?7 g6 M7 T9 u5 @: [, V
" B: ]( q3 ]( H# j" p2 K containers:
% ^& \/ Z1 q K# F D! ]: r - name: master
4 i- q9 C0 R% E9 K! | ports:: docker.io/kubeguide/redis-master:latest " m9 u/ l# z+ Q7 b
- containerPort : 6379 3 b) G8 v5 X3 b. l7 ^/ d2 m. o
volumeMounts:2 f% h: J6 `, l! ~$ ~- Q, G
- name: "gf-dir1"
, [- x* B% U9 a# }3 N' q mountPath: "/gf1" |