|
edis和docker这两个词语会自动被修改为首字母大写并链接到知识库,所以在这里先写一遍,后面就不会被改写了。 0、具体操作见 (vmware中搭建k8s),virtulbox也是相同的流程。
# p0 n3 E! t/ B H0 ^; K 1、学习k8s,读的是这本书《KUBERNETES权威指南 从DOCKET到KURBERNETES实践全接触.pdf》 2、这边书刚开始讲的是在单机上搭建一个k8s的hello world,用的是centos7. 于是我也在virtulbox中安装了centos7,并成功运行hello world。 3、然后,开始尝试集群了。在网上找了个教程,比较简单,很快就成功了,kubectl get nodes能看到各node了。 但是,应用跑的不正常,有的节点可以访问,有的节点不可以。而且从不同节点访问,查到的数据不相同,似乎是多个独立的系统。怀疑是iptables中cluster ip的规则有问题。
- R3 [% _/ z, B0 s; d- ]8 Q& J, b 接着,集群坏了,k8s的基础服务都启动不了。怀疑是不是因为我创建这些虚机时,用的是链接式拷贝,是不是原始的虚机安装了其它软件,导致k8s集群启动不了。 4、删掉重来,用完全拷贝的方式建立虚机。 一切正常,但是在启动redis-master-controller.yaml时,docker中无法建立容器, 用kubectl describe pod redis-master命令排查,发现下拉不了镜像。其实这些镜像已经存在于docker中了(搭建单机k8s时,自动pull的)。只是镜像名字前面多加了docker.io/ 我修改了redis-master-controller.yaml中镜像的名字,仍然pull失败。奇怪。 5、今天定位了网络问题(见virtualbox虚拟机无法上网),网络搞通后,hello world终于正常运行了。 之前的一些疑问,有答案了 1、随便访问哪个node的ip(比如http://192.168.56.251:30001/,http://192.168.56.252:30001/)(http://192.168.56.250:30001/是不行的),都可以访问到服务的(proxy自动转的) 2、在内部时,访问真实端口也是可以的。 [root@centm ~]# kubectl get svc4 I" u1 z* W' @' d% b7 a
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE, x! Q2 Y/ |6 c* t4 J2 \0 D
frontend 10.254.218.57 <nodes> 80/TCP 5m- q+ e2 a$ r) R( V
kubernetes 10.254.0.1 <none> 443/TCP 15d: q4 v% R1 g: ?# M+ f$ @
redis-master 10.254.142.174 <none> 6379/TCP 8m# e- h3 i, L; g$ N
redis-slave 10.254.201.123 <none> 6379/TCP 6m
8 _3 r, X# N8 i curl 10.254.218.57:80 可以通。 ping 10.254.218.57 不通 3、node中多了一个127.0.0.1 ,不知道为什么& w4 K% i# F. Q/ p+ p
[root@centm ~]# kubectl get nodes
q4 u& Y s. R$ y( V# H2 BNAME STATUS AGE
# {9 \! D% O, C, M2 Z127.0.0.1 NotReady 15d0 ^# y/ Q1 }( F8 Y
cents1 Ready 1d
: t; t3 m6 _. K; E& z B4 kcents2 Ready 1d& A3 v* {' v( ?5 ? `6 ^) i
[root@centm ~]# ps -ef|grep kube
; a6 p2 U0 u) v. I1 ukube 578 1 0 Jan20 ? 00:15:55 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:80802 B) Y( X6 ^2 A8 e* j, l& v
kube 588 1 0 Jan20 ? 00:01:09 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://127.0.0.1:8080" t' @$ h& w7 ]2 f
kube 2079 1 0 Jan20 ? 00:08:11 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota/ g3 T9 c- P& P
4、clusterip是一个虚ip,实际是iptables中的几个转发规则。 [root@cents2 ~]# ip a% V) r( R7 c3 h: i1 g
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
2 c2 V0 Q! D. w! o5 y7 {# I link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+ m* R! U* n. V, s" l1 g6 _ inet 127.0.0.1/8 scope host lo
+ V- u2 B2 w- D2 |6 S valid_lft forever preferred_lft forever
& I9 o( K0 J8 b4 t1 D8 V inet6 ::1/128 scope host8 t8 @) s; L7 i9 d4 x* o2 w9 M
valid_lft forever preferred_lft forever
* S& z% E/ {9 v, X* ?( @2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000) R" e3 {! R* {6 T
link/ether 08:00:27:58:5d:6e brd ff:ff:ff:ff:ff:ff- e [$ w$ s) ~& t( ~9 J' x3 c
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
" `& E3 N$ ^/ W+ u9 U# i8 A- E4 |/ W- Y valid_lft 82058sec preferred_lft 82058sec" c$ a8 n3 T& E+ H& d9 Y$ U& c
inet6 fe80::b171:84d0:5173:de63/64 scope link/ P9 [& C* c7 B" N* j; b
valid_lft forever preferred_lft forever
5 K+ w9 b, X0 j9 h" \" ~- s/ F( A! e0 l3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
R, y3 V; @9 W( }( z. y: B( f link/ether 08:00:27:7a:24:14 brd ff:ff:ff:ff:ff:ff. u7 s6 @2 L# j3 t, e! _/ E
inet 192.168.56.252/24 brd 192.168.56.255 scope global enp0s8* Y6 {; l1 K: p. j; ~
valid_lft forever preferred_lft forever o6 E% l! G8 l4 j% c
inet6 fe80::a00:27ff:fe7a:2414/64 scope link+ j+ _6 p/ U v3 _/ q% A4 d7 F
valid_lft forever preferred_lft forever, J( [ p9 P# V# c- n
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
6 _4 ~0 Q2 F& [4 J+ K) r7 {+ K2 z3 O link/ether fa:5a:c7:c5:aa:e5 brd ff:ff:ff:ff:ff:ff) e! ^8 c. s. M3 Q2 k) U
inet 172.16.80.0/16 scope global flannel.1" u4 }% @7 r8 P- }# B
valid_lft forever preferred_lft forever2 D9 V. P0 {- t& `3 R
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
( G" V0 f& Y! [ m$ d: t link/ether 02:42:72:2f:1f:ae brd ff:ff:ff:ff:ff:ff) ?1 _6 }$ A& O* ~8 Y& n3 [. j4 B7 i
inet 172.16.80.1/24 scope global docker0* d+ p- b1 N6 A% t }' G
valid_lft forever preferred_lft forever% B& V. a1 O3 W- |
inet6 fe80::42:72ff:fe2f:1fae/64 scope link, y6 N1 C* j5 _" U
valid_lft forever preferred_lft forever
" J4 a+ S" l& V5 M7: vethc56c1d4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP5 o* z1 S' v3 K) x9 C! N% F" w
link/ether 92:c8:3d:3f:b9:49 brd ff:ff:ff:ff:ff:ff link-netnsid 05 Q/ E, ]8 l5 G- A
inet6 fe80::90c8:3dff:fe3f:b949/64 scope link
! m; J/ M8 o W. M$ k" q B valid_lft forever preferred_lft forever
( T% _* O( ?+ ~9 i1 p, ?3 q9: vethf961994@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
- X! @9 |" M+ o" v0 m3 { link/ether d6:be:4b:6e:26:81 brd ff:ff:ff:ff:ff:ff link-netnsid 1, X* l9 Z+ Y. L1 x& |; s
inet6 fe80::d4be:4bff:fe6e:2681/64 scope link$ j9 i1 d4 }6 p
valid_lft forever preferred_lft forever) S4 p# V3 S& v0 W" ?
11: vethe4cd28e@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP. _) q5 ?9 h7 p* Z8 ]6 K) V. N
link/ether ee:55:55:df:4e:50 brd ff:ff:ff:ff:ff:ff link-netnsid 23 J1 p. B+ _) s" r3 [
inet6 fe80::ec55:55ff:fedf:4e50/64 scope link
1 Y5 D3 u4 E K. Z( b5 Y! { valid_lft forever preferred_lft forever
) W0 |( W( p G/ x
9 W, H9 u% W2 Y0 U1 K1 R[root@cents2 ~]# iptables-save: Y# J% U% @7 {& V8 x9 m" f- M
# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017/ I6 [2 u5 M6 u! p8 C0 t, e
*filter
* |$ H6 F+ n# t$ w8 y:INPUT ACCEPT [27:4324]& h: W& [5 L/ d# `/ z& L
:FORWARD ACCEPT [0:0]
- J3 i8 `3 ?* t3 z:OUTPUT ACCEPT [25:2821]5 s6 M/ x, E2 K& X: e
:DOCKER - [0:0]; g7 Z4 R* Z" p
:DOCKER-ISOLATION - [0:0]* s! w8 J x; H7 `+ E
:KUBE-SERVICES - [0:0]
1 I1 o9 {) I. _4 U" c5 l-A FORWARD -j DOCKER-ISOLATION
; Q) v( _7 E( t; |# l- R-A FORWARD -o docker0 -j DOCKER1 u! H9 }6 ^9 @0 w6 g
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
0 u* p6 d) H* b7 Q$ L-A FORWARD -i docker0 ! -o docker0 -j ACCEPT+ ` p5 j! _1 d# j
-A FORWARD -i docker0 -o docker0 -j ACCEPT Q) D( i- @! L; v8 _: z1 J/ H
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
& s! E, m" s' e-A DOCKER-ISOLATION -j RETURN9 B/ J. r$ s1 _' X# \7 ?8 A
COMMIT; r. }+ e* [. \8 j
# Completed on Sun Jan 22 00:41:01 2017
4 @& |3 N( ~/ u+ q2 |# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
9 y1 ?" l) R9 N: X, M4 z( F*nat9 z+ q4 Y; K- }2 B* v+ p( b
:PREROUTING ACCEPT [0:0]
1 |9 e7 C0 r# _- I:INPUT ACCEPT [0:0]
( m/ S: ]* j8 G% ~; U:OUTPUT ACCEPT [2:119]
( a p0 Y; a. Y1 ` Y:POSTROUTING ACCEPT [2:119]: C2 N. C) z, N7 g
:DOCKER - [0:0]5 D3 @ I' M. w) Y2 s5 _) ?
:KUBE-MARK-MASQ - [0:0]9 d4 l5 H( Y* b2 D2 m$ N
:KUBE-NODEPORTS - [0:0]
) ?- \- L; S6 F! x. U, `:KUBE-POSTROUTING - [0:0]
# M1 _5 i/ q* ?# z! A9 b6 c:KUBE-SEP-63GTHXGNEQIFF6GY - [0:0]
$ a8 y4 q, _- O# N3 z) u:KUBE-SEP-77PLGVXVTAKNHL2K - [0:0]$ y! u" i: h6 `5 l8 _5 @
:KUBE-SEP-7R2ESD4YYXMXFEFZ - [0:0]5 g& ], n+ K5 x5 D
:KUBE-SEP-GIMIRAR4ZAKGMA2Q - [0:0]% h* t& L: a7 J
:KUBE-SEP-LYGBYJFMWSAWPLXU - [0:0]
$ s, ~8 E* f# a: {% t* y" J:KUBE-SEP-Y7WMR7EBCL4N3QJX - [0:0]
$ d5 G4 l. f+ C* ^! C+ T& O:KUBE-SEP-ZDWRYP3AMCRYOGNR - [0:0]
5 d; M0 C- P) L" A$ {4 d" W:KUBE-SERVICES - [0:0]
, z: A1 B1 ^* k+ Y) q, |:KUBE-SVC-7GF4BJM3Z6CMNVML - [0:0]3 k) E/ x" {; J0 u9 b; i/ G
:KUBE-SVC-AGR3D4D4FQNH4O33 - [0:0]4 c# C5 D' a, l% C: c
:KUBE-SVC-GYQQTB6TY565JPRW - [0:0]; G5 v- I/ P( _; a$ W
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]! g5 J' x( y6 q0 O5 @* {
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
5 B# n: o3 s6 x# F# _8 E9 ^-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
/ U; J) C8 _7 i6 ^0 @-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
`* L7 N! q# x8 E% A8 M3 n7 U3 {-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER3 Y/ l* X( X& l9 ]
-A POSTROUTING -s 172.16.80.0/24 ! -o docker0 -j MASQUERADE* M/ q: \- d# C+ B' ], w8 \
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING: \$ @4 o0 }/ G) ^/ \6 H
-A DOCKER -i docker0 -j RETURN
8 ?3 j z# k9 U-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
- l& I( u9 h/ y3 V-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-MARK-MASQ
* ~# n4 o _ G' e& f-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-SVC-GYQQTB6TY565JPRW) ]; C6 R4 X3 S" u& H
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE @& E" B5 o) L( X
-A KUBE-SEP-63GTHXGNEQIFF6GY -s 172.16.62.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
5 f5 ~! Z9 |0 z' W) ]-A KUBE-SEP-63GTHXGNEQIFF6GY -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.62.4:80$ x% E. e: u' l0 y2 M+ [4 w
-A KUBE-SEP-77PLGVXVTAKNHL2K -s 172.16.80.3/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
, O- F* x/ _, J: A6 n2 l0 |-A KUBE-SEP-77PLGVXVTAKNHL2K -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.3:80
1 d9 `1 J0 b8 e; r/ p9 s! k3 O-A KUBE-SEP-7R2ESD4YYXMXFEFZ -s 172.16.80.2/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
1 k# L# g/ `& e9 H-A KUBE-SEP-7R2ESD4YYXMXFEFZ -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.80.2:6379
: x, g9 b2 r' S u; F3 _. ^; P; t-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -s 192.168.56.250/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ; N% I" }) w2 M; L$ e
-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.56.250:6443
( U9 w- x D+ {; Z-A KUBE-SEP-LYGBYJFMWSAWPLXU -s 172.16.62.3/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
% T2 w3 [ v2 r2 j! Z-A KUBE-SEP-LYGBYJFMWSAWPLXU -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.62.3:6379
& n. S7 C! V/ }* `0 f+ ^-A KUBE-SEP-Y7WMR7EBCL4N3QJX -s 172.16.62.2/32 -m comment --comment "default/redis-master:" -j KUBE-MARK-MASQ
N1 t8 @1 p1 x7 C6 ]& L, ~6 P-A KUBE-SEP-Y7WMR7EBCL4N3QJX -p tcp -m comment --comment "default/redis-master:" -m tcp -j DNAT --to-destination 172.16.62.2:6379
) n& B1 c: D7 ?6 ?7 y) j$ _: B7 ]-A KUBE-SEP-ZDWRYP3AMCRYOGNR -s 172.16.80.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
' H$ z9 K% G& z5 V- U5 C7 W1 @-A KUBE-SEP-ZDWRYP3AMCRYOGNR -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.4:805 u ]$ ?/ n0 Y# q" M) z
-A KUBE-SERVICES -d 10.254.218.57/32 -p tcp -m comment --comment "default/frontend: cluster IP" -m tcp --dport 80 -j KUBE-SVC-GYQQTB6TY565JPRW
, n8 v7 r: ]! p-A KUBE-SERVICES -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y2 ^( o( R: k5 j% M
-A KUBE-SERVICES -d 10.254.142.174/32 -p tcp -m comment --comment "default/redis-master: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-7GF4BJM3Z6CMNVML
# t* k4 v% n8 `# a3 B-A KUBE-SERVICES -d 10.254.201.123/32 -p tcp -m comment --comment "default/redis-slave: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-AGR3D4D4FQNH4O33
* ]# T5 f" H* g( T-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS" m5 I4 Y3 v: G, E7 k% l& G
-A KUBE-SVC-7GF4BJM3Z6CMNVML -m comment --comment "default/redis-master:" -j KUBE-SEP-Y7WMR7EBCL4N3QJX
7 L: _( N4 P6 l' I8 U5 w3 |-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-LYGBYJFMWSAWPLXU
7 F: u9 s* H o7 i5 x) C-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -j KUBE-SEP-7R2ESD4YYXMXFEFZ
' H+ [' L& A: p: ]/ v" }-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-63GTHXGNEQIFF6GY; H( W. u2 H9 L% N- `$ ~
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-77PLGVXVTAKNHL2K
, _2 C- P$ D5 g/ E2 \; A-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -j KUBE-SEP-ZDWRYP3AMCRYOGNR
9 ~. c! R0 A2 P6 G$ k5 R9 |-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -j KUBE-SEP-GIMIRAR4ZAKGMA2Q7 w9 x6 [7 J! O7 C" \% N
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
6 w! T) j+ p3 l) X5 R0 S8 K7 DCOMMIT
0 r" p: b4 @ l" l6 U5 g# Completed on Sun Jan 22 00:41:01 2017. ~4 f2 Z: S' b8 N" Q( a1 i9 }4 ?
2 U+ S% @9 H% r! _/ P. ^/ W---------------------------------------------------- 尝试了本地卷 [root@centm ~]# cat redis-master-controller_with_volume.yaml3 M3 g/ Q9 s U7 B9 U; N# C( d
apiVersion: v19 ~- t" a, Z& A
kind: ReplicationController : u( m! a' R/ {& k$ q
metadata: ) q; S/ v/ w5 T6 ?1 v5 t
labels:edis-master 3 @9 F# O# P+ {6 A0 N
name: redis-master
& [ [6 t0 c3 g; Y8 fspec:
g& a1 x( N2 l* \ [+ e4 d8 k9 { replicas: 1/ j) ]7 F& {6 P! e" X r# `
selector:: ?; S5 \, ]1 o" P" S! h: q8 \
name: redis-master2 [. m {% h1 V4 K% t# u$ t' K5 l
template:
@8 Y0 g0 f: v( ]& M A5 ]- X% ? metadata:0 A8 Z8 a7 b o, c: B
labels:
$ [. \# A6 a' L$ S* O& y% v name: redis-master
; W# w7 e+ l' W/ U6 K' E spec:! l$ G# H# D/ o: J9 a8 [+ L0 [, Z
volumes: |/ ~( ^5 h" K$ E3 d5 p
- name: "gf-dir1"; r( y: z4 J9 b) \
hostPath:
( ^7 p3 N3 r. V4 V' g8 K r8 J6 o path: "/tmp"
% k9 J8 z p$ h; \7 N
4 E b- _1 d1 C' E# Q containers:
+ r8 Y) W1 m# N& R5 P - name: master % c. Q" Z9 M6 E3 k3 N( T
ports:: docker.io/kubeguide/redis-master:latest
/ W) B3 K: ]7 L) z5 v: k5 F - containerPort : 6379
: M; y! q) C$ E1 X7 L" O, n volumeMounts:
3 t( I( G. [9 Y6 { - name: "gf-dir1") ~4 y6 U; v2 u3 o, A
mountPath: "/gf1" |