k8s集群

admin

edis和docker这两个词语会自动被修改为首字母大写并链接到知识库，所以在这里先写一遍，后面就不会被改写了。

0、具体操作见 （vmware中搭建k8s），virtulbox也是相同的流程。
3 Z- f; _1 I' B% Y" `% ?1 I. Q

1、学习k8s，读的是这本书《KUBERNETES权威指南  从DOCKET到KURBERNETES实践全接触.pdf》

2、这边书刚开始讲的是在单机上搭建一个k8s的hello world，用的是centos7.

    于是我也在virtulbox中安装了centos7，并成功运行hello world。

3、然后，开始尝试集群了。在网上找了个教程，比较简单，很快就成功了，kubectl get nodes能看到各node了。

    但是，应用跑的不正常，有的节点可以访问，有的节点不可以。而且从不同节点访问，查到的数据不相同，似乎是多个独立的系统。怀疑是iptables中cluster ip的规则有问题。   

     接着，集群坏了，k8s的基础服务都启动不了。怀疑是不是因为我创建这些虚机时，用的是链接式拷贝，是不是原始的虚机安装了其它软件，导致k8s集群启动不了。

4、删掉重来，用完全拷贝的方式建立虚机。

     一切正常，但是在启动redis-master-controller.yaml时，docker中无法建立容器，

     用kubectl describe pod redis-master命令排查，发现下拉不了镜像。其实这些镜像已经存在于docker中了（搭建单机k8s时，自动pull的）。只是镜像名字前面多加了docker.io/  我修改了redis-master-controller.yaml中镜像的名字，仍然pull失败。奇怪。

5、今天定位了网络问题（见virtualbox虚拟机无法上网），网络搞通后，hello world终于正常运行了。

之前的一些疑问，有答案了

1、随便访问哪个node的ip（比如http://192.168.56.251:30001/,http://192.168.56.252:30001/）(http://192.168.56.250:30001/是不行的)，都可以访问到服务的（proxy自动转的）

2、在内部时，访问真实端口也是可以的。

[root@centm ~]# kubectl get svc
. @. _! F: Q, Q& zNAME           CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
' ^5 f4 y6 O: v& u% S$ G9 Qfrontend       10.254.218.57    <nodes>       80/TCP     5m
kubernetes     10.254.0.1       <none>        443/TCP    15d
( i( L2 f; j$ }0 Yredis-master   10.254.142.174   <none>        6379/TCP   8m
redis-slave    10.254.201.123   <none>        6379/TCP   6m

curl 10.254.218.57:80     可以通。

ping  10.254.218.57 不通

3、node中多了一个127.0.0.1 ，不知道为什么

[root@centm ~]# kubectl get nodes
NAME        STATUS     AGE
5 |- P- G9 `6 y5 W& y1 B127.0.0.1   NotReady   15d
cents1      Ready      1d
cents2      Ready      1d
! j5 E$ Z5 y" _( Y: D( \2 o4 B[root@centm ~]# ps -ef|grep kube
kube       578     1  0 Jan20 ?        00:15:55 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080
8 y; I8 f# v& ]% Dkube       588     1  0 Jan20 ?        00:01:09 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://127.0.0.1:8080
0 j* V/ `; t9 \; R1 skube      2079     1  0 Jan20 ?        00:08:11 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota
' C" C7 a4 V  e5 [! h4 {4、clusterip是一个虚ip，实际是iptables中的几个转发规则。

[root@cents2 ~]# ip a
6 `/ I8 s0 [) z/ @) p( v1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
$ t2 R/ b  M5 p4 k9 @9 s: K: {    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
: m& P; \3 U. x8 A  A    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
9 S% h- q+ u7 W% k8 e' @& z    inet6 ::1/128 scope host
7 X7 e. g$ B1 c5 g/ S- J       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:58:5d:6e brd ff:ff:ff:ff:ff:ff
; u; L. u1 ?) Z  o4 S, N    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
- @9 [/ x3 h+ T* Q) B: F       valid_lft 82058sec preferred_lft 82058sec
    inet6 fe80::b171:84d0:5173:de63/64 scope link
       valid_lft forever preferred_lft forever
' F: k+ S* w; j5 n3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:7a:24:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.252/24 brd 192.168.56.255 scope global enp0s8
$ x: k: i& T9 J% {6 m" x/ ^       valid_lft forever preferred_lft forever
. M. y6 m8 k* F# @9 ~    inet6 fe80::a00:27ff:fe7a:2414/64 scope link
5 W/ m$ x" R/ ?$ e, W3 `5 L4 A       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
+ \7 P+ N0 i  L2 I% N  O    link/ether fa:5a:c7:c5:aa:e5 brd ff:ff:ff:ff:ff:ff
. M1 s8 B% v2 V* i# z1 E    inet 172.16.80.0/16 scope global flannel.1
       valid_lft forever preferred_lft forever
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
: _$ i1 U3 [( E$ R1 ]8 P6 ?    link/ether 02:42:72:2f:1f:ae brd ff:ff:ff:ff:ff:ff
& `, }% o/ J& T3 ~2 V3 y! L    inet 172.16.80.1/24 scope global docker0
! U: x2 \/ H2 i& l       valid_lft forever preferred_lft forever
! ^, z' S4 w5 _# h9 c8 z    inet6 fe80::42:72ff:fe2f:1fae/64 scope link
       valid_lft forever preferred_lft forever
7: vethc56c1d4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
1 B8 c& C4 B6 B# \: C    link/ether 92:c8:3d:3f:b9:49 brd ff:ff:ff:ff:ff:ff link-netnsid 0
1 d0 j7 U3 B! O) v% |6 C8 c    inet6 fe80::90c8:3dff:fe3f:b949/64 scope link
       valid_lft forever preferred_lft forever
! v2 q" N5 Y, n! y( s9: vethf961994@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
    link/ether d6:be:4b:6e:26:81 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::d4be:4bff:fe6e:2681/64 scope link
       valid_lft forever preferred_lft forever
! C. s, b" F: t8 l3 a; A1 V11: vethe4cd28e@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
$ l1 ?, ^  G0 h# K$ p2 h    link/ether ee:55:55:df:4e:50 brd ff:ff:ff:ff:ff:ff link-netnsid 2
, v: x- O, T8 L. `  s/ B3 N# A! m    inet6 fe80::ec55:55ff:fedf:4e50/64 scope link
       valid_lft forever preferred_lft forever
& I  k8 d* M' D$ X% v" q5 p! l
[root@cents2 ~]# iptables-save
# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
*filter
:INPUT ACCEPT [27:4324]
:FORWARD ACCEPT [0:0]
$ K0 n( }9 j: T9 A2 Y  a) a3 }& X:OUTPUT ACCEPT [25:2821]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-SERVICES - [0:0]
-A FORWARD -j DOCKER-ISOLATION
3 ]4 _1 g/ B& i-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
6 S8 [; {7 Y: H" d2 c1 D  n  V  z" V-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
9 Y1 ?  o+ v  _( E. j+ c-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A DOCKER-ISOLATION -j RETURN
COMMIT
# Completed on Sun Jan 22 00:41:01 2017
% v! K" S2 J5 ?# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
*nat
:PREROUTING ACCEPT [0:0]
1 w5 ]/ Y& c: ]8 S# g+ Z:INPUT ACCEPT [0:0]
1 P* s2 u; O3 V1 o: Y) F1 r, w:OUTPUT ACCEPT [2:119]
# y* y: W! m/ d7 F) [" W. d; `:POSTROUTING ACCEPT [2:119]
:DOCKER - [0:0]
:KUBE-MARK-MASQ - [0:0]
, U3 h. M- a$ `+ B8 w2 G:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
& W# G2 K( C$ v$ t:KUBE-SEP-63GTHXGNEQIFF6GY - [0:0]
:KUBE-SEP-77PLGVXVTAKNHL2K - [0:0]
  {4 F0 ]2 m$ M  e) e:KUBE-SEP-7R2ESD4YYXMXFEFZ - [0:0]
:KUBE-SEP-GIMIRAR4ZAKGMA2Q - [0:0]
:KUBE-SEP-LYGBYJFMWSAWPLXU - [0:0]
, W/ A* e8 Y  M( }  F:KUBE-SEP-Y7WMR7EBCL4N3QJX - [0:0]
! w: o  X) n$ x; o$ C/ |2 A! u) I:KUBE-SEP-ZDWRYP3AMCRYOGNR - [0:0]
& P8 }$ J8 E' @$ T  D3 L# t:KUBE-SERVICES - [0:0]
9 r* }5 F( Y+ w8 c:KUBE-SVC-7GF4BJM3Z6CMNVML - [0:0]
- J& M/ e. j$ z) P:KUBE-SVC-AGR3D4D4FQNH4O33 - [0:0]
:KUBE-SVC-GYQQTB6TY565JPRW - [0:0]
- M* a' Y% ^" `7 c6 O6 k7 m:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
5 G2 M" b& ?, S. m# W7 J-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
  @- g1 L; ?0 k+ ~+ y. J5 M-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
" _' {' a. F5 u-A POSTROUTING -s 172.16.80.0/24 ! -o docker0 -j MASQUERADE
8 d. o  n9 a4 y: M" w-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A DOCKER -i docker0 -j RETURN
: ^* q3 Q+ J) v-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
: j( H  y4 [% s' ^7 H; b! R-A KUBE-SEP-63GTHXGNEQIFF6GY -s 172.16.62.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-63GTHXGNEQIFF6GY -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.62.4:80
-A KUBE-SEP-77PLGVXVTAKNHL2K -s 172.16.80.3/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
7 y/ a# D+ g: y& g8 h- V; a-A KUBE-SEP-77PLGVXVTAKNHL2K -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.3:80
-A KUBE-SEP-7R2ESD4YYXMXFEFZ -s 172.16.80.2/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
) O2 j4 o9 R1 K3 G* i9 u1 o-A KUBE-SEP-7R2ESD4YYXMXFEFZ -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.80.2:6379
-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -s 192.168.56.250/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.56.250:6443
$ o. {8 p1 ?- L! p7 _5 F-A KUBE-SEP-LYGBYJFMWSAWPLXU -s 172.16.62.3/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
/ p/ u& r4 [! j4 l-A KUBE-SEP-LYGBYJFMWSAWPLXU -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.62.3:6379
-A KUBE-SEP-Y7WMR7EBCL4N3QJX -s 172.16.62.2/32 -m comment --comment "default/redis-master:" -j KUBE-MARK-MASQ
-A KUBE-SEP-Y7WMR7EBCL4N3QJX -p tcp -m comment --comment "default/redis-master:" -m tcp -j DNAT --to-destination 172.16.62.2:6379
& f+ S/ s: y* R+ w7 d6 |; j* A-A KUBE-SEP-ZDWRYP3AMCRYOGNR -s 172.16.80.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
0 g& Z% o3 W0 P* M: s-A KUBE-SEP-ZDWRYP3AMCRYOGNR -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.4:80
) j0 z6 [$ g% H8 Q- Y- j/ _-A KUBE-SERVICES -d 10.254.218.57/32 -p tcp -m comment --comment "default/frontend: cluster IP" -m tcp --dport 80 -j KUBE-SVC-GYQQTB6TY565JPRW
" e5 E- @7 Y3 m/ q: ?-A KUBE-SERVICES -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.254.142.174/32 -p tcp -m comment --comment "default/redis-master: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-7GF4BJM3Z6CMNVML
-A KUBE-SERVICES -d 10.254.201.123/32 -p tcp -m comment --comment "default/redis-slave: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-AGR3D4D4FQNH4O33
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-7GF4BJM3Z6CMNVML -m comment --comment "default/redis-master:" -j KUBE-SEP-Y7WMR7EBCL4N3QJX
1 j7 L7 S. w. R-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-LYGBYJFMWSAWPLXU
. Z* ?6 Y- n$ c& Q5 J* T-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -j KUBE-SEP-7R2ESD4YYXMXFEFZ
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-63GTHXGNEQIFF6GY
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-77PLGVXVTAKNHL2K
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -j KUBE-SEP-ZDWRYP3AMCRYOGNR
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
7 X/ t) a% i" r; a0 NCOMMIT
6 ?& \: N6 l4 z: M4 P; U# Completed on Sun Jan 22 00:41:01 2017
/ q! \, B& k  ^0 V
----------------------------------------------------

尝试了本地卷

[root@centm ~]# cat redis-master-controller_with_volume.yaml
apiVersion: v1
% ?9 e+ Z+ C' C; Y/ d+ y  Mkind: ReplicationController  
# x, y& ?4 S6 h8 L! [metadata:                              
) V1 N5 G) C" [9 \0 t1 C0 Y. t  labels:edis-master  
    name: redis-master
spec:
  replicas: 1
  selector:
    name: redis-master
2 N6 B$ I9 w" e' o  template:   
2 ~$ n+ Z2 J! E1 n3 s/ H6 `    metadata:
      labels:
3 P9 o  j( q- A5 T/ i        name: redis-master
3 S( L3 H* F" L; Q    spec:
) q  k/ Z6 n, ]7 B3 U3 b        volumes:
/ u" I( H2 t. j3 m7 N, @  O        - name: "gf-dir1"
          hostPath:
            path: "/tmp"

' ^, ]' V$ U, ]# g  d9 g        containers:
        - name: master         
          ports:: docker.io/kubeguide/redis-master:latest   
          - containerPort : 6379
          volumeMounts:
; c; H6 o" o3 X- c! p( ]( Q          - name: "gf-dir1"
( e  e- V9 w( c" L# ]; I- o            mountPath: "/gf1"

admin

export PATH=$PATH:/usr/local/bin
, u0 X& N! j: f; l+ S4 W& Q# J# alias kk="kubectl -n kube-system"
/ j3 O! n$ p4 H# alias kp="kubectl -n kube-public"
9 A' c. x. N2 T- j) T6 i2 x# alias kis="kubectl -n istio-system"
# alias ks="kubectl -n kubesphere-system"
/ I* ~" r, y! [# alias km="kubectl -n monitoring"
/ l* R; a& J3 r* X* V8 Y# alias kcm="kubectl -n caas-monitoring-system"
# alias kcs="kubectl -n caas-system"
' ?) V# K- \. G- |  G% K' B# alias kcl="kubectl -n caas-logging"
# K: ?6 l. C5 J% X6 S9 r' {# alias kcd="kubectl -n caas-devops"
# alias kc="kubectl -n caas"
# w( l# I  z5 O* l- U( e. ^0 r# alias kcp="kubectl -n cpcs"
# alias kop="kubectl -n openstack"
# alias kcmp="kubectl -n cmp"
# alias kks="kubectl -n karmada-system"
# alias kocm="kubectl -n open-cluster-management"
# alias kocma="kubectl -n open-cluster-management-agent"
# alias kt="kubectl -n test"
# alias kv="kubectl -n vela-system"
" C! F8 \  o! q" @7 {& Q# alias kfs="kubectl -n flux-system"
# alias kgo="kubectl -n gpu-operator"
9 r, j8 r6 I' g; B, q$ X# alias ki="kubectl -n infra"
# alias k="kubectl"
, p2 ~9 o# O, s# alias ctr="ctr --namespace k8s.io"
. ?5 x' Q8 W% a/ P* o# alias nerdctl="nerdctl --namespace k8s.io"
& G& Q$ u" w2 o/ F( y* Y# ps -ef|grep etcd
+ s  _0 s' o. N
: v  ]; u4 F* I6 Z3 n% E; W# ps -ef|grep etcd
# journalctl -xu kubelet -f
# ps -ef|grep etcd
9 A3 n6 w3 l5 D  ~
& }" @/ R, X7 ~) R' L6 W0 \' Q
# ~  K% R% t* @# {2 l# cd etcd/
# ls
: x( u& q2 [* D# mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/
# ls
# rm -rf member/
# ls
# cd /etc/kubernetes/
7 C8 k* }* N7 A# U# ls
# mv etcd.yaml manifests/
/ p! M: V5 ?; ~# @3 R/ q2 d# ls
# journalctl -xu kubelet -f

+ Z( p6 c  j3 q& ^# [
# ps -ef|grep etcd
( e  q5 E7 X% D7 ~- H1 p0 M* h# ls
5 c) n4 f% ^4 G- X# s% K# cd manifests/
# ls
; B" X" w& y1 U4 i2 K1 ?5 }# mv * ../
# ls
! v. l6 M/ g2 W  D# ls
' @5 }3 n* o5 Q" Q# cd
# cd /etc/kube
# cd /etc/kubernetes/
# ls
' b! P4 Q- L* k7 t) Q* k# mv etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml manifests/
# journalctl -xu kubelet -f

" Q1 ^9 D4 S( F# y/ {: y# ps -ef|grep etcd
; U9 @9 h9 C1 }. m  R; p) n# journalctl -xu kubelet -f|grep etcd
# k get po
  P# r; T7 f. ^9 d5 V# journalctl -xu kubelet -f|grep etcd
# journalctl -xu kubelet -f
" d3 b, J) V; s$ C3 S" a. l# journalctl -xu kubelet -f|grep etcd
# ps -ef|grep etcd
' q9 f' `( i+ i& T2 B4 c5 O2 l  c- s0 }# ls
: y. }5 H7 B  o3 k# cat manifests/etcd.yaml


. c, X' f% P' E! L5 M9 F2 m( T
' H  n7 L+ W9 z1 [) ]* @2 j# systemctl stop kubelet
# ls
9 }1 M0 J' A: W! g4 o# rm -rf /data/etcd/member/
1 ^' z. a( n+ }% R* t/ Z# ls /data/etcd/
# ls
% ~, E# R$ \3 s& ^, R# systemctl status kubelet
4 W- h0 y9 f) P" Q, i/ t& [# systemctl start kubelet
9 b5 ?! U, v# `1 X# ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=https://10.166.7.5:2379,https://10.166.7.6:2379,https://10.166.7.7:2379 endpoint status --write-out=table
# ps -ef|grep etcd
* [( u/ ~& a% {2 P: x3 T
& S. j- O7 [3 b' Z  v# s# ps -ef|grep etcd
: o8 {/ i- _0 |$ d4 K# k# ls /data/etcd/member/
+ J* t, O" K5 d2 b0 \# ls /data/etcd/member/snap/
9 k* z$ t# h% b5 _8 E- H# ls -lhS /data/etcd/member/snap/
! X. t2 j5 h( [6 r8 O- @# ps -ef|grep etcd
3 n/ l' ?# r6 A7 a- L- y3 ?( J4 c/ b) X+ ]
# cat /etc/hosts
# vi /etc/hosts
5 z9 Q2 M- B- n4 V3 A# kk get po
  ^% n0 o7 A, |- X2 k; o1 e# kk logs -f etcd-cmp-server02
; z# Z9 _- K) H- e* c/ U+ p
5 M8 [7 z1 C* y  J# ls
! J' d1 \8 f- Y& P# cd manifests/
# ls
# mv etcd.yaml ../
# cd ../
% ^7 N7 O- E6 l  c& N/ @# ls
# q) H4 T/ M+ x& H' {# ps -ef|grep etcd
# ls
# mv etcd.yaml manifests/
# kk get po
# k get po
# k get node
# kk get po
! L! \5 l) ]+ r0 {# kk logs -f etcd-cmp-server02

, x0 P+ u& E8 |5 e0 T. U# cd manifests/
, }) i) N$ X$ s7 x5 @) Y# ^2 b# ls
& [. n. z3 E1 U# mv etcd.yaml ../
% \0 u* \. ?1 ?* P- `+ v$ C2 }; S# rm -rf /data/etcd/
$ I( F/ u, L4 j# ls
# cd ../
+ r$ q0 E3 y% q5 v# z/ u# ls
# mv etcd.yaml manifests/
+ y$ p3 G3 \9 j% N6 B# kk get po
# kk logs -f etcd-cmp-server02
# kk get po
; n$ Q: Z" J$ T  t% K: ~% q; m# systemctl restart kubelet
# kk get po
: \" l+ I: U2 A" U' ?% w: o# kk get po
# cat /etc/hosts
# kk logs -f etcd-cmp-server02
# kk logs -f etcd-cmp-server02
* W7 G" T9 h7 j# kk logs -f etcd-cmp-server02
# kk logs -f etcd-cmp-server02
# ls
# cd manifests/
" e/ @9 x+ h1 X0 a2 X# ls
& O1 e) Y% w1 t# B# vi etcd.yaml
2 o, v' s: u, |6 p# cat etcd.yaml
# vi etcd.yaml
: z" W6 L) H; I2 N$ ~. e# ls
# mv etcd.yaml ../
9 |. {' |% L3 t+ ^* v# ls
* M# G5 k8 L* @# cd ../
# ls
1 `/ ?1 R% m1 T1 I. D3 z# mv etcd.yaml manifests/
# ls
, u. H; r4 b& P+ J( g, Q- n# kk get po
5 |. ?3 d& X/ `: j+ \' k# kk get po
/ s5 ?+ I+ I4 p' n# kk get po
' u+ G; Z3 B" r9 F# kk get po
# kk get po
# kk get po
( F5 b7 U2 \% g. |# kk get po
$ n6 c0 H/ L9 p. j$ \# kk logs -f etcd-cmp-server02
! a, \6 s* [. W* Y: O' M# P# ls
# mv manifests/etcd.yaml .
% N' R: r* k- w( g) a2 x/ @# rm -rf /data/etcd/
: q2 H' T: E  C- r2 e& e% W# ls
4 R3 t9 C/ \) j, K- S0 R# mv etcd.yaml manifests/
# cd manifests/
# F8 S& j+ ^! m/ p  l. i  W# ls
( L1 U: W3 Z$ R2 D# n# E' D( B# s
# cd ../
# ls
# kk get po

# kk get po
# kk get po
# kk get po
# kk logs -f etcd-cmp-server02
# kk get po
" g/ K4 l# z5 _2 C1 J7 Y# kk get po
0 S" |' {5 {% z/ Q7 Y# kk logs -f etcd-cmp-server02
# ls
8 D# Q. T/ ^$ A6 V# mv manifests/etcd.yaml .
3 J( x. i7 l) Q3 Z9 ^
# kk get po
# ls
# rm -rf /data/etcd/
& p; @1 ^, x3 a: r# ls
! _; u6 }* K& L: D# mv etcd.yaml manifests/
$ r8 f- Z6 N/ O/ R; i9 f, @) m#
/ [8 f# N. ?* u4 Y8 U4 p& w' P# kk get po
) `& X3 J) w$ f) p( A# kk logs -f etcd-cmp-server02
, ?1 H2 Z$ \, a7 A# ls
5 T/ p4 O: P/ U# @! \5 X# kk get po
, T, f8 R$ W! }1 o9 f" `* J' o2 a# kk logs -f kube-apiserver-cmp-server02
4 W8 I. {$ l2 X3 k( ^( K4 d. ?
! x: B, k( e9 n/ b# kk get po
: S# }; ?& n: q+ g, V2 D' g# kk logs -f etcd-cmp-server02
. ~+ M  \& [& D3 ?% F# ls
# cat manifests/etcd.yaml
# ls
, X* {% Z# H3 R6 N$ c4 e$ v6 O# ls
( O. H4 ^: |* Q2 k9 j; D# ls
# ls
# ls
# ls
# ls
3 V6 E9 ]' l  |5 h+ j: q# ls
# kk get po
( j4 w2 [  ]2 h2 B  i3 T# kk logs -f etcd-cmp-server02
# kk get po
4 |) V( O* G8 [! f) H9 w
# kk get po
9 P5 b: X: l- a2 O# M. o/ J; R" _# kk logs -f etcd-cmp-server02
, ^( L4 e) W/ S6 `% f$ o, ~# kk logs -f etcd-cmp-server02
% \$ b4 c$ H3 Z) h! G$ J" k# kk get po
+ p* ^; t2 w, O" X$ ?7 U# k get po
# kk get po
; r- Z. p- r, d# ls
5 P+ x7 Y8 o3 p# cd manifests/
# ls
6 X: e: |  r6 _/ I$ n9 U$ s7 t  n# kk get po
  ]6 S, T3 c! @# kk logs -f kube-apiserver-cmp-server02
/ Z+ v: s% n. w( C8 U) O+ ]" T# kk get po
# ls
9 Y: V& r0 R$ E# mv kube-apiserver.yaml ../
% r7 h+ r) Q/ t* T# cd ../
" d7 p5 n/ S6 I+ ^' T9 X# mv kube-apiserver.yaml manifests/
: C6 }" I9 k0 `/ r5 M4 K1 g7 H# kk get po
" u0 m) \- h% Q" y. z: o# kk get po
# kk logs -f kube-apiserver-cmp-server02
# ps -ef|grep etcd
# ss -tunlp|grep 2379
; s- e, O4 @4 w* j/ D0 y" W# kk logs -f kube-apiserver-cmp-server02
# kk get po
- e/ ?; \2 ~" e$ X1 ~: q# kk get po -owide
# s
# kk get po
' T" S6 l6 ?$ g# ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=https://10.166.7.5:2379,https://10.166.7.6:2379,https://10.166.7.7:2379 endpoint status --write-out=table
# kk get po
# kk get po
1 f* c- E5 d' c/ }& o# kk get po
( f" ^+ _/ W6 Q& t2 O, r# kk logs -f kube-apiserver-cmp-server02
4 a" X  w, \8 w; g3 w0 o# tennet 127.0.0.1:2379
! }' s  A3 q: ^* k
4 Y( V0 h9 v  l) W, a% ^# ls
# cd manifests/
9 h: ^+ D# G8 q( J4 |3 Z  I! o' p# ls
$ `$ Q+ j/ A' _8 {& f% u8 F3 C# cat kube-apiserver.yaml
# kk get po
# kk get po

# kk get po
# df -h
# ls
# cd /
# cd
# cd /home/devops/
# ls
7 k6 O# S3 {$ R' V/ r& I# cd
7 \* G  v! q- B7 s# ls
5 ?6 S+ v  W/ R! P7 A* ]+ b* V# du -sh *
# cd /
# du -sh *
# cd data/
! a0 g2 @/ Q% x# ls
# du -sh *
$ K: n6 p2 c& j- l
8 c; j2 T2 k$ y2 u/ m7 x4 T# journalctl -xefu kubelet
# ;s
1 n8 x1 T$ E' u1 k  |# /usr/local/bin/nerdctl -n k8s.io tag caas4/keystone-sync-db:latest 10.166.7.5:30443/caas/keystone-sync-db:latest
% o2 B: ~* L/ L2 ^; N0 q  S# /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/caas/keystone-sync-db:latest
& @0 b* U' B3 x: r* I# /usr/local/bin/nerdctl -n k8s.io login 10.166.7.5:30443
: n. _, ~( R% U% d4 y! W; u8 K# /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/caas/keystone-sync-db:latest
) ]+ b3 K: A2 J9 X4 B# /usr/local/bin/nerdctl -n k8s.io tag caas4/perception 10.166.7.5:30443/caas/perception:latest
# /usr/local/bin/nerdctl -n k8s.io tag sameersbn/gitlab:13.10.3 10.166.7.5:30443/cmp/gitlab:13.10.3
+ L7 G: b% S2 E9 P5 d: c) {  J#  /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/cmp/gitlab:13.10.3
) E9 t% {( u& J& e, g9 \# /usr/local/bin/nerdctl -n k8s.io tag sameersbn/postgresql:12-20200524 10.166.7.5:30443/cmp/postgresql:12-20200524
# /usr/local/bin/nerdctl -n k8s.io tag cmp/redis:5.0.6  10.166.7.5:30443/cmp/redis:5.0.6
) h0 e8 P( Y9 W: ?: b( |# c7 b# /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/cmp/redis:5.0.6
7 r% y4 [; J' N2 N' a# /usr/local/bin/nerdctl -n k8s.io tag busybox  10.166.7.5:30443/cmp/busybox:latest
# /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/cmp/busybox:latest
7 U5 b  D( p( [% [  M! q# kubectl get node --show-labels
# journalctl -xefu kubelet
; v% W7 h' n* q' M% v. |  v# journalctl -xefu kubelet
% I! N; B2 N( a- z3 X8 G0 R# journalctl -xefu kubelet
( ?. U1 q. f- q* k, W: C. _# journalctl -xefu kubelet
# kubeadm reset -f
# vi /etc/hosts
# cat /etc/hosts
; H9 S5 E: h4 o0 @2 i; z$ r# systemctl stop firewalld || true
# systemctl disable firewalld || true
# setenforce 0
# sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config
3 e+ S4 E/ z1 J* H' ]# c' O# modprobe br_netfilter && modprobe nf_conntrack
# cat > /etc/sysctl.d/98-k8s.conf << EOF
  _% K/ G( R8 m1 o! ?5 }- L! o- R# net.netfilter.nf_conntrack_tcp_be_liberal = 1
# net.netfilter.nf_conntrack_tcp_loose = 1
# net.netfilter.nf_conntrack_max = 524288
# net.netfilter.nf_conntrack_buckets = 131072
# net.netfilter.nf_conntrack_tcp_timeout_established = 21600
# net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
# net.ipv4.neigh.default.gc_thresh1 = 1024
# net.ipv4.neigh.default.gc_thresh2 = 2048
# net.ipv4.neigh.default.gc_thresh3 = 4096
# vm.max_map_count = 262144
# net.ipv4.ip_forward = 1
# net.ipv4.tcp_timestamps = 1
( c- E4 _* X9 J+ \9 d: H1 [) a1 K+ `# net.bridge.bridge-nf-call-ip6tables = 1
  o* k7 Z" a7 C3 K* p- [+ q3 W# net.bridge.bridge-nf-call-iptables = 1
3 ?5 X) Q* L7 N, ?2 R, e! x6 e- ~* \# net.ipv6.conf.all.forwarding=1
- D- X4 C3 i. f6 c8 G: g2 Z# fs.file-max=1048576
5 R/ K% k* h4 w# K6 W, q# fs.inotify.max_user_instances = 8192
# k9 t  D5 F, c5 r" x2 p7 x# fs.inotify.max_user_watches = 524288
# EOF

# cat > /etc/security/limits.d/98-k8s.conf << EOF
# * soft nproc 65535
+ H( }7 |7 V# ^6 S: \6 u# * hard nproc 65535
( I* ?, G9 c' E" P# * soft nofile 65535
# * hard nofile 65535
6 E% j* y2 X! a/ B! @# EOF
  X' g) `( l3 l1 g% P3 a
  s) a% H9 z, s. x1 P4 T# sysctl --system
  n; f5 d$ C! Q; _0 ]# sysctl -p
: n! ]) n$ p( m9 P# swapoff -a
# sed -i /swap/d /etc/fstab
) i' R, V/ e/ E* B7 B, q, a# kubeadm join apiserver.cluster.local:6443 --token 25omv5.cqx4gnuhoyh7nwxy --discovery-token-ca-cert-hash sha256:eaf7ee645c5cc0af1782df11c38519b84d643f06d9c6613009047e9b2e275524
" N/ w8 A/ w' _' K( p) Z# kubeadm reset -f
* K( @+ q! _8 `# kubeadm reset -f
# kubeadm join apiserver.cluster.local:6443 --token 25omv5.cqx4gnuhoyh7nwxy --discovery-token-ca-cert-hash sha256:eaf7ee645c5cc0af1782df11c38519b84d643f06d9c6613009047e9b2e275524
. L- r6 ?" L+ t. [/ R) p; d0 T# /usr/local/bin/nerdctl -n k8s.io images
# /usr/local/bin/nerdctl -n k8s.io tag rancher/local-path-provisioner:v0.0.24  10.166.7.5:30443/cmp/local-path-provisioner:v0.0.24
. a4 p4 `9 f' b) G' _# /usr/local/bin/nerdctl -n k8s.io push  10.166.7.5:30443/cmp/local-path-provisioner:v0.0.24
0 T5 m6 Z3 e2 H  Y. T% d  F8 W" f1 G

# cd local-path-provisioner/
% z" a- U5 k  i/ s$ a1 S* [( C# ls
- a+ F2 K% k9 s* d7 v' H# du -sh *
9 T; H- X, k% B$ K& B4 u9 ?! y# cd pvc-cedd6757-22af-46a3-a863-cdc4a02af016_caas-system_minio/
( l6 L/ E) K$ j5 t# du -sh *
* u) y- T% d; q3 o4 B% M# cd mgmt/
3 }8 a$ b! L+ B+ P; _. a# ls
; `/ d- n5 `' L( `# cat /etc/kubernetes/kubelet.conf
# cat /etc/kubernetes/kubelet.conf
# cat /etc/kubernetes/controller-manager.conf
, H$ ~1 l) K1 z1 h! r) j# cat /etc/kubernetes/kubelet.conf

) @! g; M" h( O- l9 j& q+ r
' g, r1 b# z4 a, c& ^' d# cat /etc/kubernetes/kubelet.conf
. `, l) w( J' W/ ]5 L4 {% I# cat /var/lib/kubelet/config.yaml