k8s集群

admin

edis和docker这两个词语会自动被修改为首字母大写并链接到知识库，所以在这里先写一遍，后面就不会被改写了。

0、具体操作见 （vmware中搭建k8s），virtulbox也是相同的流程。

1、学习k8s，读的是这本书《KUBERNETES权威指南  从DOCKET到KURBERNETES实践全接触.pdf》

2、这边书刚开始讲的是在单机上搭建一个k8s的hello world，用的是centos7.

    于是我也在virtulbox中安装了centos7，并成功运行hello world。

3、然后，开始尝试集群了。在网上找了个教程，比较简单，很快就成功了，kubectl get nodes能看到各node了。

    但是，应用跑的不正常，有的节点可以访问，有的节点不可以。而且从不同节点访问，查到的数据不相同，似乎是多个独立的系统。怀疑是iptables中cluster ip的规则有问题。   

     接着，集群坏了，k8s的基础服务都启动不了。怀疑是不是因为我创建这些虚机时，用的是链接式拷贝，是不是原始的虚机安装了其它软件，导致k8s集群启动不了。

4、删掉重来，用完全拷贝的方式建立虚机。

     一切正常，但是在启动redis-master-controller.yaml时，docker中无法建立容器，

     用kubectl describe pod redis-master命令排查，发现下拉不了镜像。其实这些镜像已经存在于docker中了（搭建单机k8s时，自动pull的）。只是镜像名字前面多加了docker.io/  我修改了redis-master-controller.yaml中镜像的名字，仍然pull失败。奇怪。

5、今天定位了网络问题（见virtualbox虚拟机无法上网），网络搞通后，hello world终于正常运行了。

之前的一些疑问，有答案了

1、随便访问哪个node的ip（比如http://192.168.56.251:30001/,http://192.168.56.252:30001/）(http://192.168.56.250:30001/是不行的)，都可以访问到服务的（proxy自动转的）

2、在内部时，访问真实端口也是可以的。

[root@centm ~]# kubectl get svc
* E$ ?+ \( T# i+ u- W; xNAME           CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
" n* r+ C- C) A' zfrontend       10.254.218.57    <nodes>       80/TCP     5m
) ~* s! _7 m, t0 Hkubernetes     10.254.0.1       <none>        443/TCP    15d
redis-master   10.254.142.174   <none>        6379/TCP   8m
+ j* h" ]6 w$ \$ _" k3 s: wredis-slave    10.254.201.123   <none>        6379/TCP   6m

curl 10.254.218.57:80     可以通。

ping  10.254.218.57 不通

3、node中多了一个127.0.0.1 ，不知道为什么
7 a, b! ]. _1 o8 e7 G4 s+ X# ]

[root@centm ~]# kubectl get nodes
NAME        STATUS     AGE
/ u& Q3 S5 n- U2 G. e! i; i& v127.0.0.1   NotReady   15d
" @# z& x- s; Q$ f0 N( x' ncents1      Ready      1d
- l% I6 {# U5 E* X6 S2 C( Z) wcents2      Ready      1d
[root@centm ~]# ps -ef|grep kube
kube       578     1  0 Jan20 ?        00:15:55 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080
kube       588     1  0 Jan20 ?        00:01:09 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://127.0.0.1:8080
kube      2079     1  0 Jan20 ?        00:08:11 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota
4、clusterip是一个虚ip，实际是iptables中的几个转发规则。

[root@cents2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
/ Y) k$ G) n5 u, Z- L       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
4 E8 \6 L& _  _0 C8 q       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
0 t% I, d3 |- f& g) c' k) |9 K/ w    link/ether 08:00:27:58:5d:6e brd ff:ff:ff:ff:ff:ff
! P8 Z9 ]7 M$ R5 ]    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 82058sec preferred_lft 82058sec
; d8 F- b7 O3 h% [    inet6 fe80::b171:84d0:5173:de63/64 scope link
" c; N9 J8 W4 Y% \% {2 r       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:7a:24:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.252/24 brd 192.168.56.255 scope global enp0s8
2 r2 c$ _# i( l0 H2 v' ~- C& v       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe7a:2414/64 scope link
4 d! ^. @7 w, x7 H" ^       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
: p4 U. k# t0 r+ t+ Q4 G0 b% g' d    link/ether fa:5a:c7:c5:aa:e5 brd ff:ff:ff:ff:ff:ff
+ n8 D$ B- [1 |' T! R2 e    inet 172.16.80.0/16 scope global flannel.1
       valid_lft forever preferred_lft forever
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
; S0 K$ U, a. {6 `% ]" d    link/ether 02:42:72:2f:1f:ae brd ff:ff:ff:ff:ff:ff
    inet 172.16.80.1/24 scope global docker0
/ S  r. \8 V5 o. i       valid_lft forever preferred_lft forever
    inet6 fe80::42:72ff:fe2f:1fae/64 scope link
9 V$ M5 W" s* k* {       valid_lft forever preferred_lft forever
7: vethc56c1d4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
+ L* k8 r& g+ T' b! r) l7 `    link/ether 92:c8:3d:3f:b9:49 brd ff:ff:ff:ff:ff:ff link-netnsid 0
2 A( p- F8 ^, d  Z7 _% o% r    inet6 fe80::90c8:3dff:fe3f:b949/64 scope link
       valid_lft forever preferred_lft forever
- p  z  G- R+ B9: vethf961994@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
    link/ether d6:be:4b:6e:26:81 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::d4be:4bff:fe6e:2681/64 scope link
2 \' a, t; I; P2 N& v7 m- y' {       valid_lft forever preferred_lft forever
11: vethe4cd28e@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP
9 W$ P9 e" U1 T& p  Z# Z    link/ether ee:55:55:df:4e:50 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::ec55:55ff:fedf:4e50/64 scope link
       valid_lft forever preferred_lft forever
0 _) R9 F" D$ N$ K' f
; [2 g+ Q/ J% Y+ n6 x) X( E1 y! m+ @[root@cents2 ~]# iptables-save
0 D6 I0 w/ u  d  c- C# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
*filter
0 @% [1 f, l  q4 A:INPUT ACCEPT [27:4324]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [25:2821]
. q! \- d$ [  _' Z4 E:DOCKER - [0:0]
& \7 F  ^+ M; W& l:DOCKER-ISOLATION - [0:0]
4 D/ B9 {4 ~: U  l:KUBE-SERVICES - [0:0]
-A FORWARD -j DOCKER-ISOLATION
9 M. L- m% R: k( l1 n. N" C8 b-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
, b/ C9 [: L1 `3 F& D% `-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
. j6 {0 z) G" j/ G/ @  A' e* n-A DOCKER-ISOLATION -j RETURN
COMMIT
$ ~! F3 j  Z; e8 m9 W  d# Completed on Sun Jan 22 00:41:01 2017
# Generated by iptables-save v1.4.21 on Sun Jan 22 00:41:01 2017
6 L+ O4 N) w8 h6 T) c/ K( M*nat
! u' x0 S! h# B; w3 ~8 n7 T- k:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [2:119]
:POSTROUTING ACCEPT [2:119]
:DOCKER - [0:0]
:KUBE-MARK-MASQ - [0:0]
# o4 q4 A  Z+ z8 q, U. Y  T:KUBE-NODEPORTS - [0:0]
6 o( I$ l: h* Z:KUBE-POSTROUTING - [0:0]
# r6 x5 f: F5 e- x:KUBE-SEP-63GTHXGNEQIFF6GY - [0:0]
:KUBE-SEP-77PLGVXVTAKNHL2K - [0:0]
:KUBE-SEP-7R2ESD4YYXMXFEFZ - [0:0]
5 j1 I& _5 h5 Y9 g5 o:KUBE-SEP-GIMIRAR4ZAKGMA2Q - [0:0]
:KUBE-SEP-LYGBYJFMWSAWPLXU - [0:0]
:KUBE-SEP-Y7WMR7EBCL4N3QJX - [0:0]
- B7 [, I5 W3 S& O+ T+ Z" x$ a:KUBE-SEP-ZDWRYP3AMCRYOGNR - [0:0]
# J( w/ s, b8 ~, H:KUBE-SERVICES - [0:0]
:KUBE-SVC-7GF4BJM3Z6CMNVML - [0:0]
:KUBE-SVC-AGR3D4D4FQNH4O33 - [0:0]
:KUBE-SVC-GYQQTB6TY565JPRW - [0:0]
2 i* `3 _0 X! [" E- `:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
7 \( `2 D) n* x7 ?. E9 G-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
$ g$ B* d; i  I) H8 r6 ~-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.16.80.0/24 ! -o docker0 -j MASQUERADE
- y) t! ?. i$ A/ W) M5 v-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
8 e+ m) A- j7 P-A DOCKER -i docker0 -j RETURN
3 y: F; _' Y  ^; c-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-MARK-MASQ
% a" |' w) }) p% N( d! d+ Z-A KUBE-NODEPORTS -p tcp -m comment --comment "default/frontend:" -m tcp --dport 30001 -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-63GTHXGNEQIFF6GY -s 172.16.62.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-63GTHXGNEQIFF6GY -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.62.4:80
& W% }% M1 V8 w-A KUBE-SEP-77PLGVXVTAKNHL2K -s 172.16.80.3/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
8 P4 y/ [/ u% ]5 J4 {6 [-A KUBE-SEP-77PLGVXVTAKNHL2K -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.3:80
% T7 x7 x" T% f6 W-A KUBE-SEP-7R2ESD4YYXMXFEFZ -s 172.16.80.2/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
-A KUBE-SEP-7R2ESD4YYXMXFEFZ -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.80.2:6379
-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -s 192.168.56.250/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
" J8 z# `& {. r* g9 M9 s-A KUBE-SEP-GIMIRAR4ZAKGMA2Q -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 192.168.56.250:6443
-A KUBE-SEP-LYGBYJFMWSAWPLXU -s 172.16.62.3/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
% |! F* G1 O9 u% \2 \-A KUBE-SEP-LYGBYJFMWSAWPLXU -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 172.16.62.3:6379
, E) K! ^0 _' t. [6 \( V-A KUBE-SEP-Y7WMR7EBCL4N3QJX -s 172.16.62.2/32 -m comment --comment "default/redis-master:" -j KUBE-MARK-MASQ
-A KUBE-SEP-Y7WMR7EBCL4N3QJX -p tcp -m comment --comment "default/redis-master:" -m tcp -j DNAT --to-destination 172.16.62.2:6379
-A KUBE-SEP-ZDWRYP3AMCRYOGNR -s 172.16.80.4/32 -m comment --comment "default/frontend:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZDWRYP3AMCRYOGNR -p tcp -m comment --comment "default/frontend:" -m tcp -j DNAT --to-destination 172.16.80.4:80
, Q# `8 h4 l8 ]2 L-A KUBE-SERVICES -d 10.254.218.57/32 -p tcp -m comment --comment "default/frontend: cluster IP" -m tcp --dport 80 -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-SERVICES -d 10.254.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
: x* h3 L4 f9 k+ C' K-A KUBE-SERVICES -d 10.254.142.174/32 -p tcp -m comment --comment "default/redis-master: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-7GF4BJM3Z6CMNVML
-A KUBE-SERVICES -d 10.254.201.123/32 -p tcp -m comment --comment "default/redis-slave: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-AGR3D4D4FQNH4O33
; P' L& m; f* [-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-7GF4BJM3Z6CMNVML -m comment --comment "default/redis-master:" -j KUBE-SEP-Y7WMR7EBCL4N3QJX
. x$ r9 J" g. }' X+ w-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-LYGBYJFMWSAWPLXU
-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -j KUBE-SEP-7R2ESD4YYXMXFEFZ
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-63GTHXGNEQIFF6GY
2 ]) Z/ x8 H0 e-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-77PLGVXVTAKNHL2K
-A KUBE-SVC-GYQQTB6TY565JPRW -m comment --comment "default/frontend:" -j KUBE-SEP-ZDWRYP3AMCRYOGNR
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 180 --reap --name KUBE-SEP-GIMIRAR4ZAKGMA2Q --mask 255.255.255.255 --rsource -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-GIMIRAR4ZAKGMA2Q
! C' ?  G9 W0 l+ c, P: eCOMMIT
# Completed on Sun Jan 22 00:41:01 2017
. H' t6 `- e, Z* [
----------------------------------------------------

尝试了本地卷

[root@centm ~]# cat redis-master-controller_with_volume.yaml
apiVersion: v1
6 U, F. i+ _# a. ^8 h& n: D8 Xkind: ReplicationController  
metadata:                              
  labels:edis-master  
, J) _3 w5 ~, z. Q    name: redis-master
3 N3 }9 Y  p) ]; I1 |spec:
  replicas: 1
  selector:
    name: redis-master
  template:   
    metadata:
* ^2 t7 o% d8 v' n+ p      labels:
4 g! K. b2 e: M- Y3 U4 W        name: redis-master
    spec:
        volumes:
        - name: "gf-dir1"
          hostPath:
  L& r! X% a# g4 p            path: "/tmp"
6 T5 E6 u) Q4 y% \  _( \* s
) ~+ e# K1 p& H        containers:
! \" {- G& l3 }3 B5 Q        - name: master         
7 Z3 G( v3 l; L/ b: h+ [: h          ports:: docker.io/kubeguide/redis-master:latest   
! F* h. x' ^4 O; m          - containerPort : 6379
          volumeMounts:
7 W* p7 f- G- L  o  u          - name: "gf-dir1"
            mountPath: "/gf1"

admin

export PATH=$PATH:/usr/local/bin
# alias kk="kubectl -n kube-system"
2 t. c/ z( {! t& q# alias kp="kubectl -n kube-public"
6 G5 u" U- o9 W" |. P# alias kis="kubectl -n istio-system"
# alias ks="kubectl -n kubesphere-system"
# alias km="kubectl -n monitoring"
# alias kcm="kubectl -n caas-monitoring-system"
0 c* B$ F  w! p. u; I3 o$ L  N6 n# alias kcs="kubectl -n caas-system"
+ d+ _  {; z3 o4 \+ M# alias kcl="kubectl -n caas-logging"
9 p. ?' C, A9 W, h9 x8 C8 v; C# alias kcd="kubectl -n caas-devops"
# alias kc="kubectl -n caas"
# alias kcp="kubectl -n cpcs"
/ a* I1 K0 o; }& X# alias kop="kubectl -n openstack"
# alias kcmp="kubectl -n cmp"
; I( T  {1 }3 U9 z$ ?# alias kks="kubectl -n karmada-system"
0 c! q7 R. b) b8 w% J+ o# o# alias kocm="kubectl -n open-cluster-management"
# alias kocma="kubectl -n open-cluster-management-agent"
# alias kt="kubectl -n test"
4 p6 I7 Q0 z% A2 _( }- N# alias kv="kubectl -n vela-system"
# alias kfs="kubectl -n flux-system"
: D& P+ g& G, }0 B7 J# alias kgo="kubectl -n gpu-operator"
/ `6 p) f9 h4 Y8 W) i% G# alias ki="kubectl -n infra"
# alias k="kubectl"
# alias ctr="ctr --namespace k8s.io"
/ `" o9 i! G/ m( y9 M! S& w' P# alias nerdctl="nerdctl --namespace k8s.io"
% {3 ?1 R) D$ p# @* G# ps -ef|grep etcd
  d6 u5 _& D$ i9 P; [
# ps -ef|grep etcd
# journalctl -xu kubelet -f
# ps -ef|grep etcd
( [0 Q: d! `9 \

$ g( C2 v: Y- ?5 g) G; e# cd etcd/
7 y# T8 Q- U5 C5 i9 Y# ls
# mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/
# ls
% j! E4 k& C9 O' e. i# rm -rf member/
# ls
+ |2 R+ q" j& n3 c* E# cd /etc/kubernetes/
# ls
  H: h& s4 r: R9 {; E# mv etcd.yaml manifests/
# ls
, i" w8 C8 n8 Y9 ]1 ?0 |# journalctl -xu kubelet -f


* j4 X8 J6 @" d8 C: A# ps -ef|grep etcd
( j. O# _" k5 P' V# ls
8 ]& F* v7 K0 M3 W: q* c$ D# cd manifests/
# ls
# mv * ../
# ls
# ls
# cd
# cd /etc/kube
$ Y3 I% W, o; M. Z. u# }# cd /etc/kubernetes/
: F5 R- i$ x) K0 {2 N5 a# ls
# mv etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml manifests/
# journalctl -xu kubelet -f

- ^/ m: K- ^; e7 v0 a+ O* O# ps -ef|grep etcd
. ~3 o1 A8 M! |' h) T2 S0 b# journalctl -xu kubelet -f|grep etcd
# k get po
# journalctl -xu kubelet -f|grep etcd
. K# g. q0 W( w* I8 m% J# journalctl -xu kubelet -f
4 \2 B* o- _- o8 l$ B9 b) K# journalctl -xu kubelet -f|grep etcd
. G0 _8 e/ Z; S# ps -ef|grep etcd
# ls
9 j2 i0 O. Q% b# cat manifests/etcd.yaml
& z( x9 I; b! Z. T( f
6 K3 o7 ^4 ^1 P) ~3 C* m
2 x4 ^% {- G3 k
3 U% K0 ^! Z6 a4 f; a# systemctl stop kubelet
# ls
# rm -rf /data/etcd/member/
/ y" c- T8 L+ ~" d9 w# ls /data/etcd/
# ls
# systemctl status kubelet
) G( M: @: m2 i+ G3 X" w: V# systemctl start kubelet
# ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=https://10.166.7.5:2379,https://10.166.7.6:2379,https://10.166.7.7:2379 endpoint status --write-out=table
# ps -ef|grep etcd

1 w9 {2 L* c, S+ m! Q2 I# ps -ef|grep etcd
( u- k. ~$ ]4 O( a9 I# ls /data/etcd/member/
# ls /data/etcd/member/snap/
( S' [- R0 @2 E7 D# ls -lhS /data/etcd/member/snap/
4 r$ T+ `0 C* d. e( H, A# ps -ef|grep etcd

# cat /etc/hosts
& o$ ^, e: R# p1 l, z# vi /etc/hosts
- p& P6 e; T8 v# K' D) A# kk get po
# kk logs -f etcd-cmp-server02
# h. X: L3 P3 t) T! u( Z' F9 i
# ls
0 |: P& v6 @$ {7 w/ Z8 C* T# cd manifests/
5 D' F: s5 `* u& O4 |! Z# ls
4 E* e# u. |9 ]* t8 S4 D# H# mv etcd.yaml ../
# cd ../
# ls
# ps -ef|grep etcd
; `5 @5 ?3 _' N5 Q# ls
, }1 P9 ~+ o' f, P$ H1 B# mv etcd.yaml manifests/
# kk get po
, S8 ?5 H" ^  w# k get po
0 ~6 @' D: M/ j' s; \8 G; G# k get node
# kk get po
7 P) Q5 k& o, l0 ^" H. c3 C# kk logs -f etcd-cmp-server02

# cd manifests/
5 o1 z3 f0 y# Z+ j. m# ls
# mv etcd.yaml ../
0 `8 w$ ~- L4 n# rm -rf /data/etcd/
# ls
# cd ../
3 P+ L. N: B- [1 ^# ls
! W: e. x( a) W" @, s) T: Z9 n# mv etcd.yaml manifests/
: B/ {$ u4 H) ?+ g0 x9 }4 l5 w# kk get po
* X8 X+ x* D5 {* d6 Z# kk logs -f etcd-cmp-server02
# kk get po
# systemctl restart kubelet
4 \/ H& j: o; u* @9 j# kk get po
/ u. z6 Y$ q. u# k3 L/ ^# kk get po
5 l2 G. `) E9 u# cat /etc/hosts
6 ^$ u/ n( V$ j/ ?6 n# kk logs -f etcd-cmp-server02
+ a( b. A5 m$ i) N* J# kk logs -f etcd-cmp-server02
# kk logs -f etcd-cmp-server02
( ~+ a0 m* |' o  ~% c# kk logs -f etcd-cmp-server02
# ls
$ C" g( J* v( `6 u6 d, G5 \) f# cd manifests/
9 {% ]9 G/ M9 P% i) L( f# ls
# vi etcd.yaml
' c" n' R8 f5 L0 x! @- q6 G# cat etcd.yaml
" `3 U! N: H# `' e! W# vi etcd.yaml
. ~) R' z" P0 |3 V; P' N# ls
# mv etcd.yaml ../
$ d, K. K' e3 |  N0 i- n7 g' q0 L2 ?# ls
6 Z# G  o% S& S8 e" ~# cd ../
2 A3 [' Q6 h7 h/ ?3 S) `# ls
% P; w8 m) h4 Y3 h" ^( O* Y# mv etcd.yaml manifests/
# ls
: p) Y9 _% n5 Q" R, b8 L; w, c0 ]+ `9 n# kk get po
# kk get po
7 e& D3 }7 E+ k6 n5 v8 w" ]( D1 f# kk get po
# kk get po
0 q  C# `: u( q( u( t/ ]& D# kk get po
5 H% T; `) W- P6 S% R8 w# kk get po
# kk get po
# kk logs -f etcd-cmp-server02
  U2 S- t- S5 I5 r' j  s# ls
# mv manifests/etcd.yaml .
# rm -rf /data/etcd/
+ x9 n+ Q9 g+ h9 c# ls
# mv etcd.yaml manifests/
2 j0 O9 S4 d5 g4 ?3 f+ B3 p# cd manifests/
# ls
# s
8 K+ C% E. ^* J% f" }# cd ../
# ls
# kk get po
2 _$ h! ^! V1 A( P
* k( b# L! I+ C! h0 r; G# kk get po
# kk get po
) E1 o' }3 [% a2 ~% }; e7 ]4 E# kk get po
# A' ?* M1 r: Z# kk logs -f etcd-cmp-server02
; j6 l7 P* s" X* j7 y' s: e% p# kk get po
/ o8 g5 A) O/ g) K: H1 Q* r0 m# kk get po
+ F9 b; h: n1 O" C: v! M# kk logs -f etcd-cmp-server02
, G0 Q  b5 o7 S! j! i6 R3 @! v  a# ls
  `  i7 I' z1 A) @' \+ o9 S# mv manifests/etcd.yaml .

# kk get po
# ls
8 a! j: H2 Y, x2 b# rm -rf /data/etcd/
$ N/ B# S- K/ o' Z6 f8 G5 r# ls
# mv etcd.yaml manifests/
#
6 M4 R6 @; p3 t# kk get po
# kk logs -f etcd-cmp-server02
# ls
& Q# s* {# M/ x  Y# kk get po
" n1 c, }; o1 R; ~6 x( e# kk logs -f kube-apiserver-cmp-server02
6 P# I4 C2 H" X! `8 N: u
# kk get po
# kk logs -f etcd-cmp-server02
- a; R* Y4 N% h! X- E# ls
# cat manifests/etcd.yaml
# ls
/ N/ u5 y" s$ y4 n7 J7 @# ls
- t0 n4 ~. d5 ~5 g/ G# ls
# ls
# ls
# ls
# ls
  S: X# ~* C! w# u( b' a" N# ls
" Y& N- W* x+ Q6 o# kk get po
. ^9 `% H! z! d) U8 c# N# kk logs -f etcd-cmp-server02
8 Z! r. S% ^/ e# kk get po

# kk get po
# kk logs -f etcd-cmp-server02
# kk logs -f etcd-cmp-server02
1 q, Z$ }- `) {& c: c* j# kk get po
# k get po
9 H9 @: L& E) m. b$ k4 C# kk get po
# ls
# cd manifests/
7 `: u" z+ e) J: w# ls
# kk get po
# kk logs -f kube-apiserver-cmp-server02
# kk get po
# ls
# mv kube-apiserver.yaml ../
# cd ../
2 `+ v2 ?  v. d# mv kube-apiserver.yaml manifests/
# kk get po
# kk get po
; ?; S* ^9 r8 |. ]* M! B# kk logs -f kube-apiserver-cmp-server02
, G8 N$ t' Y: @8 f+ M9 V7 j( N2 ?# ps -ef|grep etcd
! s; I& |& H$ H, s( i* U# ss -tunlp|grep 2379
8 e- Y4 ~5 ]5 H' J, E  X( p# kk logs -f kube-apiserver-cmp-server02
* k; R8 e' Z- U# kk get po
# kk get po -owide
5 u! G6 ^0 ?4 @, Y# s
# kk get po
# ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=https://10.166.7.5:2379,https://10.166.7.6:2379,https://10.166.7.7:2379 endpoint status --write-out=table
# kk get po
2 o( u6 M$ [0 Z6 l9 ?8 I# ^# kk get po
# kk get po
# kk logs -f kube-apiserver-cmp-server02
# tennet 127.0.0.1:2379

. O9 ]5 G& I' j: G% e3 ?# ls
! L, a5 w  e5 G2 b# cd manifests/
7 `5 @4 E6 M) a+ t, @* J) M  j- M% I# ls
  \# K# g2 [/ N# k0 _# cat kube-apiserver.yaml
# kk get po
6 [( x! P! ]" q, B# kk get po
5 e. J( l5 y2 z* u; d( a6 a
; p  {+ B% s- i+ ?* X1 r, ?. X# kk get po
' ?; u- m+ U5 P- T! o/ v" G. f9 `) S# df -h
. E& w3 y2 F- ]9 V# ls
1 ?* g- D  L( T8 Y( J, \# cd /
* q6 A( A  G# N# cd
' U9 i* u/ K7 Q# cd /home/devops/
# ls
# cd
# ls
# du -sh *
: K, j1 j" U- x  B- g* I* f# cd /
( n3 M2 Q. _+ {; j  N$ f# du -sh *
2 K. P5 S0 p# n# cd data/
# ls
# du -sh *
( N4 V0 ^- ^3 l0 [' V% Y- \
# journalctl -xefu kubelet
8 p  }- z4 `/ T' @4 H: b# ;s
# /usr/local/bin/nerdctl -n k8s.io tag caas4/keystone-sync-db:latest 10.166.7.5:30443/caas/keystone-sync-db:latest
# w: C* Q3 N$ ^9 M$ i# /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/caas/keystone-sync-db:latest
# /usr/local/bin/nerdctl -n k8s.io login 10.166.7.5:30443
5 j3 v* {; I% F5 Q3 a! W; v# /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/caas/keystone-sync-db:latest
# /usr/local/bin/nerdctl -n k8s.io tag caas4/perception 10.166.7.5:30443/caas/perception:latest
" p1 y( M# A$ s# /usr/local/bin/nerdctl -n k8s.io tag sameersbn/gitlab:13.10.3 10.166.7.5:30443/cmp/gitlab:13.10.3
) Q) }# G  u0 u1 {/ Y#  /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/cmp/gitlab:13.10.3
# /usr/local/bin/nerdctl -n k8s.io tag sameersbn/postgresql:12-20200524 10.166.7.5:30443/cmp/postgresql:12-20200524
. c/ I1 m* ~' R& Z+ L2 e, d+ m# /usr/local/bin/nerdctl -n k8s.io tag cmp/redis:5.0.6  10.166.7.5:30443/cmp/redis:5.0.6
# /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/cmp/redis:5.0.6
! f! I8 O' o" f: u. v: ?4 S# /usr/local/bin/nerdctl -n k8s.io tag busybox  10.166.7.5:30443/cmp/busybox:latest
# /usr/local/bin/nerdctl -n k8s.io push 10.166.7.5:30443/cmp/busybox:latest
# kubectl get node --show-labels
5 a. [! b3 A  O% o# journalctl -xefu kubelet
# journalctl -xefu kubelet
# journalctl -xefu kubelet
# t" C/ _6 c- m# n. l# journalctl -xefu kubelet
3 {0 s# ^3 [% P' h, W5 k# kubeadm reset -f
1 F; ~/ h6 e! ~0 B2 O! U( [0 ^2 [" V# vi /etc/hosts
. I1 h% ^, _4 B8 d4 z% @8 e/ ~# cat /etc/hosts
# systemctl stop firewalld || true
3 j0 G2 N# ?/ G" c& t# systemctl disable firewalld || true
, {% L4 j+ z/ W- B" m# setenforce 0
# sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config
& s0 ]) D$ q, n8 Y0 I& C1 S# R3 R# modprobe br_netfilter && modprobe nf_conntrack
# cat > /etc/sysctl.d/98-k8s.conf << EOF
# net.netfilter.nf_conntrack_tcp_be_liberal = 1
# net.netfilter.nf_conntrack_tcp_loose = 1
; O! y/ @( D- p/ [* I  ]$ y# net.netfilter.nf_conntrack_max = 524288
# net.netfilter.nf_conntrack_buckets = 131072
# net.netfilter.nf_conntrack_tcp_timeout_established = 21600
# net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
# net.ipv4.neigh.default.gc_thresh1 = 1024
# net.ipv4.neigh.default.gc_thresh2 = 2048
4 @8 m" a# ~! v' N# net.ipv4.neigh.default.gc_thresh3 = 4096
# vm.max_map_count = 262144
# net.ipv4.ip_forward = 1
# net.ipv4.tcp_timestamps = 1
$ Y1 b! D3 O$ C, g5 y0 S9 u1 j# net.bridge.bridge-nf-call-ip6tables = 1
$ ~( V+ _9 a  _1 |" S2 Z. A' p# net.bridge.bridge-nf-call-iptables = 1
# net.ipv6.conf.all.forwarding=1
# fs.file-max=1048576
9 g, `! N8 [0 K) s9 m# fs.inotify.max_user_instances = 8192
# fs.inotify.max_user_watches = 524288
# EOF
" q3 Q. x8 q) I  f& m
! L3 O& a+ k* g; L$ I# cat > /etc/security/limits.d/98-k8s.conf << EOF
6 {3 j/ @, \2 b& }# * soft nproc 65535
/ [1 }/ d! a, k# * hard nproc 65535
# * soft nofile 65535
# ~8 `3 G! }! q/ ^4 m3 ^# * hard nofile 65535
# EOF
! x) e7 h- ^. z) l" _
# sysctl --system
9 @$ ]" k4 J8 ?, r3 }- Z" [# sysctl -p
. H' A. y- F; K0 V# swapoff -a
# sed -i /swap/d /etc/fstab
3 Y1 [" @# u% f& W# kubeadm join apiserver.cluster.local:6443 --token 25omv5.cqx4gnuhoyh7nwxy --discovery-token-ca-cert-hash sha256:eaf7ee645c5cc0af1782df11c38519b84d643f06d9c6613009047e9b2e275524
4 }; f, |# b. {, A2 O# kubeadm reset -f
! v: y  b5 F+ z; q+ L3 S7 {# kubeadm reset -f
# kubeadm join apiserver.cluster.local:6443 --token 25omv5.cqx4gnuhoyh7nwxy --discovery-token-ca-cert-hash sha256:eaf7ee645c5cc0af1782df11c38519b84d643f06d9c6613009047e9b2e275524
# /usr/local/bin/nerdctl -n k8s.io images
% w4 X5 @3 E$ f9 l! B2 Z# /usr/local/bin/nerdctl -n k8s.io tag rancher/local-path-provisioner:v0.0.24  10.166.7.5:30443/cmp/local-path-provisioner:v0.0.24
2 n! }& s3 j3 N6 M5 v- v( x2 y# /usr/local/bin/nerdctl -n k8s.io push  10.166.7.5:30443/cmp/local-path-provisioner:v0.0.24


# cd local-path-provisioner/
$ g( ~8 N, k- ^. Z8 d) F# ls
+ [" S! _( k1 d* P# du -sh *
# cd pvc-cedd6757-22af-46a3-a863-cdc4a02af016_caas-system_minio/
# du -sh *
# cd mgmt/
4 D  ]5 r' f, H% B# ls
# cat /etc/kubernetes/kubelet.conf
# cat /etc/kubernetes/kubelet.conf
# cat /etc/kubernetes/controller-manager.conf
1 n" M2 Z. @: l- v# cat /etc/kubernetes/kubelet.conf


# cat /etc/kubernetes/kubelet.conf
# cat /var/lib/kubelet/config.yaml
( H7 U  w& T  q! q' `: x
% Q  D5 O" w$ `% o) e