|
|
kubectl describe pod -n kube-system kube-flannel-ds-amd64-42rl7% y+ {/ T+ y3 I8 c' \* {
' I2 }. L% @" P9 M: S1 X4 M1 h* _/ s" {# o
! n4 c- |( u/ M
" ^. d% I! f0 Z) y7 qName: kube-flannel-ds-amd64-42rl7Namespace: $ ^1 Y& M: _8 c, q' Q" L G& T% g+ [
kube-systemPriority: 0PriorityClassName: <none>0 G! {' U- K* ~, s5 l' W m6 h
Node: node5/10.168.209.17% d$ o% I+ _* B: N* {7 j* u: B
Start Time: Wed, 22 Aug 2018 16:47:10 +0300Labels: app=flannel controller-revision-hash=911701653 pod-template-generation=1 tier=nodeAnnotations: <none>Status: RunningIP: 10.168.209.17Controlled By: DaemonSet/kube-flannel-ds-amd64Init Containers: install-cni: Container ID: docker://eb7ee47459a54d401969b1770ff45b39dc5768b0627eec79e189249790270169 Image: quay.io/coreos/flannel:v0.10.0-amd64 Image ID: docker-pullable://quay.io/coreos/flannel@sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac Port: <none> Host Port: <none> Command: cp Args: -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conflist State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 22 Aug 2018 16:47:24 +0300 Finished: Wed, 22 Aug 2018 16:47:24 +0300 Ready: True Restart Count: 0 Environment: <none> Mounts: /etc/cni/net.d from cni (rw) /etc/kube-flannel/ from flannel-cfg (rw) /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-9wmch (ro)Containers: kube-flannel: Container ID: docker://521b457c648baf10f01e26dd867b8628c0f0a0cc0ea416731de658e67628d54e Image: quay.io/coreos/flannel:v0.10.0-amd64 Image ID: docker-pullable://quay.io/coreos/flannel@sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac Port: <none> Host Port: <none> Command: /opt/bin/flanneld Args: --ip-masq --kube-subnet-mgr State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Thu, 30 Aug 2018 10:15:04 +0300 Finished: Thu, 30 Aug 2018 10:15:08 +0300 Ready: False Restart Count: 2136 Limits: cpu: 100m memory: 50Mi Requests: cpu: 100m memory: 50Mi Environment: POD_NAME: kube-flannel-ds-amd64-42rl7 (v1:metadata.name) POD_NAMESPACE: kube-system (v1:metadata.namespace) Mounts: /etc/kube-flannel/ from flannel-cfg (rw) /run from run (rw) /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-9wmch (ro)Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled TrueVolumes: run: Type: HostPath (bare host directory volume) Path: /run HostPathType: cni: Type: HostPath (bare host directory volume) Path: /etc/cni/net.d HostPathType: flannel-cfg: Type: ConfigMap (a volume populated by a ConfigMap) Name: kube-flannel-cfg Optional: false flannel-token-9wmch: Type: Secret (a volume populated by a Secret) SecretName: flannel-token-9wmch Optional: falseQoS Class: GuaranteedNode-Selectors: beta.kubernetes.io/arch=amd64Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/disk-pressure:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute node.kubernetes.io/unreachable:NoExecuteEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 51m (x2128 over 7d) kubelet, node5 Container image "quay.io/coreos/flannel:v0.10.0-amd64" already present on machine Warning BackOff 1m (x48936 over 7d) kubelet, node5 Back-off restarting failed container5 q5 z0 ?) u' Z3 w6 [9 r
0 g9 D; A2 P3 R& Z/ ^, Z0 v检查kube-controller-manager.yaml7 r. H% x: h v2 Q, ?. _0 d& v
& v. d) e( V$ ]
9 v& F! _1 V# T* H4 v3 k$ v:apiVersion: v11 [( l9 N7 A: A4 w& B+ g2 X
kind: Podmetadata: annotations: scheduler.alpha.kubernetes.io/critical-pod: "" creationTimestamp: null labels: component: kube-controller-manager tier: control-plane name: kube-controller-manager namespace: kube-systemspec: containers: - command: - kube-controller-manager - --address=127.0.0.1 - --allocate-node-cidrs=true - --cluster-cidr=192.168.0.0/24 - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key - --controllers=*,bootstrapsigner,tokencleaner - --kubeconfig=/etc/kubernetes/controller-manager.conf - --leader-elect=true - --node-cidr-mask-size=24 - --root-ca-file=/etc/kubernetes/pki/ca.crt - --service-account-private-key-file=/etc/kubernetes/pki/sa.key - --use-service-account-credentials=true image: k8s.gcr.io/kube-controller-manager-amd64:v1.11.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /healthz port: 10252 scheme: HTTP initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-controller-manager resources: requests: cpu: 200m volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/kubernetes/controller-manager.conf name: kubeconfig readOnly: true - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec name: flexvolume-dir - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true hostNetwork: true priorityClassName: system-cluster-critical volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/kubernetes/controller-manager.conf type: FileOrCreate name: kubeconfig - hostPath: path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec type: DirectoryOrCreate name: flexvolume-dir - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certsstatus: {}/ s- j; ~& y3 Q, n) C
5 V+ Z1 L- ?% W% Bkubectl logs --namespace kube-system kube-flannel-ds-amd64-5fx28 n) f- q2 Y7 q# v0 U* I
2 M. w4 J7 m+ a4 k- e
* o4 S0 ^ ^; m3 i( H
pmain.go:475] Determining IP address of default interfacemain.go:488]
5 m8 D+ v* C4 `) |; \; ]6 |' X- @2 P/ t/ Q8 H+ v7 c7 y& J9 K( F9 w
6 t9 H- E" Q( G9 ]$ tUsing interface with name eth0 and address 10.168.209.14main.go:505] Defaulting external address to interface address (10.168.209.14)kube.go:131] Waiting 10m0s for node controller to synckube.go:294] Starting kube subnet managerkube.go:138] Node controller sync successfulmain.go:235] Created subnet manager: Kubernetes Subnet Manager - node2main.go:238] Installing signal handlersmain.go:353] Found network config - Backend type: vxlanvxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=falsemain.go:280] Error registering network: failed to acquire lease: node "node2" pod cidr not assignedmain.go:333] Stopping shutdownHandler...# v, I' T* c+ c7 ?2 L4 V5 ]; r
1 F) m- O5 ~7 K+ X7 M) N
. b D' m8 P+ L3 h9 t8 K8 p& E0 [! O$ g* F- E: E7 q
6 n; g ?& O; ~' P" i
cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep -i cluster-cidr- --cluster-cidr=172.168.10.0/24
8 e0 z9 N! u% `( F F
: L7 B- {9 r0 H! e& L5 W+ C x
2 V- m% l, L; {* m/ K6 M" P, B( w, w3 X
kubectl patch node podname -p '{"spec":{"podCIDR":"172.168.10.0/24"}}'
6 H, n3 g* k% B9 O4 b5 F4 T$ v! {8 c/ e3 x& l" G
" T6 \ t0 L- h) B! |
例如:kubectl patch node slave-node-1 -p '{"spec":{"podCIDR":"172.168.10.0/24"}}'
# n, w+ p& f( i5 q0 o7 w2 @, b; M. f( T5 \% {5 L5 W
5 Z' f% k, c0 r& h/ E% _; tsudo ifconfig cni0 down;1 J1 V$ L$ @$ c4 e" x$ m# A- x
sudo ifconfig flannel.1 down;
7 ~& U' Z& p5 ~3 fsudo ip link delete cni0;" @* g" i' i0 R( t; ?
sudo ip link delete flannel.1;To fix this, please following the step below: - Step 0: Reset all Nodes within your Cluster. Run all nodes with
3 L+ B6 f* R5 J1 w$ t
kubeadm reset --force;
- Step 1: Down Interface cni0 and flannel.1.
& G4 l$ Y. [4 `/ D0 O+ h
sudo ifconfig cni0 down;
# d ~/ m: T8 dsudo ifconfig flannel.1 down;
- Step 2: Delete Interface cni0 and flannel.1.
' ^6 k7 _( v; e g; X# |) C
sudo ip link delete cni0;
$ W7 q' K* j8 H0 C/ ~+ [3 a4 Xsudo ip link delete flannel.1;
- Step 3: Remove all items within /etc/cni/net.d/.
7 {( V8 C3 G3 c* F) F1 n
sudo rm -rf /etc/cni/net.d/;
- Step 4: Re-Bootstrap your Kubernetes Cluster again.
1 F" |: f& p- `7 f4 _
kubeadm init --control-plane-endpoint="..." --pod-network-cidr=10.244.0.0/16;
- Step 5: Re-deploy CNIs.
1 b& r( n) o h- J% ]8 y6 D
- Step 6: Restart your CNIs, here I used Container Daemon (Containerd).
. J9 ~! |, j: X
systemctl restart containerd;
5 U) I1 H- a* B$ N
9 O$ W u+ |6 Q+ s+ d Q) M
( v7 }: }3 l1 N4 F _3 ]$ J |
|