본문 바로가기
Compute/kubernetis

[CKA] 11. cluster upgrade | 노드 drain | cordon / uncordon

by 조청유곽 2025. 1. 14.
반응형

01. cluster upgrade 진행 / 조건1: 업그레이드 전 drain, 조건2: 업그레이드 후 uncordon

 

[solve]

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE     VERSION
k8s-master   Ready    control-plane   6h48m   v1.27.2
k8s-node1    Ready    <none>          6h32m   v1.27.2
k8s-node2    Ready    <none>          6h31m   v1.27.2
[root@k8s-master ~]# 
[root@k8s-master ~]# 
[root@k8s-master ~]# 
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl drain k8s-master --ignore-daemonsets --delete-emptydir-data 
node/k8s-master cordoned
Warning: ignoring DaemonSet-managed Pods: calico-system/calico-node-6pfjk, calico-system/csi-node-driver-gfpv7, kube-system/kube-proxy-xhtzx
evicting pod tigera-operator/tigera-operator-84cf9b6dbb-6dkz4
evicting pod calico-system/calico-kube-controllers-bf6cb6475-lnzf7
evicting pod calico-apiserver/calico-apiserver-574c754579-hrqqg
evicting pod calico-apiserver/calico-apiserver-574c754579-x4qqw
evicting pod kube-system/coredns-5d78c9869d-4nbr6
evicting pod calico-system/calico-typha-6cb494cb8d-65scs
evicting pod kube-system/coredns-5d78c9869d-q96tb
pod/calico-apiserver-574c754579-x4qqw evicted
pod/calico-kube-controllers-bf6cb6475-lnzf7 evicted
pod/tigera-operator-84cf9b6dbb-6dkz4 evicted
pod/calico-apiserver-574c754579-hrqqg evicted
pod/coredns-5d78c9869d-4nbr6 evicted
pod/coredns-5d78c9869d-q96tb evicted
pod/calico-typha-6cb494cb8d-65scs evicted
node/k8s-master drained
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS                     ROLES           AGE     VERSION
k8s-master   Ready,SchedulingDisabled   control-plane   6h51m   v1.27.2
k8s-node1    Ready                      <none>          6h34m   v1.27.2
k8s-node2    Ready                      <none>          6h34m   v1.27.2
[root@k8s-master ~]# kubectl cordon k8s-master
node/k8s-master already cordoned
[root@k8s-master ~]# 
[root@k8s-master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:18:49Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master ~]# 
[root@k8s-master ~]# kubeadm upgrade apply v1.27.2
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade/version] You have chosen to change the cluster version to "v1.27.2"
[upgrade/versions] Cluster version: v1.27.16
[upgrade/versions] kubeadm version: v1.27.2
[upgrade] Are you sure you want to proceed? [y/N]: n
won't proceed; the user didn't answer (Y|y) in order to continue
To see the stack trace of this error execute with --v=5 or higher
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS                     ROLES           AGE     VERSION
k8s-master   Ready,SchedulingDisabled   control-plane   6h53m   v1.27.2
k8s-node1    Ready                      <none>          6h36m   v1.27.2
k8s-node2    Ready                      <none>          6h36m   v1.27.2
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl uncordon k8s-master
node/k8s-master uncordoned
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE     VERSION
k8s-master   Ready    control-plane   6h53m   v1.27.2
k8s-node1    Ready    <none>          6h36m   v1.27.2
k8s-node2    Ready    <none>          6h36m   v1.27.2
[root@k8s-master ~]# 
[root@k8s-master ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Thu 2024-12-26 22:40:23 KST; 6h ago
     Docs: https://kubernetes.io/docs/
 Main PID: 27147 (kubelet)
    Tasks: 14 (limit: 31209)
   Memory: 62.5M
   CGroup: /system.slice/kubelet.service
           └─27147 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.c>

Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.464124   27147 memory_manager.go:346] "RemoveStaleState removing state" podUID>
Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.464136   27147 memory_manager.go:346] "RemoveStaleState removing state" podUID>
Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.464142   27147 memory_manager.go:346] "RemoveStaleState removing state" podUID>
Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.464148   27147 memory_manager.go:346] "RemoveStaleState removing state" podUID>
Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.464153   27147 memory_manager.go:346] "RemoveStaleState removing state" podUID>
Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.464158   27147 memory_manager.go:346] "RemoveStaleState removing state" podUID>
Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.597922   27147 reconciler_common.go:258] "operationExecutor.VerifyControllerAt>
Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.598018   27147 reconciler_common.go:258] "operationExecutor.VerifyControllerAt>
Dec 27 05:33:52 k8s-master kubelet[27147]: I1227 05:33:52.598349   27147 reconciler_common.go:258] "operationExecutor.VerifyControllerAt>
Dec 27 05:33:53 k8s-master kubelet[27147]: I1227 05:33:53.284816   27147 pod_startup_latency_tracker.go:102] "Observed pod startup durat>

[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.16", GitCommit:"cbb86e0d7f4a049666fac0551e8b02ef3d6c3d9a", GitTreeState:"clean", BuildDate:"2024-07-17T01:44:26Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master ~]# 
[root@k8s-master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:18:49Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master ~]#

02. 노드를 스케줄링 되지 않도록 만들고 해당 노드의 모든 pod를 rechedule 한다. pod rechedule 후 다시 활성화한다. 

 

[solve]

[root@k8s-master ~]# kubectl cordon k8s-node1
node/k8s-node1 cordoned
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS                     ROLES           AGE   VERSION
k8s-master   Ready                      control-plane   10h   v1.27.2
k8s-node1    Ready,SchedulingDisabled   <none>          9h    v1.27.2
k8s-node2    Ready                      <none>          9h    v1.27.2
[root@k8s-master ~]# kubectl drain --ignore-daemonsets --delete-emptydir-data k8s-node1
node/k8s-node1 already cordoned
error: unable to drain node "k8s-node1" due to error:cannot delete Pods declare no controller (use --force to override): default/foo, default/kucc8, default/nginx, default/non-persistent-redis, continuing command...
There are pending nodes to be drained:
 k8s-node1
cannot delete Pods declare no controller (use --force to override): default/foo, default/kucc8, default/nginx, default/non-persistent-redis
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl drain --ignore-daemonsets --delete-emptydir-data k8s-node1 --force
node/k8s-node1 already cordoned
Warning: deleting Pods that declare no controller: default/foo, default/kucc8, default/nginx, default/non-persistent-redis; ignoring DaemonSet-managed Pods: calico-system/calico-node-pf4s2, calico-system/csi-node-driver-969dd, kube-system/kube-proxy-jv7xk
evicting pod kube-system/coredns-5d78c9869d-8fpgb
evicting pod default/nginx-deployment-cbdccf466-6nqjr
evicting pod calico-apiserver/calico-apiserver-574c754579-t5fdm
evicting pod calico-system/calico-kube-controllers-bf6cb6475-bhbm9
evicting pod default/foo
evicting pod default/kucc8
evicting pod default/nginx
evicting pod default/non-persistent-redis
evicting pod default/nginx-deployment-cbdccf466-nsp68
evicting pod development/icandor-cbdccf466-5mtg7
evicting pod default/nginx-deployment-cbdccf466-xx9hr
evicting pod development/icandor-cbdccf466-m5wqf
I1227 09:00:03.168531  271456 request.go:696] Waited for 1.092228621s due to client-side throttling, not priority and fairness, request: GET:https://192.168.56.30:6443/api/v1/namespaces/calico-system/pods/calico-kube-controllers-bf6cb6475-bhbm9
pod/icandor-cbdccf466-m5wqf evicted
pod/non-persistent-redis evicted
pod/icandor-cbdccf466-5mtg7 evicted
pod/nginx-deployment-cbdccf466-xx9hr evicted
pod/nginx-deployment-cbdccf466-nsp68 evicted
pod/nginx-deployment-cbdccf466-6nqjr evicted
pod/kucc8 evicted
pod/calico-apiserver-574c754579-t5fdm evicted
pod/calico-kube-controllers-bf6cb6475-bhbm9 evicted
pod/nginx evicted
pod/coredns-5d78c9869d-8fpgb evicted
 pod/foo evicted
node/k8s-node1 drained
[root@k8s-master ~]#  
[root@k8s-master ~]# kubectl get pods -o wide
NAME                               READY   STATUS             RESTARTS        AGE     IP              NODE        NOMINATED NODE   READINESS GATES
dnsutils                           1/1     Running            0               136m    20.96.169.144   k8s-node2   <none>           <none>
env-pod                            1/1     Running            0               9h      20.96.169.133   k8s-node2   <none>           <none>
frontend                           0/1     CrashLoopBackOff   121 (47s ago)   9h      20.96.169.130   k8s-node2   <none>           <none>
nginx-deployment-cbdccf466-4k8t8   1/1     Running            0               5h21m   20.96.169.139   k8s-node2   <none>           <none>
nginx-deployment-cbdccf466-bjtj5   1/1     Running            0               5h21m   20.96.169.140   k8s-node2   <none>           <none>
nginx-deployment-cbdccf466-d5kxz   1/1     Running            0               2m57s   20.96.169.146   k8s-node2   <none>           <none>
nginx-deployment-cbdccf466-kz992   1/1     Running            0               2m57s   20.96.169.148   k8s-node2   <none>           <none>
nginx-deployment-cbdccf466-rvflr   1/1     Running            0               2m57s   20.96.169.149   k8s-node2   <none>           <none>
nginx-deployment-cbdccf466-wf8gk   1/1     Running            0               7h25m   20.96.169.137   k8s-node2   <none>           <none>
nginx1                             1/1     Running            0               9h      20.96.169.132   k8s-node2   <none>           <none>
nginx2                             1/1     Running            0               9h      20.96.169.131   k8s-node2   <none>           <none>
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl uncordon k8s-node1
node/k8s-node1 uncordoned
[root@k8s-master ~]# 
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   10h   v1.27.2
k8s-node1    Ready    <none>          10h   v1.27.2
k8s-node2    Ready    <none>          10h   v1.27.2
[root@k8s-master ~]#

 

반응형
저작자표시 비영리 변경금지

'Compute > kubernetis' 카테고리의 다른 글

[CKA] 13. clusterRole과 serviceAccount를 생성  (0) 2025.01.14
[CKA] 12. ingress 생성 | 특정 조건의 ingress 생성  (0) 2025.01.14
[CKA] 10. kubectl top 명령어 사용 문제 | --sort-by | -l 옵션 사용  (0) 2025.01.14
[CKA] 09. persistent volume | persistent volume claim 생성  (0) 2025.01.14
[CKA] 08. deployment 생성 | deployment scale out  (0) 2025.01.14

관련글

티스토리툴바