[Kubernetes] taint&toleraton, cordon&drain

taint&toleraton

 - node taint, Pod tolertation

 - worker node에 taint가 설정된 경우 동일 값의 toleration이 있는 Pod만 배치된다.

 - toleration이 있는 Pod는 동일한 taint가 있는 node를 포함하여 모든 node에 배치된다.

 - effect 필드 종류

   1) NoSchedule : toleration이 맞지 않으면 배치 되지 않는다.

   2) PreferNoSchedule : toleration이 맞지 않으면 배치되지 않으나, 클러스터 리소스가 부족하면 할당된다.

   3) NoExecute : toleration이 맞으면 동작중인 pod를 종료

 

master 노드에서는 어플리케이션 pod 가 실행되지 않는 이유는 아래와 같이 NoSchedule가 설정되어 있기 때문이다.

# kubectl describe nodes master.example.com | grep -i taint
Taints:             node-role.kubernetes.io/control-plane:NoSchedule

 

node1에다가 tainit 설정 후 확인

# kubectl taint nodes node1.example.com role=web:NoSchedule
node/node1.example.com tainted

# kubectl describe nodes node{1,2}.example.com | grep -i taint
Taints:             role=web:NoSchedule
Taints:             <none>

 

taint가 설정된 노드에는 pod 가 구성되지 않음

# cat deploy-nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webui
spec:
  replicas: 4
  selector:
    matchLabels:
      app: webui
  template:
    metadata:
      name: nginx-pod
      labels:
        app: webui
    spec:
      containers:
      - name: nginx-container
        image: nginx:1.14
#      tolerations:
#      - key: "role"
#        operator: "Equal"
#        value: "web"
#        effect: "NoSchedule

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE                NOMINATED NODE   READINESS GATES
webui-5cfbcf5f65-8q244   1/1     Running   0          40s   10.44.0.2   node2.example.com   <none>           <none>
webui-5cfbcf5f65-9lmsp   1/1     Running   0          40s   10.44.0.1   node2.example.com   <none>           <none>
webui-5cfbcf5f65-dqx49   1/1     Running   0          40s   10.44.0.4   node2.example.com   <none>           <none>
webui-5cfbcf5f65-nxmfj   1/1     Running   0          40s   10.44.0.3   node2.example.com   <none>           <none>

 

taint가 설정된 노드에는 pod를 구성하려면 toleration을 설정해야함, 만약 taint와 toleration 값이 일치하면 taint가 설정된 곳에도 구성 가능

# cat deploy-nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webui
spec:
  replicas: 4
  selector:
    matchLabels:
      app: webui
  template:
    metadata:
      name: nginx-pod
      labels:
        app: webui
    spec:
      containers:
      - name: nginx-container
        image: nginx:1.14
      tolerations:
      - key: "role"
        operator: "Equal"
        value: "web"
        effect: "NoSchedule"

# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP          NODE                NOMINATED NODE   READINESS GATES
webui-78b99b44b-9qfp8   1/1     Running   0          9s    10.44.0.2   node2.example.com   <none>           <none>
webui-78b99b44b-bgtmd   1/1     Running   0          9s    10.36.0.2   node1.example.com   <none>           <none>
webui-78b99b44b-dqzjh   1/1     Running   0          9s    10.44.0.1   node2.example.com   <none>           <none>
webui-78b99b44b-jq2mh   1/1     Running   0          9s    10.36.0.3   node1.example.com   <none>           <none>

 

taint 삭제 후 확인

# kubectl taint nodes node1.example.com role-
node/node1.example.com untainted

# kubectl describe nodes node{1,2}.example.com | grep -i taint
Taints:             <none>
Taints:             <none>

 

cordon & drain

 - 노드 스케줄링 중단(cordon) 및 허용(uncordon)

  1) 특정 노드에 pod 스케줄을 금지하거나 해제

# kubectl get nodes 
NAME                 STATUS   ROLES           AGE   VERSION
master.example.com   Ready    control-plane   64d   v1.25.4
node1.example.com    Ready    <none>          64d   v1.25.4
node2.example.com    Ready    <none>          64d   v1.25.4

# kubectl cordon node2.example.com
node/node2.example.com cordoned

# kubectl get nodes
NAME                 STATUS                     ROLES           AGE   VERSION
master.example.com   Ready                      control-plane   64d   v1.25.4
node1.example.com    Ready                      <none>          64d   v1.25.4
node2.example.com    Ready,SchedulingDisabled   <none>          64d   v1.25.4

# kubectl apply -f deploy-nginx.yaml
deployment.apps/webui created

# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP          NODE                NOMINATED NODE   READINESS GATES
webui-78b99b44b-2mgmk   1/1     Running   0          17s   10.36.0.5   node1.example.com   <none>           <none>
webui-78b99b44b-6hjdh   1/1     Running   0          17s   10.36.0.3   node1.example.com   <none>           <none>
webui-78b99b44b-bkqpm   1/1     Running   0          17s   10.36.0.2   node1.example.com   <none>           <none>
webui-78b99b44b-p5v8p   1/1     Running   0          17s   10.36.0.4   node1.example.com   <none>           <none>

# kubectl uncordon node2.example.com
node/node2.example.com uncordoned

# kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
master.example.com   Ready    control-plane   64d   v1.25.4
node1.example.com    Ready    <none>          64d   v1.25.4
node2.example.com    Ready    <none>          64d   v1.25.4

 

 - 노드 비우기(drain)

  1) 특정 노드에서 동작중인 모든 Pod를 제거

    --ignore-daemonsets : DaemonSet-managed pod들은 ignore

    --force=false : RC, RS, Job, DaemonSet 또는 StatefulSet에서 관리하지 않는 Pod까지 제거

 

nginx 및 redis pod를 실행한 후 node1에 동작인 모든 pod를 제거하려고 하니 삭제할 수 없다는 문구나 나온다.

# kubectl apply -f deploy-nginx.yaml
deployment.apps/webui created

# kubectl run db --image=redis
pod/db created

# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE    IP          NODE                NOMINATED NODE   READINESS GATES
db                      1/1     Running   0          87s    10.36.0.4   node1.example.com   <none>           <none>
webui-78b99b44b-5xhbm   1/1     Running   0          103s   10.36.0.2   node1.example.com   <none>           <none>
webui-78b99b44b-mrfjs   1/1     Running   0          103s   10.44.0.2   node2.example.com   <none>           <none>
webui-78b99b44b-s7xjg   1/1     Running   0          103s   10.44.0.1   node2.example.com   <none>           <none>
webui-78b99b44b-vkpmd   1/1     Running   0          103s   10.36.0.3   node1.example.com   <none>           <none>

# kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
master.example.com   Ready    control-plane   64d   v1.25.4
node1.example.com    Ready    <none>          64d   v1.25.4
node2.example.com    Ready    <none>          64d   v1.25.4

# kubectl drain node1.example.com 
node/node1.example.com cordoned
error: unable to drain node "node1.example.com" due to error:[cannot delete Pods declare no controller (use --force to override): default/db, cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/kube-proxy-skhgb, kube-system/weave-net-t92nl], continuing command...
There are pending nodes to be drained:
 node1.example.com
cannot delete Pods declare no controller (use --force to override): default/db
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/kube-proxy-skhgb, kube-system/weave-net-t92nl

 

 --ignore-daemonsets, --force 옵션을 넣어서 다시 명령 실행

uncordon명령을 이용하여 node1의 스케줄링 허용

# kubectl drain node1.example.com --ignore-daemonsets --force 
node/node1.example.com already cordoned
Warning: deleting Pods that declare no controller: default/db; ignoring DaemonSet-managed Pods: kube-system/kube-proxy-skhgb, kube-system/weave-net-t92nl
evicting pod default/webui-78b99b44b-5xhbm
evicting pod ingress-nginx/ingress-nginx-controller-6c56945c75-d6xlj
evicting pod default/webui-78b99b44b-vkpmd
evicting pod ingress-nginx/ingress-nginx-admission-create-w49g8
evicting pod default/db
pod/ingress-nginx-admission-create-w49g8 evicted
pod/db evicted
pod/webui-78b99b44b-vkpmd evicted
pod/webui-78b99b44b-5xhbm evicted
pod/ingress-nginx-controller-6c56945c75-d6xlj evicted
node/node1.example.com drained

# kubectl get nodes
NAME                 STATUS                     ROLES           AGE   VERSION
master.example.com   Ready                      control-plane   64d   v1.25.4
node1.example.com    Ready,SchedulingDisabled   <none>          64d   v1.25.4
node2.example.com    Ready                      <none>          64d   v1.25.4

# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE    IP          NODE                NOMINATED NODE   READINESS GATES
webui-78b99b44b-ckjc4   1/1     Running   0          20s    10.44.0.3   node2.example.com   <none>           <none>
webui-78b99b44b-mgnrc   1/1     Running   0          20s    10.44.0.5   node2.example.com   <none>           <none>
webui-78b99b44b-mrfjs   1/1     Running   0          4m7s   10.44.0.2   node2.example.com   <none>           <none>
webui-78b99b44b-s7xjg   1/1     Running   0          4m7s   10.44.0.1   node2.example.com   <none>           <none>

# kubectl uncordon node1.example.com 
node/node1.example.com uncordoned

# kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
master.example.com   Ready    control-plane   64d   v1.25.4
node1.example.com    Ready    <none>          64d   v1.25.4
node2.example.com    Ready    <none>          64d   v1.25.4

 

 

[참고]

- 유투브 따배쿠 강의