[Kubernetes] Horizontal Pod Autoscaling 운영

Horizontal Pod Autoscaling 운영

Metrics-Server

 - Pod와 Node들의 CPU/Memory 사용량을 주기적으로 모니터링하고 metrics 정보를 수집하여 API에 제공

 - kubectl top 커맨드 지원

 - Horizontal Pod Autoscaler를 통해 원하는 Replicas 수 결정 :

   원하는 레플리카 수 = ceil[현재 레플리카 수 * ( 현재 메트릭 값 / 원하는 메트릭 값)]

 

Merics-server 설치

// 현재 metrics-server 미설치 상태
kubectl top nodes
error: Metrics API not available

kubectl top pod
error: Metrics API not available

// metrics-server 설치
git clone  https://github.com/237summit/kubernetes-metrics-server.git

cd kubernetes-metrics-server

kubectl apply -f .

kubectl get deploy -A 
NAMESPACE     NAME             READY   UP-TO-DATE   AVAILABLE   AGE
default       web              3/3     3            3           47h
kube-system   coredns          2/2     2            2           79d
kube-system   metrics-server   1/1     1            1           18s

kubectl top nodes
NAME                 CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master.example.com   264m         13%    1025Mi          74%       
node1.example.com    130m         6%     468Mi           75%       
node2.example.com    63m          3%     572Mi           65%       

kubectl top pods -A
NAMESPACE     NAME                                         CPU(cores)   MEMORY(bytes)   
default       client-pod                                   0m           2Mi             
default       dns-example                                  0m           5Mi             
default       web-67b56f4c4c-5r7qm                         0m           1Mi             
default       web-67b56f4c4c-l8bw5                         0m           1Mi             
default       web-67b56f4c4c-thlqw                         0m           4Mi             
kube-system   coredns-565d847f94-k9kvr                     5m           12Mi            
kube-system   coredns-565d847f94-tdhmn                     4m           14Mi            
kube-system   etcd-master.example.com                      45m          45Mi            
kube-system   kube-apiserver-master.example.com            84m          306Mi           
kube-system   kube-controller-manager-master.example.com   34m          52Mi            
kube-system   kube-proxy-6529g                             1m           16Mi            
kube-system   kube-proxy-skhgb                             3m           12Mi            
kube-system   kube-proxy-x5gsk                             2m           19Mi            
kube-system   kube-scheduler-master.example.com            6m           22Mi            
kube-system   metrics-server-84fc559c86-xhgsg              10m          15Mi            
kube-system   weave-net-bwqrp                              2m           49Mi            
kube-system   weave-net-r8v7n                              2m           53Mi            
kube-system   weave-net-t92nl                              8m           44Mi

 

Autoscaling 테스트 진행을 위한 deploy, service 준비

cat deploy_web.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - image: smlinux/hpa-example
        name: web
        ports:
        - containerPort: 80
        resources: 
          requests: 
            cpu: 200m  
---
apiVersion: v1
kind: Service
metadata:
  name: svc-web
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: web

kubectl apply -f deploy_web.yaml
deployment.apps/deploy-web created
service/svc-web created

kubectl get all
NAME                              READY   STATUS              RESTARTS   AGE
pod/deploy-web-59899f68c4-962rh   0/1     ContainerCreating   0          8s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   59d
service/svc-web      ClusterIP   10.96.169.111   <none>        80/TCP    8s

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/deploy-web   0/1     1            0           9s

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/deploy-web-59899f68c4   1         1         0       9s

NAME                   COMPLETIONS   DURATION   AGE
job.batch/centos-job   0/1           60d        60d

kubectl get deploy,service
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/deploy-web   1/1     1            1           28s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   59d
service/svc-web      ClusterIP   10.96.169.111   <none>        80/TCP    28s

kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
deploy-web-59899f68c4-962rh   1/1     Running   0          69s

 

Horizontal Pod Autoscaling 생성

 - 이름은 hpe-web 으로 만들고 scaleTargetRef 의 replicas 를 조정하게 요청

 - cpu 사용률이 50% 이상이면 min: 1, max: 10 까지 증설

cat hpa_web.yaml 
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: hpe-web
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deploy-web
  targetCPUUtilizationPercentage: 50

 

hpa_web.yaml 파일을 실행

kubectl apply -f hpa_web.yaml 
horizontalpodautoscaler.autoscaling/hpe-web created

kubectl get hpa
NAME      REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
hpe-web   Deployment/deploy-web   <unknown>/50%   1         10        1         49s

 

kubetctl get hpa로 확인시 TARGETS 에 <unknown> 이 계속 나타나는것이 확인되며, metrics-server 설치가 제대로 되지 않는거라고 판단하여 재설치를 진행함

- --kubelet-insecure-tls 추가

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

kubectl edit deploy -n kube-system metrics-server
   spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443        
        - --kubelet-insecure-tls              -> 추가
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s

※ yaml 파일 수정시 아래와 같은 에러가 나타났을 경우에는 tab 대신 스페이스를 이용해서 작성해보자

"error: error parsing components.yaml: error converting YAML to JSON: yaml: line 24: found a tab character that violates indentation"

 

kubectl top nodes 명령어 사용시 아래와 같은 에러가 발생한다.

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

 

metrics-server 를 수정해서 spec.template.spec 라인에 hostNetwork: true 를 추가

kubectl top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

kubectl edit deploy -n kube-system metrics-server
      dnsPolicy: ClusterFirst
      hostNetwork: true       -> 추가
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      restartPolicy: Always

 

hpa 확인시 TARGETS 부분 정상 확인

kubectl get hpa
NAME      REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpe-web   Deployment/deploy-web   0%/50%    1         10        1          44m

 

curl 명령을 통해 request 요청을 주면 cpu 사용률이 높아지고 REPLICAS 개수가 증가됨

curl 명령 종료하면 축소는 대기 후 5분 정도 뒤에 진행됨 

while true
> do
> curl 10.96.169.111
> done

OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!

kubectl get hpa
NAME      REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpe-web   Deployment/deploy-web   64%/50%   1         10        9          50m

kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
deploy-web-59899f68c4-6m2s5   1/1     Running   0          41s
deploy-web-59899f68c4-95r8h   1/1     Running   0          41s
deploy-web-59899f68c4-962rh   1/1     Running   0          62m
deploy-web-59899f68c4-cfngl   1/1     Running   0          41s
deploy-web-59899f68c4-gmtv2   1/1     Running   0          41s
deploy-web-59899f68c4-hp4pk   1/1     Running   0          26s
deploy-web-59899f68c4-lvkrj   1/1     Running   0          56s
deploy-web-59899f68c4-qxfz5   1/1     Running   0          56s
deploy-web-59899f68c4-thd8s   1/1     Running   0          56s

// 요청 종료 후 5분 정도 지난 후 축소된 hpa 확인
kubectl get hpa
NAME      REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpe-web   Deployment/deploy-web   0%/50%    1         10        1          60m

 

hpa는 CPU 뿐만 아니라 메모리 사용률로도 Autoscaling 가능함

 

알고리즘 세부 정보
가장 기본적인 관점에서, HorizontalPodAutoscaler 컨트롤러는 원하는(desired) 메트릭 값과 현재(current) 메트릭 값 사이의 비율로 작동한다.

원하는 레플리카 수 = ceil[현재 레플리카 수 * ( 현재 메트릭 값 / 원하는 메트릭 값 )]
예를 들어 현재 메트릭 값이 200m이고 원하는 값이 100m인 경우 200.0 / 100.0 == 2.0이므로 복제본 수가 두 배가 된다. 만약 현재 값이 50m 이면, 50.0 / 100.0 == 0.5 이므로 복제본 수를 반으로 줄일 것이다. 컨트롤 플레인은 비율이 1.0(기본값이 0.1인 -horizontal-pod-autoscaler-tolerance 플래그를 사용하여 전역적으로 구성 가능한 허용 오차 내)에 충분히 가깝다면 스케일링을 건너 뛸 것이다.

targetAverageValue 또는 targetAverageUtilization가 지정되면, currentMetricValue는 HorizontalPodAutoscaler의 스케일 목표 안에 있는 모든 파드에서 주어진 메트릭의 평균을 취하여 계산된다.

 

 

 

[참고]

- 유투브 따배쿠 강의

- https://kubernetes.io/ko/docs/tasks/run-application/horizontal-pod-autoscale/