[Kubernetes] metrics-server

• Kubernetes의 Metrics-Server는 각 Node에 설치된 kubelet을 통해서 node 및 pod의 CPU, Memory 의 사용량 Metric을 수집

• node 리소스 사용량 확인

  1kubectl top node
  2NAME          CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
  3mp-master     180m         2%       1593Mi          20%
  4mp-worker-1   115m         1%       2409Mi          30%
  5mp-worker-2   79m          0%       786Mi           10%

• pod 리소스 사용량 확인

   1kubectl top po -A
   2NAMESPACE     NAME                                         CPU(cores)   MEMORY(bytes)
   3alloy         alloy-5ms5k                                  4m           49Mi
   4alloy         alloy-7q8fc                                  4m           49Mi
   5alloy         alloy-mcswx                                  4m           49Mi
   6grafana       grafana-55cdd669f-rmwmr                      12m          80Mi
   7kafka         kafbat-ui-kafka-ui-6fd44cc8fc-ksq8w          2m           426Mi
   8kafka         my-cluster-dual-role-0                       17m          886Mi
   9kafka         my-cluster-entity-operator-6f988ccdb-cwwq7   4m           434Mi
  10kafka         strimzi-cluster-operator-74f9cd5689-x6sjp    29m          275Mi
  11kube-system   calico-kube-controllers-658d97c59c-h5c5t     2m           22Mi
  12kube-system   calico-node-85fsz                            26m          104Mi
  13kube-system   calico-node-sz2q8                            22m          104Mi
  14kube-system   calico-node-wttp9                            24m          104Mi
  15kube-system   coredns-76f75df574-hnhwx                     2m           18Mi
  16kube-system   coredns-76f75df574-pfvts                     2m           17Mi
  17kube-system   etcd-mp-master                               32m          55Mi
  18kube-system   kube-apiserver-mp-master                     53m          329Mi
  19kube-system   kube-controller-manager-mp-master            17m          53Mi
  20kube-system   kube-proxy-64pst                             1m           22Mi
  21kube-system   kube-proxy-bnfnp                             1m           22Mi
  22kube-system   kube-proxy-h79m7                             1m           22Mi
  23kube-system   kube-scheduler-mp-master                     3m           22Mi
  24kube-system   metrics-server-596474b58-bjhf8               4m           23Mi
  25mgmt-system   gitlab-runner-56c8f85494-4hhz4               11m          78Mi

Install metrics-server 참고

• https://github.com/kubernetes-sigs/metrics-server {: .prompt-info }

Metrics-Server 설치 #

1kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Metrics-server 설치 중 오류 #

Readiness probe failed: HTTP probe failed with statuscode: 500 #

• 아래와 같이 상태가 0/1이고, Readiness probe failed: HTTP probe failed with statuscode: 500라는 Event가 발견

   1Events:
   2  Type     Reason          Age                   From               Message
   3  ----     ------          ----                  ----               -------
   4  Normal   Scheduled       16m                   default-scheduler  Successfully assigned kube-system/metrics-server-75bf97fcc9-xhrnz to mp-worker-2
   5  Normal   Pulling         16m                   kubelet            Pulling image "registry.k8s.io/metrics-server/metrics-server:v0.7.2"
   6  Normal   Pulled          16m                   kubelet            Successfully pulled image "registry.k8s.io/metrics-server/metrics-server:v0.7.2" in 14.31s (14.31s including waiting)
   7  Normal   Created         16m                   kubelet            Created container: metrics-server
   8  Normal   Started         16m                   kubelet            Started container metrics-server
   9  Warning  Unhealthy       11m (x31 over 15m)    kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  10  Normal   SandboxChanged  6m3s (x2 over 6m34s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  11  Normal   Pulled          6m1s                  kubelet            Container image "registry.k8s.io/metrics-server/metrics-server:v0.7.2" already present on machine
  12  Normal   Created         6m                    kubelet            Created container: metrics-server
  13  Normal   Started         6m                    kubelet            Started container metrics-server
  14  Warning  Unhealthy       89s (x29 over 5m39s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500

• 설치된 metrics-server에서 아래와 같이 --kubelet-insecure-tls라는 Command를 추가

  1kubectl edit deploy -n kube-system metrics-server 

   1...✂...
   2
   3    spec:
   4      containers:
   5      - args:
   6        - --cert-dir=/tmp
   7        - --secure-port=443
   8        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
   9        - --kubelet-use-node-status-port
  10        - --metric-resolution=15s
  11        - --kubelet-insecure-tls   << 추가
  12
  13...✂...

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io) #

• kubecetl top 명령어 오류

  1kubectl top node
  2
  3Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

• metrics-server를 수정해서 spec.template.spec 라인에 hostNetwork: true 를 추가

  1kubectl edit deploy -n kube-system metrics-server 

  1dnsPolicy: ClusterFirst
  2hostNetwork: true    << 추가
  3nodeSelector:
  4  kubernetes.io/os: linux

High Availability #

• On Kubernetes v1.21+

  1kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml

• On Kubernetes v1.19-1.21

  1kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability.yaml