[Kubernetes] Upgrade Kubernetes(v1.29.x) using Kubekey(v3.0.13) Artifact

offline 설치 위한 artifact 참고

version 참고
- kubernetes와 관련된 image는 https://github.com/kubesphere/ks-installer/releases에서 주요 release에만 포함되는 image-list.txt파일을 참고
- kubekey의 버전별로 kubernetes, kubesphere의 최신 지원 버전이 있음
  - kubekey/version/components.json
  - kubekey/cmd/kk/pkg/version/kubesphere/version_enum.go
  - kubekey/cmd/kk/pkg/version/kubernetes/version_enum.go
- default 버전에 대한 설정은 kubekey/cmd/kk/apis/kubekey/v1alpha2/default.go 파일에 있다
https://github.com/kubesphere/kubekey/blob/v3.0.13/docs/manifest_and_artifact.md
https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/images-list.txt
https://kubesphere.io/docs/v3.4/installing-on-linux/introduction/air-gapped-installation
https://github.com/kubesphere/kubekey/blob/v3.0.13/docs/manifest-example.md {: .prompt-info }

script 다운로드 #

1curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.13 sh -

artifact-3.0.13.yaml 작성 #

  1apiVersion: kubekey.kubesphere.io/v1alpha2
  2kind: Manifest
  3metadata:
  4  name: artifact-v3.0.13
  5spec:
  6  arches:
  7  - amd64
  8  operatingSystems:
  9  - arch: amd64
 10    type: linux
 11    id: ubuntu
 12    version: "20.04"
 13    osImage: Ubuntu 20.04.4 LTS
 14    repository:
 15      iso:
 16        localPath: ""
 17        url: "https://github.com/kubesphere/kubekey/releases/download/v3.0.13/ubuntu-20.04-debs-amd64.iso"
 18  kubernetesDistributions:
 19  - type: kubernetes
 20    version: v1.26.5
 21  components:
 22    helm:
 23      version: v3.9.0
 24    cni:
 25      version: v1.2.0
 26    etcd:
 27      version: v3.4.13
 28    calicoctl:
 29      version: v3.26.1
 30    ## For now, if your cluster container runtime is containerd, KubeKey will add a docker 20.10.8 container runtime in the below list.
 31    ## The reason is KubeKey creates a cluster with containerd by installing a docker first and making kubelet connect the socket file of containerd which docker contained.
 32    containerRuntimes:
 33    - type: docker
 34      version: 20.10.8
 35    - type: containerd
 36      version: 1.6.4
 37    crictl:
 38      version: v1.24.0
 39    docker-registry:
 40      version: "2"
 41    harbor:
 42      version: v2.5.3
 43    docker-compose:
 44      version: v2.2.2
 45  images:
 46  - docker.io/kubesphere/kube-apiserver:v1.27.2
 47  - docker.io/kubesphere/kube-apiserver:v1.26.5
 48  - docker.io/kubesphere/kube-controller-manager:v1.27.2
 49  - docker.io/kubesphere/kube-controller-manager:v1.26.5
 50  - docker.io/kubesphere/kube-scheduler:v1.27.2
 51  - docker.io/kubesphere/kube-scheduler:v1.26.5
 52  - docker.io/kubesphere/kube-proxy:v1.27.2
 53  - docker.io/kubesphere/kube-proxy:v1.26.5
 54  - docker.io/kubesphere/pause:3.8
 55  - docker.io/kubesphere/pause:3.7
 56  - docker.io/kubesphere/pause:3.6
 57  - docker.io/coredns/coredns:1.9.3
 58  - docker.io/coredns/coredns:1.8.6
 59  - docker.io/calico/cni:v3.26.1
 60  - docker.io/calico/cni:v3.23.2
 61  - docker.io/calico/kube-controllers:v3.26.1
 62  - docker.io/calico/kube-controllers:v3.23.2
 63  - docker.io/calico/node:v3.26.1
 64  - docker.io/calico/node:v3.23.2
 65  - docker.io/calico/pod2daemon-flexvol:v3.26.1
 66  - docker.io/calico/pod2daemon-flexvol:v3.23.2
 67  - docker.io/calico/typha:v3.26.1
 68  - docker.io/calico/typha:v3.23.2
 69  - docker.io/kubesphere/flannel:v0.12.0
 70  - docker.io/openebs/provisioner-localpv:3.3.0
 71  - docker.io/openebs/linux-utils:3.3.0
 72  - docker.io/library/haproxy:2.3
 73  - docker.io/kubesphere/nfs-subdir-external-provisioner:v4.0.2
 74  - docker.io/kubesphere/k8s-dns-node-cache:1.15.12
 75  # https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/images-list.txt
 76  ##kubesphere-images
 77  - docker.io/kubesphere/ks-installer:v3.4.1
 78  - docker.io/kubesphere/ks-installer:v3.3.2
 79  - docker.io/kubesphere/ks-apiserver:v3.4.1
 80  - docker.io/kubesphere/ks-apiserver:v3.3.2
 81  - docker.io/kubesphere/ks-console:v3.4.1
 82  - docker.io/kubesphere/ks-console:v3.3.2
 83  - docker.io/kubesphere/ks-controller-manager:v3.4.1
 84  - docker.io/kubesphere/ks-controller-manager:v3.3.2
 85  - docker.io/kubesphere/kubectl:v1.22.0
 86  - docker.io/kubesphere/kubectl:v1.20.0
 87  - docker.io/kubesphere/kubefed:v0.8.1
 88  - docker.io/kubesphere/tower:v0.2.1
 89  - docker.io/kubesphere/tower:v0.2.0
 90  - docker.io/minio/minio:RELEASE.2019-08-07T01-59-21Z
 91  - docker.io/minio/mc:RELEASE.2019-08-07T23-14-43Z
 92  - docker.io/csiplugin/snapshot-controller:v4.0.0
 93  - docker.io/kubesphere/nginx-ingress-controller:v1.3.1
 94  - docker.io/kubesphere/nginx-ingress-controller:v1.1.0
 95  - docker.io/mirrorgooglecontainers/defaultbackend-amd64:1.4
 96  - docker.io/kubesphere/metrics-server:v0.4.2
 97  - docker.io/library/redis:5.0.14-alpine
 98  - docker.io/library/haproxy:2.0.25-alpine
 99  - docker.io/library/alpine:3.14
100  - docker.io/osixia/openldap:1.3.0
101  - docker.io/kubesphere/netshoot:v1.0
102  ##kubeedge-images
103  # - docker.io/kubeedge/cloudcore:v1.13.0
104  # - docker.io/kubeedge/cloudcore:v1.9.2
105  # - docker.io/kubesphere/iptables-manager:v1.13.0
106  # - docker.io/kubeedge/iptables-manager:v1.9.2
107  # - docker.io/kubesphere/edgeservice:v0.3.0
108  # - docker.io/kubesphere/edgeservice:v0.2.0
109  ##gatekeeper-images
110  # - docker.io/openpolicyagent/gatekeeper:v3.5.2
111  ##openpitrix-images
112  # - docker.io/kubesphere/openpitrix-jobs:v3.3.2
113  ##kubesphere-devops-images
114  # - docker.io/kubesphere/devops-apiserver:ks-v3.4.1
115  # - docker.io/kubesphere/devops-apiserver:ks-v3.3.2
116  # - docker.io/kubesphere/devops-controller:ks-v3.4.1
117  # - docker.io/kubesphere/devops-controller:ks-v3.3.2
118  # - docker.io/kubesphere/devops-tools:ks-v3.4.1
119  # - docker.io/kubesphere/devops-tools:ks-v3.3.2
120  # - docker.io/kubesphere/ks-jenkins:v3.4.0-2.319.3-1
121  # - docker.io/kubesphere/ks-jenkins:v3.3.0-2.319.1
122  # - docker.io/jenkins/inbound-agent:4.10-2
123  # - docker.io/kubesphere/builder-base:v3.2.2
124  # - docker.io/kubesphere/builder-nodejs:v3.2.0
125  # - docker.io/kubesphere/builder-maven:v3.2.1-jdk11
126  # - docker.io/kubesphere/builder-maven:v3.2.0
127  # - docker.io/kubesphere/builder-python:v3.2.0
128  # - docker.io/kubesphere/builder-go:v3.2.2-1.18
129  # - docker.io/kubesphere/builder-go:v3.2.2-1.17
130  # - docker.io/kubesphere/builder-go:v3.2.2-1.16
131  # - docker.io/kubesphere/builder-go:v3.2.0
132  # - docker.io/kubesphere/builder-base:v3.2.2-podman
133  # - docker.io/kubesphere/builder-nodejs:v3.2.0-podman
134  # - docker.io/kubesphere/builder-maven:v3.2.1-jdk11-podman
135  # - docker.io/kubesphere/builder-maven:v3.2.0-podman
136  # - docker.io/kubesphere/builder-python:v3.2.0-podman
137  # - docker.io/kubesphere/builder-go:v3.2.0-podman
138  # - docker.io/kubesphere/builder-go:v3.2.2-1.18-podman
139  # - docker.io/kubesphere/builder-go:v3.2.2-1.17-podman
140  # - docker.io/kubesphere/builder-go:v3.2.2-1.16-podman
141  # - docker.io/kubesphere/s2ioperator:v3.2.1
142  # - docker.io/kubesphere/s2irun:v3.2.0
143  # - docker.io/kubesphere/s2i-binary:v3.2.0
144  # - docker.io/kubesphere/tomcat85-java11-centos7:v3.2.0
145  # - docker.io/kubesphere/tomcat85-java11-runtime:v3.2.0
146  # - docker.io/kubesphere/tomcat85-java8-centos7:v3.2.0
147  # - docker.io/kubesphere/tomcat85-java8-runtime:v3.2.0
148  # - docker.io/kubesphere/java-11-centos7:v3.2.0
149  # - docker.io/kubesphere/java-11-runtime:v3.2.0
150  # - docker.io/kubesphere/java-8-centos7:v3.2.0
151  # - docker.io/kubesphere/java-8-runtime:v3.2.0
152  # - docker.io/kubesphere/nodejs-8-centos7:v3.2.0
153  # - docker.io/kubesphere/nodejs-6-centos7:v3.2.0
154  # - docker.io/kubesphere/nodejs-4-centos7:v3.2.0
155  # - docker.io/kubesphere/python-36-centos7:v3.2.0
156  # - docker.io/kubesphere/python-35-centos7:v3.2.0
157  # - docker.io/kubesphere/python-34-centos7:v3.2.0
158  # - docker.io/kubesphere/python-27-centos7:v3.2.0
159  # - quay.io/argoproj/argocd:v2.3.3
160  # - quay.io/argoproj/argocd-applicationset:v0.4.1
161  # - ghcr.io/dexidp/dex:v2.30.2
162  # - docker.io/library/redis:6.2.6-alpine
163  ##kubesphere-monitoring-images
164  # - docker.io/jimmidyson/configmap-reload:v0.7.1
165  # - docker.io/jimmidyson/configmap-reload:v0.5.0
166  # - docker.io/prom/prometheus:v2.39.1
167  # - docker.io/prom/prometheus:v2.34.0
168  # - docker.io/kubesphere/prometheus-config-reloader:v0.55.1
169  # - docker.io/kubesphere/prometheus-operator:v0.55.1
170  # - docker.io/kubesphere/kube-rbac-proxy:v0.11.0
171  # - docker.io/kubesphere/kube-state-metrics:v2.6.0
172  # - docker.io/kubesphere/kube-state-metrics:v2.5.0
173  # - docker.io/prom/node-exporter:v1.3.1
174  # - docker.io/prom/alertmanager:v0.23.0
175  # - docker.io/thanosio/thanos:v0.31.0
176  # - docker.io/thanosio/thanos:v0.25.2
177  # - docker.io/grafana/grafana:8.3.3
178  # - docker.io/kubesphere/kube-rbac-proxy:v0.11.0
179  # - docker.io/kubesphere/kube-rbac-proxy:v0.8.0
180  # - docker.io/kubesphere/notification-manager-operator:v2.3.0
181  # - docker.io/kubesphere/notification-manager-operator:v1.4.0
182  # - docker.io/kubesphere/notification-manager:v2.3.0
183  # - docker.io/kubesphere/notification-manager:v1.4.0
184  # - docker.io/kubesphere/notification-tenant-sidecar:v3.2.0
185  ##kubesphere-logging-images
186  # - docker.io/kubesphere/elasticsearch-curator:v5.7.6
187  # - docker.io/kubesphere/opensearch-curator:v0.0.5
188  # - docker.io/kubesphere/elasticsearch-oss:6.8.22
189  # - docker.io/opensearchproject/opensearch:2.6.0
190  # - docker.io/opensearchproject/opensearch-dashboards:2.6.0
191  # - docker.io/kubesphere/fluentbit-operator:v0.14.0
192  # - docker.io/kubesphere/fluentbit-operator:v0.13.0
193  # - docker.io/library/docker:19.03
194  # - docker.io/kubesphere/fluent-bit:v1.9.4
195  # - docker.io/kubesphere/fluent-bit:v1.8.11
196  # - docker.io/kubesphere/log-sidecar-injector:v1.2.0
197  # - docker.io/elastic/filebeat:6.7.0
198  # - docker.io/kubesphere/kube-events-operator:v0.6.0
199  # - docker.io/kubesphere/kube-events-operator:v0.4.0
200  # - docker.io/kubesphere/kube-events-exporter:v0.6.0
201  # - docker.io/kubesphere/kube-events-exporter:v0.4.0
202  # - docker.io/kubesphere/kube-events-ruler:v0.6.0
203  # - docker.io/kubesphere/kube-events-ruler:v0.4.0
204  # - docker.io/kubesphere/kube-auditing-operator:v0.2.0
205  # - docker.io/kubesphere/kube-auditing-webhook:v0.2.0
206  ##istio-images
207  # - docker.io/istio/pilot:1.14.6
208  # - docker.io/istio/pilot:1.11.1
209  # - docker.io/istio/proxyv2:1.14.6
210  # - docker.io/istio/proxyv2:1.11.1
211  # - docker.io/jaegertracing/jaeger-operator:1.29
212  # - docker.io/jaegertracing/jaeger-operator:1.27
213  # - docker.io/jaegertracing/jaeger-agent:1.29
214  # - docker.io/jaegertracing/jaeger-agent:1.27
215  # - docker.io/jaegertracing/jaeger-collector:1.29
216  # - docker.io/jaegertracing/jaeger-collector:1.27
217  # - docker.io/jaegertracing/jaeger-query:1.29
218  # - docker.io/jaegertracing/jaeger-query:1.27
219  # - docker.io/jaegertracing/jaeger-es-index-cleaner:1.29
220  # - docker.io/jaegertracing/jaeger-es-index-cleaner:1.27
221  # - docker.io/kubesphere/kiali-operator:v1.50.1
222  # - docker.io/kubesphere/kiali-operator:v1.38.1
223  # - docker.io/kubesphere/kiali:v1.50
224  # - docker.io/kubesphere/kiali:v1.38
225  # ##example-images
226  # - docker.io/library/busybox:1.31.1
227  # - docker.io/library/nginx:1.14-alpine
228  # - docker.io/joosthofman/wget:1.0
229  # - docker.io/nginxdemos/hello:plain-text
230  # - docker.io/library/wordpress:4.8-apache
231  # - docker.io/mirrorgooglecontainers/hpa-example:latest
232  # - docker.io/fluent/fluentd:v1.4.2-2.0
233  # - docker.io/library/perl:latest
234  # - docker.io/kubesphere/examples-bookinfo-productpage-v1:1.16.2
235  # - docker.io/kubesphere/examples-bookinfo-reviews-v1:1.16.2
236  # - docker.io/kubesphere/examples-bookinfo-reviews-v2:1.16.2
237  # - docker.io/kubesphere/examples-bookinfo-details-v1:1.16.2
238  # - docker.io/kubesphere/examples-bookinfo-ratings-v1:1.16.3
239  # ##weave-scope-images
240  # - docker.io/weaveworks/scope:1.13.0
241  registry:
242    auths:
243      "docker.io":
244        username: "username"
245        password: "password"

Export Artifact #

1sudo ./kk artifact export -m artifact-3.0.13.yaml -o artifact-3.0.13.tar.gz

Cluster 설치를 위한 config 파일 생성 및 작성 #

config 파일 생성

1sudo ./kk create config --with-kubesphere v3.4.1 --with-kubernetes v1.26.5 -f config-sample.yaml

config 파일 작성

  1apiVersion: kubekey.kubesphere.io/v1alpha2
  2kind: Cluster
  3metadata:
  4  name: sample
  5spec:
  6  hosts:
  7  - {name: manage-master, address: 192.168.10.100, internalAddress: 192.168.10.100, user: root, password: vagrant}
  8  - {name: manage-worker-1, address: 192.168.10.110, internalAddress: 192.168.10.110, user: root, password: vagrant}
  9  - {name: manage-worker-2, address: 192.168.10.120, internalAddress: 192.168.10.120, user: root, password: vagrant}
 10  roleGroups:
 11    etcd:
 12    - manage-master
 13    control-plane:
 14    - manage-master
 15    worker:
 16    - manage-worker-1
 17    - manage-worker-2
 18    registry:
 19    - manage-worker-1
 20  controlPlaneEndpoint:
 21    ## Internal loadbalancer for apiservers
 22    # internalLoadbalancer: haproxy
 23
 24    #domain: lb.kubesphere.local
 25    domain: 192.168.10.100
 26    address: ""
 27    port: 6443
 28  kubernetes:
 29    version: v1.26.5
 30    clusterName: cluster.local
 31    autoRenewCerts: true
 32    containerManager: containerd
 33  etcd:
 34    type: kubekey
 35  network:
 36    plugin: calico
 37    kubePodsCIDR: 10.233.64.0/18
 38    kubeServiceCIDR: 10.233.0.0/18
 39    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
 40    multusCNI:
 41      enabled: false
 42  registry:
 43    type: harbor
 44    auths:
 45      "dockerhub.kubekey.local":
 46        username: admin
 47        password: Harbor12345
 48    privateRegistry: "dockerhub.kubekey.local"
 49    namespaceOverride: "kubesphereio"
 50    registryMirrors: []
 51    insecureRegistries: []
 52  addons: []
 53---
 54apiVersion: installer.kubesphere.io/v1alpha1
 55kind: ClusterConfiguration
 56metadata:
 57  name: ks-installer
 58  namespace: kubesphere-system
 59  labels:
 60    version: v3.4.1
 61spec:
 62  persistence:
 63    storageClass: ""
 64  authentication:
 65    jwtSecret: ""
 66  local_registry: ""
 67  # dev_tag: ""
 68  etcd:
 69    monitoring: false
 70    endpointIps: localhost
 71    port: 2379
 72    tlsEnable: true
 73  common:
 74    core:
 75      console:
 76        enableMultiLogin: true
 77        port: 30880
 78        type: NodePort
 79    # apiserver:
 80    #  resources: {}
 81    # controllerManager:
 82    #  resources: {}
 83    redis:
 84      enabled: false
 85      enableHA: false
 86      volumeSize: 2Gi
 87    openldap:
 88      enabled: false
 89      volumeSize: 2Gi
 90    minio:
 91      volumeSize: 20Gi
 92    monitoring:
 93      # type: external
 94      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
 95      GPUMonitoring:
 96        enabled: false
 97    gpu:
 98      kinds:
 99      - resourceName: "nvidia.com/gpu"
100        resourceType: "GPU"
101        default: true
102    es:
103      # master:
104      #   volumeSize: 4Gi
105      #   replicas: 1
106      #   resources: {}
107      # data:
108      #   volumeSize: 20Gi
109      #   replicas: 1
110      #   resources: {}
111      enabled: false
112      logMaxAge: 7
113      elkPrefix: logstash
114      basicAuth:
115        enabled: false
116        username: ""
117        password: ""
118      externalElasticsearchHost: ""
119      externalElasticsearchPort: ""
120    opensearch:
121      # master:
122      #   volumeSize: 4Gi
123      #   replicas: 1
124      #   resources: {}
125      # data:
126      #   volumeSize: 20Gi
127      #   replicas: 1
128      #   resources: {}
129      enabled: true
130      logMaxAge: 7
131      opensearchPrefix: whizard
132      basicAuth:
133        enabled: true
134        username: "admin"
135        password: "admin"
136      externalOpensearchHost: ""
137      externalOpensearchPort: ""
138      dashboard:
139        enabled: false
140  alerting:
141    enabled: false
142    # thanosruler:
143    #   replicas: 1
144    #   resources: {}
145  auditing:
146    enabled: false
147    # operator:
148    #   resources: {}
149    # webhook:
150    #   resources: {}
151  devops:
152    enabled: false
153    jenkinsCpuReq: 0.5
154    jenkinsCpuLim: 1
155    jenkinsMemoryReq: 4Gi
156    jenkinsMemoryLim: 4Gi
157    jenkinsVolumeSize: 16Gi
158  events:
159    enabled: false
160    # operator:
161    #   resources: {}
162    # exporter:
163    #   resources: {}
164    ruler:
165      enabled: true
166      replicas: 2
167    #   resources: {}
168  logging:
169    enabled: false
170    logsidecar:
171      enabled: true
172      replicas: 2
173      # resources: {}
174  metrics_server:
175    enabled: false
176  monitoring:
177    storageClass: ""
178    node_exporter:
179      port: 9100
180      # resources: {}
181    # kube_rbac_proxy:
182    #   resources: {}
183    # kube_state_metrics:
184    #   resources: {}
185    # prometheus:
186    #   replicas: 1
187    #   volumeSize: 20Gi
188    #   resources: {}
189    #   operator:
190    #     resources: {}
191    # alertmanager:
192    #   replicas: 1
193    #   resources: {}
194    # notification_manager:
195    #   resources: {}
196    #   operator:
197    #     resources: {}
198    #   proxy:
199    #     resources: {}
200    gpu:
201      nvidia_dcgm_exporter:
202        enabled: false
203        # resources: {}
204  multicluster:
205    clusterRole: none
206  network:
207    networkpolicy:
208      enabled: false
209    ippool:
210      type: none
211    topology:
212      type: none
213  openpitrix:
214    store:
215      enabled: false
216  servicemesh:
217    enabled: false
218    istio:
219      components:
220        ingressGateways:
221        - name: istio-ingressgateway
222          enabled: false
223        cni:
224          enabled: false
225  edgeruntime:
226    enabled: false
227    kubeedge:
228      enabled: false
229      cloudCore:
230        cloudHub:
231          advertiseAddress:
232            - ""
233        service:
234          cloudhubNodePort: "30000"
235          cloudhubQuicNodePort: "30001"
236          cloudhubHttpsNodePort: "30002"
237          cloudstreamNodePort: "30003"
238          tunnelNodePort: "30004"
239        # resources: {}
240        # hostNetWork: false
241      iptables-manager:
242        enabled: true
243        mode: "external"
244        # resources: {}
245      # edgeService:
246      #   resources: {}
247  gatekeeper:
248    enabled: false
249    # controller_manager:
250    #   resources: {}
251    # audit:
252    #   resources: {}
253  terminal:
254    timeout: 600

Upgrade #

1sudo ./kk upgrade -f config-sample.yaml -a artifact-3.0.13.tar.gz

--skip-dependency-check를 추가하면 Kubernetes 및 KubeSphere 버전 의존성 검사를 생략할 수 있다.

1sudo ./kk upgrade -f config-sample.yaml -a artifact-3.0.13.tar.gz --skip-dependency-check

{: .prompt-tip }

image 별도로 push 방법

1sudo ./kk artifact image push -f config-sample.yaml -a artifact-3.0.7.tar.gz

{: .prompt-tip }

[ERROR] Harbor에 image push 할 때 Unauthorized 에러 발생 때

다시 로그인

1docker login [your.host.com]:port -u username -p password
2sudo docker login https://cr.harbor.kubekey.com -u admin -p Harbor12345

{: .prompt-danger }

kubekey command 참고

https://github.com/kubesphere/kubekey/blob/master/docs/commands/kk-upgrade.md {: .prompt-info }

Upgrade log 확인 #

1kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f