[Kubernetes] Upgrade Kubernetes(v1.32.x) using Kubekey(v3.1.9) Artifact

offline 설치 위한 artifact 참고

version 참고
- kubernetes와 관련된 image는 https://github.com/kubesphere/ks-installer/releases에서 주요 release에만 포함되는 image-list.txt파일을 참고
- kubekey의 버전별로 kubernetes, kubesphere의 최신 지원 버전이 있음
  - kubekey/version/components.json
  - kubekey/cmd/kk/pkg/version/kubesphere/version_enum.go
  - kubekey/cmd/kk/pkg/version/kubernetes/version_enum.go
- default 버전에 대한 설정은 kubekey/cmd/kk/apis/kubekey/v1alpha2/default.go 파일에 있다
https://github.com/kubesphere/kubekey/blob/v3.1.9/docs/manifest_and_artifact.md
https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/images-list.txt
https://kubesphere.io/docs/v3.4/installing-on-linux/introduction/air-gapped-installation
https://github.com/kubesphere/kubekey/blob/v3.1.9/docs/manifest-example.md {: .prompt-info }

script 다운로드 #

1curl -sfL https://get-kk.kubesphere.io | VERSION=v3.1.9 sh -

ubuntu-20.04-debs-amd64.iso 다운로드 #

1wget https://github.com/kubesphere/kubekey/releases/download/v3.1.9/ubuntu-20.04-debs-amd64.iso

artifact-3.1.9.yaml 작성 #

 1apiVersion: kubekey.kubesphere.io/v1alpha2
 2kind: Manifest
 3metadata:
 4  name: artifact-v3.1.9
 5spec:
 6  arches:
 7  - amd64
 8  operatingSystems:
 9  - arch: amd64
10    type: linux
11    id: ubuntu
12    version: "20.04"
13    osImage: Ubuntu 20.04.4 LTS
14    repository:
15      iso:
16        localPath: "/home/ubuntu/kk_install/ubuntu-20.04-debs-amd64.iso"
17        # url: "https://github.com/kubesphere/kubekey/releases/download/v3.1.1/ubuntu-20.04-debs-amd64.iso"
18  kubernetesDistributions:
19  - type: kubernetes
20    version: v1.32.4
21  components:
22    helm:
23      version: v3.14.3
24    cni:
25      version: v1.2.0
26    etcd:
27      version: v3.5.13
28    calicoctl:
29      version: v3.27.4
30    containerRuntimes:
31    - type: containerd
32      version: 1.7.13
33    crictl:
34      version: v1.29.0
35    harbor:
36      version: v2.10.1
37    docker-compose:
38      version: v2.26.1
39  images:
40  - docker.io/kubesphere/kube-apiserver:v1.30.12
41  - docker.io/kubesphere/kube-apiserver:v1.31.8
42  - docker.io/kubesphere/kube-apiserver:v1.32.4
43  - docker.io/kubesphere/kube-controller-manager:v1.30.12
44  - docker.io/kubesphere/kube-controller-manager:v1.31.8
45  - docker.io/kubesphere/kube-controller-manager:v1.32.4
46  - docker.io/kubesphere/kube-scheduler:v1.30.12
47  - docker.io/kubesphere/kube-scheduler:v1.31.8
48  - docker.io/kubesphere/kube-scheduler:v1.32.4
49  - docker.io/kubesphere/kube-proxy:v1.30.12
50  - docker.io/kubesphere/kube-proxy:v1.31.8
51  - docker.io/kubesphere/kube-proxy:v1.32.4
52  - docker.io/kubesphere/pause:3.9
53  - docker.io/coredns/coredns:1.9.3
54  - docker.io/calico/cni:v3.23.2
55  - docker.io/calico/cni:v3.27.3
56  - docker.io/calico/kube-controllers:v3.23.2
57  - docker.io/calico/kube-controllers:v3.27.4
58  - docker.io/calico/node:v3.23.2
59  - docker.io/calico/node:v3.27.4
60  - docker.io/calico/pod2daemon-flexvol:v3.23.2
61  - docker.io/calico/pod2daemon-flexvol:v3.27.4
62  - docker.io/calico/typha:v3.23.2
63  - docker.io/calico/typha:v3.27.4
64  - docker.io/kubesphere/flannel:v0.12.0
65  - docker.io/openebs/provisioner-localpv:3.3.0
66  - docker.io/openebs/linux-utils:3.3.0
67  - docker.io/library/haproxy:2.3
68  - docker.io/kubesphere/nfs-subdir-external-provisioner:v4.0.2
69  - docker.io/kubesphere/k8s-dns-node-cache:1.15.12
70  registry:
71    auths:
72      "docker.io":
73        username: "username"
74        password: "password"

components version 확인(지원하는 version이 없을 시 아래와 같이 Error) #

Failed to download docker binary: curl -L -o /home/ubuntu/kk_install/kubekey/artifact/docker/20.10.8/amd64/docker-20.10.8.tgz https://download.docker.com/linux/static/stable/x86_64/docker-20.10.8.tgz error: No SHA256 found for docker. 20.10.8 is not supported.
17:40:24 KST failed: [LocalHost]
error: Pipeline[ArtifactExportPipeline] execute failed: Module[ArtifactBinariesModule] exec failed:
failed: [LocalHost] [DownloadBinaries] exec failed after 1 retries: Failed to download docker binary: curl -L -o /home/ubuntu/kk_install/kubekey/artifact/docker/20.10.8/amd64/docker-20.10.8.tgz https://download.docker.com/linux/static/stable/x86_64/docker-20.10.8.tgz error: No SHA256 found for docker. 20.10.8 is not supported.

Components 참고

https://github.com/kubesphere/kubekey/blob/v3.1.9/version/components.json {: .prompt-info }

Export Artifact #

1sudo ./kk artifact export -m artifact-3.1.9.yaml -o artifact-3.1.9.tar.gz

Cluster 업그레이드를 위한 config 파일 생성 및 작성 #

1sudo ./kk create config --with-kubesphere v3.3.2 --with-kubernetes v1.32.4 -f config-v1.32.4.yaml

1vi config-v1.32.4.yaml

 1apiVersion: kubekey.kubesphere.io/v1alpha2
 2kind: Cluster
 3metadata:
 4  name: sample
 5spec:
 6  hosts:
 7  - {name: kk-repo, address: 192.168.0.100, internalAddress: 192.168.0.100, privateKeyPath: "/home/ubuntu/.ssh/id_rsa_multipass"}
 8  - {name: kk-master, address: 192.168.0.101, internalAddress: 192.168.0.101, privateKeyPath: "/home/ubuntu/.ssh/id_rsa_multipass"}
 9  - {name: kk-worker-1, address: 192.168.0.102, internalAddress: 192.168.0.102, privateKeyPath: "/home/ubuntu/.ssh/id_rsa_multipass"}
10  - {name: kk-worker-2, address: 192.168.0.103, internalAddress: 192.168.0.103, privateKeyPath: "/home/ubuntu/.ssh/id_rsa_multipass"}
11  roleGroups:
12    etcd:
13    - kk-master
14    control-plane:
15    - kk-master
16    worker:
17    - kk-worker-1
18    - kk-worker-2
19    registry:
20    - kk-repo
21  controlPlaneEndpoint:
22    ## Internal loadbalancer for apiservers
23    # internalLoadbalancer: haproxy
24
25    domain: lb.kubesphere.local
26    # domain: 192.168.0.101
27    address: "192.168.0.101"
28    port: 6443
29  kubernetes:
30    version: v1.29.3
31    imageRepo: kubesphere
32    clusterName: cluster.local
33    masqueradeAll: false
34    maxPods: 150
35    nodeCidrMaskSize: 24
36    proxyMode: ipvs
37    autoRenewCerts: true
38    containerManager: containerd
39    featureGates:
40      RotateKubeletServerCertificate: true
41    apiserverArgs:
42    - default-not-ready-toleration-seconds=30
43    - default-unreachable-toleration-seconds=30
44    controllerManagerArgs:
45    - node-monitor-period=2s
46    - node-monitor-grace-period=16s
47    kubeletConfiguration:
48      nodeStatusUpdateFrequency: 4s
49  # etcd:
50    # type: kubekey
51  network:
52    plugin: calico
53    calico:
54      ipipMode: Always
55      vxianMode: Never
56      vethMTU: 1440
57    kubePodsCIDR: 10.233.64.0/18
58    kubeServiceCIDR: 10.233.0.0/18
59    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
60    multusCNI:
61      enabled: false
62  registry:
63    type: harbor
64    auths:
65      "cr.harbor.kubekey.com":
66        username: admin
67        password: Harbor12345
68    privateRegistry: "cr.harbor.kubekey.com"
69    namespaceOverride: "kubesphereio"
70    registryMirrors: []
71    insecureRegistries: ["cr.harbor.kubekey.com"]
72  addons: []

Upgrade #

1sudo ./kk upgrade cluster -f config-v1.32.4.yaml -a artifact-3.1.9.tar.gz

--skip-dependency-check를 추가하면 Kubernetes 및 KubeSphere 버전 의존성 검사를 생략할 수 있다.

1sudo ./kk upgrade cluster -f config-v1.32.4.yaml -a artifact-3.1.9.tar.gz --skip-dependency-check

{: .prompt-tip }

image 별도로 push 방법

1sudo ./kk artifact image push -f config-v1.32.4.yaml -a artifact-3.1.9.tar.gz

{: .prompt-tip }

[ERROR] Harbor에 image push 할 때 Unauthorized 에러 발생 때

다시 로그인

1docker login [your.host.com]:port -u username -p password
2sudo docker login https://cr.harbor.kubekey.com -u admin -p Harbor12345

{: .prompt-danger }

kubekey command 참고

https://github.com/kubesphere/kubekey/blob/master/docs/commands/kk-upgrade.md {: .prompt-info }

Upgrade log 확인 #

1kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f