[Kubernetes] Install Kubernetes(v1.29.x) using Kubekey(v3.1.1) Artifact on Multipass

offline 설치 위한 artifact 참고

version 참고
- kubernetes와 관련된 image는 https://github.com/kubesphere/ks-installer/releases에서 주요 release에만 포함되는 image-list.txt파일을 참고
- kubekey의 버전별로 kubernetes, kubesphere의 최신 지원 버전이 있음
  - kubekey/version/components.json
  - kubekey/cmd/kk/pkg/version/kubesphere/version_enum.go
  - kubekey/cmd/kk/pkg/version/kubernetes/version_enum.go
- default 버전에 대한 설정은 kubekey/cmd/kk/apis/kubekey/v1alpha2/default.go 파일에 있다
https://github.com/kubesphere/kubekey/blob/v3.1.1/docs/manifest_and_artifact.md
https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/images-list.txt
https://kubesphere.io/docs/v3.4/installing-on-linux/introduction/air-gapped-installation
https://github.com/kubesphere/kubekey/blob/v3.1.1/docs/manifest-example.md {: .prompt-info }

Multipass 접속을 위한 ssh key 생성 #

1ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa_multipass

cloud-init 구성 #

cloud-init 생성
```
1vi cloud-init.yaml
```

cloud-init 작성

 1users:
 2  - default
 3  - name: root
 4    sudo: ALL=(ALL) NOPASSWD:ALL
 5    ssh_authorized_keys:
 6      - <content of YOUR public key>
 7
 8  - name: ubuntu
 9    sudo: ALL=(ALL) NOPASSWD:ALL
10    ssh_authorized_keys:
11      - <content of YOUR public key>
12
13runcmd:
14  - sudo apt-get update
15  - sudo timedatectl set-timezone "Asia/Seoul"
16  - sudo swapoff -a
17  - sudo sed -i "/swap/d" /etc/fstab
18  - sudo apt-get install -y conntrack
19  - sudo apt-get install -y socat

cloud-init의 ssh_authorized_keys 설정을 하지 않았을 시 #

각 Node의 ~/.ssh 경로의 있는 authorized_keys에 id_rsa_multipass.pub 내용 붙여넣기
```
1cat $HOME/.ssh/id_rsa_multipass.pub
```

root 계정일 때

1# root 접속
2sudo -i
3
4# 수정 또는 .ssh 폴더 생성 후 authorized_keys 작성
5vi .ssh/authorized_keys

Multipass 생성 #

Repository 생성

1multipass launch focal --name kk-repo --memory 8G --disk 100G --cpus 4 --network name=multipass,mode=manual --cloud-init cloud-init.yaml

Master 생성

1multipass launch focal --name kk-master --memory 8G --disk 50G --cpus 4 --network name=multipass,mode=manual --cloud-init cloud-init.yaml

Worker-1 생성

1multipass launch focal --name kk-worker-1 --memory 8G --disk 50G --cpus 4 --network name=multipass,mode=manual --cloud-init cloud-init.yaml

Worker-2 생성

1multipass launch focal --name kk-worker-2 --memory 8G --disk 50G --cpus 4 --network name=multipass,mode=manual --cloud-init cloud-init.yaml

Multipass 접속 #

kk-repo

1ssh -i $HOME/.ssh/id_rsa_multipass ubuntu@192.168.0.100

1multipass shell kk-repo

kk-master

1ssh -i $HOME/.ssh/id_rsa_multipass ubuntu@192.168.0.101

1multipass shell kk-master

kk-worker-1

1ssh -i $HOME/.ssh/id_rsa_multipass ubuntu@192.168.0.102

1multipass shell kk-worker-1

kk-worker-2

1ssh -i $HOME/.ssh/id_rsa_multipass ubuntu@192.168.0.103

1multipass shell kk-worker-2

각 Node별로 Static IP 설정 #

1sudo vi /etc/netplan/50-cloud-init.yaml

kk-repo

 1network:
 2    ethernets:
 3        eth0:
 4            dhcp4: true
 5            dhcp6: true
 6            match:
 7                macaddress: 52:54:00:80:6b:21
 8            set-name: eth0
 9--- 추가
10        eth1:
11            addresses: [192.168.0.100/24]
12            gateway4: 192.168.0.1
13            dhcp4: no
14---
15    version: 2

kk-master

 1network:
 2    ethernets:
 3        eth0:
 4            dhcp4: true
 5            dhcp6: true
 6            match:
 7                macaddress: 52:54:00:80:6b:21
 8            set-name: eth0
 9--- 추가
10        eth1:
11            addresses: [192.168.0.101/24]
12            gateway4: 192.168.0.1
13            dhcp4: no
14---
15    version: 2

kk-worker-1

 1network:
 2    ethernets:
 3        eth0:
 4            dhcp4: true
 5            dhcp6: true
 6            match:
 7                macaddress: 52:54:00:80:6b:21
 8            set-name: eth0
 9--- 추가
10        eth1:
11            addresses: [192.168.0.102/24]
12            gateway4: 192.168.0.1
13            dhcp4: no
14---
15    version: 2

kk-worker-2

 1network:
 2    ethernets:
 3        eth0:
 4            dhcp4: true
 5            dhcp6: true
 6            match:
 7                macaddress: 52:54:00:80:6b:21
 8            set-name: eth0
 9--- 추가
10        eth1:
11            addresses: [192.168.0.103/24]
12            gateway4: 192.168.0.1
13            dhcp4: no
14---
15    version: 2

kubekey artifact 구성 및 설치 #

script 다운로드 #

1curl -sfL https://get-kk.kubesphere.io | VERSION=v3.1.1 sh -

ubuntu-20.04-debs-amd64.iso 다운로드 #

1wget https://github.com/kubesphere/kubekey/releases/download/v3.1.1/ubuntu-20.04-debs-amd64.iso

artifact-3.1.1.yaml 작성 #

  1apiVersion: kubekey.kubesphere.io/v1alpha2
  2kind: Manifest
  3metadata:
  4  name: artifact-v3.1.1
  5spec:
  6  arches:
  7  - amd64
  8  operatingSystems:
  9  - arch: amd64
 10    type: linux
 11    id: ubuntu
 12    version: "20.04"
 13    osImage: Ubuntu 20.04.4 LTS
 14    repository:
 15      iso:
 16        localPath: "/home/ubuntu/kk_install/ubuntu-20.04-debs-amd64.iso"
 17        # url: "https://github.com/kubesphere/kubekey/releases/download/v3.1.1/ubuntu-20.04-debs-amd64.iso"
 18  kubernetesDistributions:
 19  - type: kubernetes
 20    version: v1.29.3
 21  components:
 22    helm:
 23      version: v3.14.3
 24    cni:
 25      version: v1.2.0
 26    etcd:
 27      version: v3.5.13
 28    calicoctl:
 29      version: v3.27.3
 30    ## For now, if your cluster container runtime is containerd, KubeKey will add a docker 20.10.8 container runtime in the below list.
 31    ## The reason is KubeKey creates a cluster with containerd by installing a docker first and making kubelet connect the socket file of containerd which docker contained.
 32    containerRuntimes:
 33    - type: docker
 34      version: 24.0.9
 35    - type: containerd
 36      version: 1.7.13
 37    crictl:
 38      version: v1.29.0
 39    docker-registry:
 40      version: "2"
 41    harbor:
 42      version: v2.10.1
 43    docker-compose:
 44      version: v2.26.1
 45  images:
 46  - docker.io/kubesphere/kube-apiserver:v1.29.3
 47  - docker.io/kubesphere/kube-controller-manager:v1.29.3
 48  - docker.io/kubesphere/kube-scheduler:v1.29.3
 49  - docker.io/kubesphere/kube-proxy:v1.29.3
 50  - docker.io/kubesphere/pause:3.9
 51  - docker.io/coredns/coredns:1.9.3
 52  - docker.io/calico/cni:v3.23.2
 53  - docker.io/calico/cni:v3.27.3
 54  - docker.io/calico/kube-controllers:v3.23.2
 55  - docker.io/calico/kube-controllers:v3.27.3
 56  - docker.io/calico/node:v3.23.2
 57  - docker.io/calico/node:v3.27.3
 58  - docker.io/calico/pod2daemon-flexvol:v3.23.2
 59  - docker.io/calico/typha:v3.23.2
 60  - docker.io/kubesphere/flannel:v0.12.0
 61  - docker.io/openebs/provisioner-localpv:3.3.0
 62  - docker.io/openebs/linux-utils:3.3.0
 63  - docker.io/library/haproxy:2.3
 64  - docker.io/kubesphere/nfs-subdir-external-provisioner:v4.0.2
 65  - docker.io/kubesphere/k8s-dns-node-cache:1.22.20
 66  - docker.io/kubesphere/k8s-dns-node-cache:1.15.12
 67  # https://github.com/kubesphere/ks-installer/releases/download/v3.3.2/images-list.txt
 68  ##kubesphere-images
 69  - docker.io/kubesphere/ks-installer:v3.4.1
 70  - docker.io/kubesphere/ks-apiserver:v3.4.1
 71  - docker.io/kubesphere/ks-console:v3.4.1
 72  - docker.io/kubesphere/ks-controller-manager:v3.4.1
 73  - docker.io/kubesphere/kubectl:v1.22.0
 74  - docker.io/kubesphere/kubefed:v0.8.1
 75  - docker.io/kubesphere/tower:v0.2.1
 76  - docker.io/minio/minio:RELEASE.2019-08-07T01-59-21Z
 77  - docker.io/minio/mc:RELEASE.2019-08-07T23-14-43Z
 78  - docker.io/csiplugin/snapshot-controller:v4.0.0
 79  - docker.io/kubesphere/nginx-ingress-controller:v1.3.1
 80  - docker.io/mirrorgooglecontainers/defaultbackend-amd64:1.4
 81  - docker.io/kubesphere/metrics-server:v0.4.2
 82  - docker.io/library/redis:5.0.14-alpine
 83  - docker.io/library/haproxy:2.0.25-alpine
 84  - docker.io/library/alpine:3.14
 85  - docker.io/osixia/openldap:1.3.0
 86  - docker.io/kubesphere/netshoot:v1.0
 87  ##kubeedge-images
 88  - docker.io/kubeedge/cloudcore:v1.13.0
 89  - docker.io/kubesphere/iptables-manager:v1.13.0
 90  - docker.io/kubeedge/iptables-manager:v1.9.2
 91  - docker.io/kubesphere/edgeservice:v0.3.0
 92  - docker.io/kubesphere/edgeservice:v0.2.0
 93  ##gatekeeper-images
 94  - docker.io/openpolicyagent/gatekeeper:v3.5.2
 95  ##openpitrix-images
 96  - docker.io/kubesphere/openpitrix-jobs:v3.3.2
 97  ##kubesphere-devops-images
 98  - docker.io/kubesphere/devops-apiserver:ks-v3.4.1
 99  - docker.io/kubesphere/devops-controller:ks-v3.4.1
100  - docker.io/kubesphere/devops-tools:ks-v3.4.1
101  - docker.io/kubesphere/ks-jenkins:v3.4.0-2.319.3-1
102  - docker.io/jenkins/inbound-agent:4.10-2
103  - docker.io/kubesphere/builder-base:v3.2.2
104  - docker.io/kubesphere/builder-nodejs:v3.2.0
105  - docker.io/kubesphere/builder-maven:v3.2.1-jdk11
106  - docker.io/kubesphere/builder-maven:v3.2.0
107  - docker.io/kubesphere/builder-python:v3.2.0
108  - docker.io/kubesphere/builder-go:v3.2.2-1.18
109  - docker.io/kubesphere/builder-go:v3.2.2-1.17
110  - docker.io/kubesphere/builder-go:v3.2.2-1.16
111  - docker.io/kubesphere/builder-go:v3.2.0
112  - docker.io/kubesphere/builder-base:v3.2.2-podman
113  - docker.io/kubesphere/builder-nodejs:v3.2.0-podman
114  - docker.io/kubesphere/builder-maven:v3.2.1-jdk11-podman
115  - docker.io/kubesphere/builder-maven:v3.2.0-podman
116  - docker.io/kubesphere/builder-python:v3.2.0-podman
117  - docker.io/kubesphere/builder-go:v3.2.0-podman
118  - docker.io/kubesphere/builder-go:v3.2.2-1.18-podman
119  - docker.io/kubesphere/builder-go:v3.2.2-1.17-podman
120  - docker.io/kubesphere/builder-go:v3.2.2-1.16-podman
121  - docker.io/kubesphere/s2ioperator:v3.2.1
122  - docker.io/kubesphere/s2irun:v3.2.0
123  - docker.io/kubesphere/s2i-binary:v3.2.0
124  - docker.io/kubesphere/tomcat85-java11-centos7:v3.2.0
125  - docker.io/kubesphere/tomcat85-java11-runtime:v3.2.0
126  - docker.io/kubesphere/tomcat85-java8-centos7:v3.2.0
127  - docker.io/kubesphere/tomcat85-java8-runtime:v3.2.0
128  - docker.io/kubesphere/java-11-centos7:v3.2.0
129  - docker.io/kubesphere/java-11-runtime:v3.2.0
130  - docker.io/kubesphere/java-8-centos7:v3.2.0
131  - docker.io/kubesphere/java-8-runtime:v3.2.0
132  - docker.io/kubesphere/nodejs-8-centos7:v3.2.0
133  - docker.io/kubesphere/nodejs-6-centos7:v3.2.0
134  - docker.io/kubesphere/nodejs-4-centos7:v3.2.0
135  - docker.io/kubesphere/python-36-centos7:v3.2.0
136  - docker.io/kubesphere/python-35-centos7:v3.2.0
137  - docker.io/kubesphere/python-34-centos7:v3.2.0
138  - docker.io/kubesphere/python-27-centos7:v3.2.0
139  - quay.io/argoproj/argocd:v2.3.3
140  - quay.io/argoproj/argocd-applicationset:v0.4.1
141  - ghcr.io/dexidp/dex:v2.30.2
142  - docker.io/library/redis:6.2.6-alpine
143  ##kubesphere-monitoring-images
144  - docker.io/jimmidyson/configmap-reload:v0.7.1
145  - docker.io/prom/prometheus:v2.39.1
146  - docker.io/kubesphere/prometheus-config-reloader:v0.55.1
147  - docker.io/kubesphere/prometheus-operator:v0.55.1
148  - docker.io/kubesphere/kube-rbac-proxy:v0.11.0
149  - docker.io/kubesphere/kube-state-metrics:v2.6.0
150  - docker.io/prom/node-exporter:v1.3.1
151  - docker.io/prom/alertmanager:v0.23.0
152  - docker.io/thanosio/thanos:v0.31.0
153  - docker.io/grafana/grafana:8.3.3
154  - docker.io/kubesphere/kube-rbac-proxy:v0.11.0
155  - docker.io/kubesphere/notification-manager-operator:v2.3.0
156  - docker.io/kubesphere/notification-manager:v2.3.0
157  - docker.io/kubesphere/notification-tenant-sidecar:v3.2.0
158  ##kubesphere-logging-images
159  - docker.io/kubesphere/elasticsearch-curator:v5.7.6
160  - docker.io/kubesphere/opensearch-curator:v0.0.5
161  - docker.io/kubesphere/elasticsearch-oss:6.8.22
162  - docker.io/opensearchproject/opensearch:2.6.0
163  - docker.io/opensearchproject/opensearch-dashboards:2.6.0
164  - docker.io/kubesphere/fluentbit-operator:v0.14.0
165  - docker.io/library/docker:19.03
166  - docker.io/kubesphere/fluent-bit:v1.9.4
167  - docker.io/kubesphere/log-sidecar-injector:v1.2.0
168  - docker.io/elastic/filebeat:6.7.0
169  - docker.io/kubesphere/kube-events-operator:v0.6.0
170  - docker.io/kubesphere/kube-events-ruler:v0.6.0
171  - docker.io/kubesphere/kube-auditing-operator:v0.2.0
172  - docker.io/kubesphere/kube-auditing-webhook:v0.2.0
173  ##istio-images
174  - docker.io/istio/pilot:1.14.6
175  - docker.io/istio/proxyv2:1.14.6
176  - docker.io/jaegertracing/jaeger-operator:1.29
177  - docker.io/jaegertracing/jaeger-agent:1.29
178  - docker.io/jaegertracing/jaeger-collector:1.29
179  - docker.io/jaegertracing/jaeger-query:1.29
180  - docker.io/jaegertracing/jaeger-es-index-cleaner:1.29
181  - docker.io/kubesphere/kiali-operator:v1.50.1
182  - docker.io/kubesphere/kiali:v1.50
183  # ##example-images
184  # - docker.io/library/busybox:1.31.1
185  # - docker.io/library/nginx:1.14-alpine
186  # - docker.io/joosthofman/wget:1.0
187  # - docker.io/nginxdemos/hello:plain-text
188  # - docker.io/library/wordpress:4.8-apache
189  # - docker.io/mirrorgooglecontainers/hpa-example:latest
190  # - docker.io/fluent/fluentd:v1.4.2-2.0
191  # - docker.io/library/perl:latest
192  # - docker.io/kubesphere/examples-bookinfo-productpage-v1:1.16.2
193  # - docker.io/kubesphere/examples-bookinfo-reviews-v1:1.16.2
194  # - docker.io/kubesphere/examples-bookinfo-reviews-v2:1.16.2
195  # - docker.io/kubesphere/examples-bookinfo-details-v1:1.16.2
196  # - docker.io/kubesphere/examples-bookinfo-ratings-v1:1.16.3
197  # ##weave-scope-images
198  # - docker.io/weaveworks/scope:1.13.0
199  registry:
200    auths:
201      "docker.io":
202        username: "username"
203        password: "password"

components version 확인(지원하는 version이 없을 시 아래와 같이 Error) #

Failed to download docker binary: curl -L -o /home/ubuntu/kk_install/kubekey/artifact/docker/20.10.8/amd64/docker-20.10.8.tgz https://download.docker.com/linux/static/stable/x86_64/docker-20.10.8.tgz error: No SHA256 found for docker. 20.10.8 is not supported.
17:40:24 KST failed: [LocalHost]
error: Pipeline[ArtifactExportPipeline] execute failed: Module[ArtifactBinariesModule] exec failed:
failed: [LocalHost] [DownloadBinaries] exec failed after 1 retries: Failed to download docker binary: curl -L -o /home/ubuntu/kk_install/kubekey/artifact/docker/20.10.8/amd64/docker-20.10.8.tgz https://download.docker.com/linux/static/stable/x86_64/docker-20.10.8.tgz error: No SHA256 found for docker. 20.10.8 is not supported.

Components 참고

https://github.com/kubesphere/kubekey/blob/v3.1.1/version/components.json {: .prompt-info }

Export Artifact #

1sudo ./kk artifact export -m artifact-3.1.1.yaml -o artifact-3.1.1.tar.gz

Cluster 설치를 위한 config 파일 생성 및 작성 #

config 파일 생성

1sudo ./kk create config --with-kubesphere v3.3.2 --with-kubernetes v1.29.3 -f config-v1.29.3.yaml

config 파일 편집
```
1vi config-v1.29.3.yaml
```

config 파일 작성

  1apiVersion: kubekey.kubesphere.io/v1alpha2
  2kind: Cluster
  3metadata:
  4  name: sample
  5spec:
  6  hosts:
  7  - {name: kk-repo, address: 192.168.0.100, internalAddress: 192.168.0.100, privateKeyPath: "/home/ubuntu/.ssh/id_rsa_multipass"}
  8  - {name: kk-master, address: 192.168.0.101, internalAddress: 192.168.0.101, privateKeyPath: "/home/ubuntu/.ssh/id_rsa_multipass"}
  9  - {name: kk-worker-1, address: 192.168.0.102, internalAddress: 192.168.0.102, privateKeyPath: "/home/ubuntu/.ssh/id_rsa_multipass"}
 10  - {name: kk-worker-2, address: 192.168.0.103, internalAddress: 192.168.0.103, privateKeyPath: "/home/ubuntu/.ssh/id_rsa_multipass"}
 11  roleGroups:
 12    etcd:
 13    - kk-master
 14    control-plane:
 15    - kk-master
 16    worker:
 17    - kk-worker-1
 18    - kk-worker-2
 19    registry:
 20    - kk-repo
 21  controlPlaneEndpoint:
 22    ## Internal loadbalancer for apiservers
 23    # internalLoadbalancer: haproxy
 24
 25    domain: lb.kubesphere.local
 26    # domain: 192.168.0.101
 27    address: "192.168.0.101"
 28    port: 6443
 29  kubernetes:
 30    version: v1.29.3
 31    imageRepo: kubesphere
 32    clusterName: cluster.local
 33    masqueradeAll: false
 34    maxPods: 150
 35    nodeCidrMaskSize: 24
 36    proxyMode: ipvs
 37    autoRenewCerts: true
 38    containerManager: containerd
 39    featureGates:
 40      RotateKubeletServerCertificate: true
 41    apiserverArgs:
 42    - default-not-ready-toleration-seconds=30
 43    - default-unreachable-toleration-seconds=30
 44    controllerManagerArgs:
 45    - node-monitor-period=2s
 46    - node-monitor-grace-period=16s
 47    kubeletConfiguration:
 48      nodeStatusUpdateFrequency: 4s
 49  # etcd:
 50    # type: kubekey
 51  network:
 52    plugin: calico
 53    calico:
 54      ipipMode: Always
 55      vxianMode: Never
 56      vethMTU: 1440
 57    kubePodsCIDR: 10.233.64.0/18
 58    kubeServiceCIDR: 10.233.0.0/18
 59    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
 60    multusCNI:
 61      enabled: false
 62  registry:
 63    type: harbor
 64    auths:
 65      "cr.harbor.kubekey.com":
 66        username: admin
 67        password: Harbor12345
 68    privateRegistry: "cr.harbor.kubekey.com"
 69    namespaceOverride: "kubesphereio"
 70    registryMirrors: []
 71    insecureRegistries: ["cr.harbor.kubekey.com"]
 72  addons: []
 73---
 74apiVersion: installer.kubesphere.io/v1alpha1
 75kind: ClusterConfiguration
 76metadata:
 77  name: ks-installer
 78  namespace: kubesphere-system
 79  labels:
 80    version: v3.4.1
 81spec:
 82  persistence:
 83    storageClass: ""
 84  authentication:
 85    jwtSecret: ""
 86  zone: ""
 87  local_registry: ""
 88  namespace_override: ""
 89  # dev_tag: ""
 90  etcd:
 91    monitoring: false
 92    endpointIps: localhost
 93    port: 2379
 94    tlsEnable: true
 95  common:
 96    core:
 97      console:
 98        enableMultiLogin: true
 99        port: 30880
100        type: NodePort
101    # apiserver:
102    #  resources: {}
103    # controllerManager:
104    #  resources: {}
105    redis:
106      enabled: false
107      volumeSize: 2Gi
108    openldap:
109      enabled: false
110      volumeSize: 2Gi
111    minio:
112      volumeSize: 20Gi
113    monitoring:
114      # type: external
115      endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
116      GPUMonitoring:
117        enabled: false
118    gpu:
119      kinds:
120      - resourceName: "nvidia.com/gpu"
121        resourceType: "GPU"
122        default: true
123    es:
124      # master:
125      #   volumeSize: 4Gi
126      #   replicas: 1
127      #   resources: {}
128      # data:
129      #   volumeSize: 20Gi
130      #   replicas: 1
131      #   resources: {}
132      logMaxAge: 7
133      elkPrefix: logstash
134      basicAuth:
135        enabled: false
136        username: ""
137        password: ""
138      externalElasticsearchHost: ""
139      externalElasticsearchPort: ""
140  alerting:
141    enabled: false
142    # thanosruler:
143    #   replicas: 1
144    #   resources: {}
145  auditing:
146    enabled: false
147    # operator:
148    #   resources: {}
149    # webhook:
150    #   resources: {}
151  devops:
152    enabled: false
153    # resources: {}
154    jenkinsMemoryLim: 8Gi
155    jenkinsMemoryReq: 4Gi
156    jenkinsVolumeSize: 8Gi
157  events:
158    enabled: false
159    # operator:
160    #   resources: {}
161    # exporter:
162    #   resources: {}
163    # ruler:
164    #   enabled: true
165    #   replicas: 2
166    #   resources: {}
167  logging:
168    enabled: false
169    logsidecar:
170      enabled: true
171      replicas: 2
172      # resources: {}
173  metrics_server:
174    enabled: false
175  monitoring:
176    storageClass: ""
177    node_exporter:
178      port: 9100
179      # resources: {}
180    # kube_rbac_proxy:
181    #   resources: {}
182    # kube_state_metrics:
183    #   resources: {}
184    # prometheus:
185    #   replicas: 1
186    #   volumeSize: 20Gi
187    #   resources: {}
188    #   operator:
189    #     resources: {}
190    # alertmanager:
191    #   replicas: 1
192    #   resources: {}
193    # notification_manager:
194    #   resources: {}
195    #   operator:
196    #     resources: {}
197    #   proxy:
198    #     resources: {}
199    gpu:
200      nvidia_dcgm_exporter:
201        enabled: false
202        # resources: {}
203  multicluster:
204    clusterRole: none
205  network:
206    networkpolicy:
207      enabled: false
208    ippool:
209      type: none
210    topology:
211      type: none
212  openpitrix:
213    store:
214      enabled: false
215  servicemesh:
216    enabled: false
217    istio:
218      components:
219        ingressGateways:
220        - name: istio-ingressgateway
221          enabled: false
222        cni:
223          enabled: false
224  edgeruntime:
225    enabled: false
226    kubeedge:
227      enabled: false
228      cloudCore:
229        cloudHub:
230          advertiseAddress:
231            - ""
232        service:
233          cloudhubNodePort: "30000"
234          cloudhubQuicNodePort: "30001"
235          cloudhubHttpsNodePort: "30002"
236          cloudstreamNodePort: "30003"
237          tunnelNodePort: "30004"
238        # resources: {}
239        # hostNetWork: false
240      iptables-manager:
241        enabled: true
242        mode: "external"
243        # resources: {}
244      # edgeService:
245      #   resources: {}
246  terminal:
247    timeout: 600

Repo에서 각 Node 접속을 위해 `id_rsa_multipass` 파일 복사 #

1multipass copy-files $HOME/.ssh/id_rsa_multipass kk-repo:/home/ubuntu/.ssh/id_rsa_multipass

Registry 설치 #

1sudo ./kk init registry -f config-v1.29.3.yaml -a artifact-3.1.1.tar.gz

harbor 주소 : [harbor 설치한 ip]:80 {: .prompt-info }

[ERROR] ssh error

각 node 별로 ssh가 안될시 root passwd가 맞지 않아 발생함.
Multipass에서 vm이 생성되면 root 비번을 설정해줘야 하는 듯

1sudo passwd root

{: .prompt-danger }

Harbor 인증서 복사 및 업데이트 (`harbor curl: (60) SSL certificate problem: unable to get local issuer certificate`) #

인증서 업데이트를 하지 않았을 시, 아래와 같이 Error

[WARNING ImagePull]: failed to pull image cr.harbor.kubekey.com/kubesphereio/kube-apiserver:v1.29.3: output: E0501 22:53:12.616927    4525 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"cr.harbor.kubekey.com/kubesphereio/kube-apiserver:v1.29.3\": failed to resolve reference \"cr.harbor.kubekey.com/kubesphereio/kube-apiserver:v1.29.3\": failed to do request: Head \"https://cr.harbor.kubekey.com:443/v2/kubesphereio/kube-apiserver/manifests/v1.29.3\": tls: failed to verify certificate: x509: certificate signed by unknown authority" image="cr.harbor.kubekey.com/kubesphereio/kube-apiserver:v1.29.3"
time="2025-05-01T22:53:12+09:00" level=fatal msg="pulling image: failed to pull and unpack image \"cr.harbor.kubekey.com/kubesphereio/kube-apiserver:v1.29.3\": failed to resolve reference \"cr.harbor.kubekey.com/kubesphereio/kube-apiserver:v1.29.3\": failed to do request: Head \"https://cr.harbor.kubekey.com:443/v2/kubesphereio/kube-apiserver/manifests/v1.29.3\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
, error: exit status 1

{: .prompt-danger }

Repo 및 각 Node의 인증서 복사 #

1sudo cp /etc/docker/certs.d/cr.harbor.kubekey.com/ca.crt /usr/local/share/ca-certificates/harbor-ca.crt
2sudo scp -i /home/ubuntu/.ssh/id_rsa_multipass /usr/local/share/ca-certificates/harbor-ca.crt root@192.168.0.101:/usr/local/share/ca-certificates/harbor-ca.crt
3sudo scp -i /home/ubuntu/.ssh/id_rsa_multipass /usr/local/share/ca-certificates/harbor-ca.crt root@192.168.0.102:/usr/local/share/ca-certificates/harbor-ca.crt
4sudo scp -i /home/ubuntu/.ssh/id_rsa_multipass /usr/local/share/ca-certificates/harbor-ca.crt root@192.168.0.103:/usr/local/share/ca-certificates/harbor-ca.crt

각 Node 별로 인증서 업데이트 #

1sudo update-ca-certificates

인증서 적용 확인 #

1ls -lrt /etc/ssl/certs

- harbor-ca.pem -> /usr/local/share/ca-certificates/harbor-ca.crt
- ca-certificates.crt

Container Restart #

1sudo systemctl restart containerd

Harbor Project 생성 #

Sample Bash 파일 다운로드 #

1curl -O https://raw.githubusercontent.com/kubesphere/ks-installer/master/scripts/create_project_harbor.sh

Harbor 프로젝트 수정 및 url 수정(https://dockerhub.kubekey.local) #

파일 편집
```
1vi create_project_harbor.sh
```

url 수정(https://dockerhub.kubekey.local)

 1#!/usr/bin/env bash
 2
 3# Copyright 2018 The KubeSphere Authors.
 4#
 5# Licensed under the Apache License, Version 2.0 (the "License");
 6# you may not use this file except in compliance with the License.
 7# You may obtain a copy of the License at
 8#
 9#     http://www.apache.org/licenses/LICENSE-2.0
10#
11# Unless required by applicable law or agreed to in writing, software
12# distributed under the License is distributed on an "AS IS" BASIS,
13# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14# See the License for the specific language governing permissions and
15# limitations under the License.
16
17url="https://cr.harbor.kubekey.com"  #Change the value of url to https://cr.harbor.kubekey.com.
18user="admin"
19passwd="Harbor12345"
20
21harbor_projects=(library
22    kubesphereio
23    kubesphere
24    argoproj
25    calico
26    coredns
27    openebs
28    csiplugin
29    minio
30    mirrorgooglecontainers
31    osixia
32    prom
33    thanosio
34    jimmidyson
35    grafana
36    elastic
37    istio
38    jaegertracing
39    jenkins
40    weaveworks
41    openpitrix
42    joosthofman
43    nginxdemos
44    fluent
45    kubeedge
46    openpolicyagent
47)
48
49for project in "${harbor_projects[@]}"; do
50    echo "creating $project"
51    curl -u "${user}:${passwd}" -X POST -H "Content-Type: application/json" "${url}/api/v2.0/projects" -d "{ \"project_name\": \"${project}\", \"public\": true}" -k #Add -k at the end of the curl command.
52done

파일 권한 변경 #

1chmod +x create_project_harbor.sh

실행 #

1./create_project_harbor.sh

Cluster 설치 #

1sudo ./kk create cluster -f config-v1.29.3.yaml -a artifact-3.1.1.tar.gz

Install operating system packages

1sudo ./kk create cluster -f config-v1.29.3.yaml -a artifact-3.1.1.tar.gz --with-packages

{: .prompt-tip }

image 별도로 push 방법

1sudo ./kk artifact image push -f config-v1.29.3.yaml -a artifact-3.1.1.tar.gz

{: .prompt-tip }

--skip-push-images를 추가하면 harbor에 image를 push하는 과정으로 생략할 수 있다.

1sudo ./kk create cluster -f config-v1.29.3.yaml -a artifact-3.1.1.tar.gz --skip-push-images

{: .prompt-tip }

[ERROR] Harbor에 image push 할 때 Unauthorized 에러 발생 때

다시 로그인

1docker login [your.host.com]:port -u username -p password
2sudo docker login https://cr.harbor.kubekey.com -u admin -p Harbor12345

{: .prompt-danger }

kubekey command 참고

https://github.com/kubesphere/kubekey/blob/master/docs/commands/kk-upgrade.md {: .prompt-info }

Cluster 설치하면서 log 확인 #

1kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

Kubernetes 일반 유저 일 때 #

1mkdir -p $HOME/.kube
2sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
3sudo chown $(id -u):$(id -g) $HOME/.kube/config

만약 일반 계정에서 아래와 sudo 명령어 없이 kubectl 명령어 사용시 아래와 같은 오류가 발생하면

[ERROR] error loading config file /etc/kubernetes/admin.conf: open /etc/kubernetes/admin.conf: permission denied
- 아래 명령어를 입력하면 sudo 없이 사용 가능하다.
```
1export KUBECONFIG=$HOME/.kube/config
```

{: .prompt-danger }

[ERROR] error making pod data directories: mkdir /var/lib/kubelet/pods/86cfe394-ba32-4a9f-ad65-1fb21f98a4ba: read-only file system

1chown -R kubelet:kubelet /var/lib/kubelet/pods
2chmod 750 /var/lib/kubelet/pods
3systemctl restart kubelet

{: .prompt-danger }

Cluster 설치 완료 #

 1#####################################################
 2###              Welcome to KubeSphere!           ###
 3#####################################################
 4
 5Console: http://192.168.0.101:30880
 6Account: admin
 7Password: P@88w0rd
 8NOTES：
 9  1. After you log into the console, please check the
10     monitoring status of service components in
11     "Cluster Management". If any service is not
12     ready, please wait patiently until all components
13     are up and running.
14  2. Please change the default password after login.
15
16#####################################################
17https://kubesphere.io             2025-05-01 22:32:53
18#####################################################
1922:32:54 KST success: [kk-master]
2022:32:54 KST Pipeline[CreateClusterPipeline] execute successfully
21Installation is complete.
22
23Please check the result using the command:
24
25        kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

설치 후, 이상 증세 #

Cluster 재시작 이후, 연결이 되지 않을 때 #

1Unable to connect to the server: dial tcp: lookup lb.kubesphere.local on 127.0.0.53:53: server misbehaving

kk-master, kk-worker-1, kk-worker-2 각 Node의 /etc/hosts 수정
```
1sudo vi /etc/hosts
```

아래와 같이 추가

192.168.0.100 cr.harbor.kubekey.com

192.168.0.101 lb.kubesphere.local

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 kk-worker-1 kk-worker-1
127.0.0.1 localhost

## 추가
192.168.0.100 cr.harbor.kubekey.com
192.168.0.101 lb.kubesphere.local
##

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

다만, 재시작하면 다시 초기화 됨 #

/etc/cloud/templates/hosts.debian.tmpl 해당 파일을 수정해야 다시 시작하더라도 변경됨
```
1sudo vi /etc/cloud/templates/hosts.debian.tmpl
```

아래와 같이 추가

192.168.0.100 cr.harbor.kubekey.com

192.168.0.101 lb.kubesphere.local

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 kk-worker-1 kk-worker-1
127.0.0.1 localhost

## 추가
192.168.0.100 cr.harbor.kubekey.com
192.168.0.101 lb.kubesphere.local
##

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

아래와 같이 docker-compose 재시작

1sudo -i
2cd /opt/harbor
3docker-compose restart

[Kubernetes] Install Kubernetes(v1.29.x) using Kubekey(v3.1.1) Artifact on Multipass

Multipass 접속을 위한 ssh key 생성 #

cloud-init 구성 #

cloud-init의 ssh_authorized_keys 설정을 하지 않았을 시 #

Multipass 생성 #

Multipass 접속 #

각 Node별로 Static IP 설정 #

kubekey artifact 구성 및 설치 #

script 다운로드 #

ubuntu-20.04-debs-amd64.iso 다운로드 #

artifact-3.1.1.yaml 작성 #

components version 확인(지원하는 version이 없을 시 아래와 같이 Error) #

Export Artifact #

Cluster 설치를 위한 config 파일 생성 및 작성 #

Repo에서 각 Node 접속을 위해 id_rsa_multipass 파일 복사 #

Registry 설치 #

Harbor 인증서 복사 및 업데이트 (harbor curl: (60) SSL certificate problem: unable to get local issuer certificate) #

Repo 및 각 Node의 인증서 복사 #

각 Node 별로 인증서 업데이트 #

인증서 적용 확인 #

Container Restart #

Harbor Project 생성 #

Sample Bash 파일 다운로드 #

Harbor 프로젝트 수정 및 url 수정(https://dockerhub.kubekey.local) #

파일 권한 변경 #

실행 #

Cluster 설치 #

Cluster 설치하면서 log 확인 #

Kubernetes 일반 유저 일 때 #

Cluster 설치 완료 #

설치 후, 이상 증세 #

Cluster 재시작 이후, 연결이 되지 않을 때 #

다만, 재시작하면 다시 초기화 됨 #

Harbor Login 실패 또는 Cluster에서 Image를 가져오지 못할 때 #

Repo에서 각 Node 접속을 위해 `id_rsa_multipass` 파일 복사 #

Harbor 인증서 복사 및 업데이트 (`harbor curl: (60) SSL certificate problem: unable to get local issuer certificate`) #