[k8s] kubernetes 설치 (containerd 사용)

kubeadm

• k8s에서 공식 제공하는 클러스터 생성/관리 도구

 

kubespray

• k8s 클러스터를 배포하는 오픈소스 프로젝트
• 다양한 형식으로 쿠버네티스 클러스터 구성 가능
• on-premise에서 상용 서비스 클러스터 운영 시 유용
• 다양한 CNI 제공

 

CNI (Container Network Interface)

• container간 통신을 지원하는 VxLAN
• pod network라고도 부름
• 다양한 종류의 플러그 인이 존재

 

Control plane (Master node)

• worker node들의 상태를 관리하고 제어
• single master
• multi master(3, 5개의 master nodes)

 

Worker node

• docker platform을 통해 container를 동작하며 실제 서비스 제공

<이미지 출처 : [따배쿠] 2-2. 도커 쿠버네티스 설치 / PC에 직접 설치하기>

 

[ Hands - on ]

• computer resources
  • Azure VM : D2s_v3
  • CPU : 2core
  • Memory : 8GB
  • Disk : 16GB
  • OS : Ubuntu 20.04 LTS - Gen2

 

Linux 환경구성

• set timezone

rm /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime

• set vi (ctrl + d)

cat > .vimrc << EOF
set paste
EOF

• change hostname (such as master, node1, node2)

hostnamectl set-hostname master.example.com
hostnamectl set-hostname node1.example.com
hostnamectl set-hostname node2.example.com

vi /etc/hosts

10.100.0.104  master.example.com  master
10.100.0.101  node1.example.com  node1
10.100.0.102  node2.example.com  node2

 

설치 전 환경설정

• Installing kubeadm 참고
• Swap disabled.
  • control-plane, worker node1, worker node2 모두 진행

swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab

• Letting iptables see bridged traffic

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

• 방화벽 해제

ufw disable

 

Containerd 설치하기

• Install Docker Engine on Ubuntu 참고
• Update the apt package index and install packages to allow apt to use a repository over HTTPS:

sudo apt-get update

sudo apt-get install \\
    ca-certificates \\
    curl \\
    gnupg \\
    lsb-release

• Add Docker’s official GPG key:

sudo mkdir -p /etc/apt/keyrings
curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

• Use the following command to set up the repository:

echo \\
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] <https://download.docker.com/linux/ubuntu> \\
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

• Install containerd

sudo apt-get update
sudo apt-get install -y containerd.io

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd

• systemd cgroup 드라이버 사용

vi /etc/containerd/config.toml

...
 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
...

sudo systemctl restart containerd

 

Kubelet, kubeadm, kubectl 설치하기

• apt패키지 인덱스를 업데이트하고 Kubernetes apt저장소를 사용하는 데 필요한 패키지를 설치

apt-get update
apt-get install -y apt-transport-https ca-certificates curl

• Google Cloud 공개 서명 키를 다운로드

curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg <https://packages.cloud.google.com/apt/doc/apt-key.gpg>

• Kubernetes apt리포지토리를 추가

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] <https://apt.kubernetes.io/> kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list

• apt패키지 인덱스를 업데이트 하고 kubelet, kubeadm 및 kubectl을 설치하고 해당 버전을 고정

apt-get update
apt-get install -y kubelet kubeadm kubectl

apt-mark hold kubelet kubeadm kubectl

 

Control-plane 구성

• Creating a cluster with kubeadm 참고
• control-plane 초기화
  • master node의 API, controller, scheduler, etcd, Core DNS가 생성

kubeadm init

⚠️ 에러 발생 시

root@master:~# kubeadm init

[init] Using Kubernetes version: v1.24.1
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: E0531 10:59:11.626033    2264 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-05-31T10:59:11+09:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

rm -f /etc/containerd/config.toml
systemctl restart containerd
kubeadm init

• worker node join을 위한 token을 token.txt에 별도로 저장

vi token.txt

kubeadm join 10.100.0.104:6443 --token i6p1uj.mklmx7mouiatrx2a \\
    --discovery-token-ca-cert-hash sha256:88cf20000ed8543e2c5813cec05bd5d66982e13f9898e3eb5d354170e19ec63e

• user가 kubectl command를 사용할 수 있도록 설정 (user별로 진행)

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

• pod network add-on (CNI) 설치 - Weave

kubectl apply -f "<https://cloud.weave.works/k8s/net?k8s-version=$>(kubectl version | base64 | tr -d '\\n')"

• control-plane bash-completion 활성화
  • 리눅스에서 bash 자동 완성 사용하기 참고
  • 각 user별로 진행

source <(kubectl completion bash)
source <(kubeadm completion bash)

echo "source <(kubectl completion bash)" >> ~/.bashrc
echo "source <(kubeadm completion bash)" >> ~/.bashrc

 

Worker node 구성

• Creating a cluster with kubeadm 참고
• token.txt에 있는 token을 worker node1,2에 입력

kubeadm join 10.100.0.104:6443 --token i6p1uj.mklmx7mouiatrx2a \\
    --discovery-token-ca-cert-hash sha256:88cf20000ed8543e2c5813cec05bd5d66982e13f9898e3eb5d354170e19ec63e

⚠️ 에러 발생 시

root@node1:~# kubeadm join 10.100.0.104:6443 --token i6p1uj.mklmx7mouiatrx2a \\
>     --discovery-token-ca-cert-hash sha256:88cf20000ed8543e2c5813cec05bd5d66982e13f9898e3eb5d354170e19ec63e

[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: E0531 11:36:06.238046    5703 remote_runtime.go:925] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2022-05-31T11:36:06+09:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

rm -f /etc/containerd/config.toml
systemctl restart containerd

• control-plane에서 worker node1,2 연결 확인

kubectl get nodes -o wide

NAME                 STATUS   ROLES           AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master.example.com   Ready    control-plane   38m    v1.24.1   10.100.0.104   <none>        Ubuntu 20.04.4 LTS   5.13.0-1025-azure   containerd://1.6.4
node1.example.com    Ready    <none>          102s   v1.24.1   10.100.0.101   <none>        Ubuntu 20.04.4 LTS   5.13.0-1023-azure   containerd://1.6.4
node2.example.com    Ready    <none>          58s    v1.24.1   10.100.0.102   <none>        Ubuntu 20.04.4 LTS   5.13.0-1023-azure   containerd://1.6.4

• node1, node2에 kubelet명령어 사용 가능하게 설정하기
  • 인증서 이동 (node1, node2)
  • 계정마다 진행

vi /etc/ssh/sshd_config

..
PermitRootLogin yes
...

mkdir -p $HOME/.kube
scp root@master:/etc/kubernetes/admin.conf ~/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

• worker node bash-completion 활성화
  • 리눅스에서 bash 자동 완성 사용하기 참고
  • 각 user별로 진행

source <(kubectl completion bash)
source <(kubeadm completion bash)

echo "source <(kubectl completion bash)" >> ~/.bashrc
echo "source <(kubeadm completion bash)" >> ~/.bashrc

 

참고 :

[따배쿠] 2-2. 도커 쿠버네티스 설치 / PC에 직접 설치하기

쿠버네티스 docker 지원 중단에 따른 대안 🐳 🥊 🐙