본문 바로가기
INFRA/DevOps

[K8s] Multi-master - HA Kuberenetes cluster 운영

by BTC_프로틴 2023. 1. 30.

K8s cluster를 직접 구성하는 도구

  • kubeadm
    • K8s에서 공식 제공하는 cluster 생성/관리 도구
  • kubespray
    • K8s cluster를 배포하는 오픈 소스 프로젝트
    • 다양한 형식으로 k8s cluster 구성 가능
    • On-premise에서 상용 서비스 cluster 운영 시 유용

K8s cluster 구성

  • Control plane(master node)
    • worker node 들의 상태를 관리하고 제어
    • High Available(HA) cluster 운영
    • API는 loadbalancer를 통해 worker node에 노출
    • 최소 3개의 중첩된 control plane을 구성(5,7개의 master nodes)
  • Worker node
    • container를 동작하며 실제 서비스를 제공

<이미지 출처 : [따배쿠] Appendix1. 멀티마스터 쿠버네티스 클러스터(multi-master - HA Kuberenetes cluster) 운영 >

 

Highly Available cluster 구성 순서

0. 사전준비

  • Compute resource
    • Azure VM : B2s
    • CPU : 2core
    • Memory : 4 GiB
    • OS : Ubuntu Server 20.04 LTS - GEN2
  • set timezone
rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
  • set vi (ctrl + d)
cat > .vimrc << EOF
set paste
EOF
  • change hostname (such as master1,2,3 , worker node1,2)
hostnamectl set-hostname master1.example.com
hostnamectl set-hostname master2.example.com
hostnamectl set-hostname master3.example.com
hostnamectl set-hostname worker1.example.com
hostnamectl set-hostname worker2.example.com

vi /etc/hosts

10.100.0.101  master1.example.com master1
10.100.0.102  master2.example.com master2
10.100.0.103  master3.example.com master3
10.100.0.104  worker1.example.com worker1
10.100.0.105  worker2.example.com worker2
10.100.0.106  lb.example.com lb
  • Swap disabled.
    • control-plane, worker node1, worker node2 모두 진행
swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab
  • Letting iptables see bridged traffic
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system
  • 방화벽 해제
ufw disable

1. all system - runtime install (containerd)

sudo apt-get update

sudo apt-get install \\
    ca-certificates \\
    curl \\
    gnupg \\
    lsb-release
  • Add Docker’s official GPG key:
sudo mkdir -p /etc/apt/keyrings
curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  • Use the following command to set up the repository:
echo \\
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] <https://download.docker.com/linux/ubuntu> \\
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  • Install containerd
sudo apt-get update
sudo apt-get install containerd.io

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd
  • systemd cgroup 드라이버 사용
vi /etc/containerd/config.toml

...
 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
...

sudo systemctl restart containerd

2. control plane, worker node - kubeadm install

  • apt패키지 인덱스를 업데이트하고 Kubernetes apt저장소를 사용하는 데 필요한 패키지를 설치
apt-get update
apt-get install -y apt-transport-https ca-certificates curl
  • Google Cloud 공개 서명 키를 다운로드
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg <https://packages.cloud.google.com/apt/doc/apt-key.gpg>
  • Kubernetes apt리포지토리를 추가
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] <https://apt.kubernetes.io/> kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list
  • apt패키지 인덱스를 업데이트 하고 kubelet, kubeadm 및 kubectl을 설치하고 해당 버전을 고정
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

systemctl start kubelet
systemctl enable kubelet

3. LB 구성

mkdir /etc/nginx

cat << END > /etc/nginx/nginx.conf
events { }
stream {
  upstream stream_backend {
    least_conn;
    server 10.100.0.101:6443;
    server 10.100.0.102:6443;
    server 10.100.0.103:6443;
  }

  server {
    listen                6443;
    proxy_pass            stream_backend;
    proxy_timeout         300s;
  }
}
END
  • Docker container로 nginx를 실행하면서 LB를 운영
docker run -d --name proxy -v /etc/nginx/nginx.conf:/etc/nginx/nginx.conf:ro --restart=always -p 6443:6443 nginx

****

4. kubeadm을 이용한 HA cluster 구성

kubeadm init --control-plane-endpoint "lb.example.com:6443" --upload-certs
  • 등록 시 계정마다 별도 진행
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
  • master2, master3을 master1에 join 및 확인
kubeadm join 10.100.0.106:6443 --token 4tj8em.7lmgpz3uiyjmqyle \\
        --discovery-token-ca-cert-hash sha256:96f4ede8a734c7eb21b99c866161216dc8223b3a781dbe55d9fbbdd1cb22100b \\
        --control-plane --certificate-key 7f3278b11f7f6aba464c7425be594b017bd6b27e033b7e03bf398f1ecfee015d
  • 등록 시 계정마다 별도 진행
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get nodes

NAME                  STATUS     ROLES           AGE   VERSION
master1.example.com   NotReady   control-plane   65m   v1.24.1
master2.example.com   NotReady   control-plane   97s   v1.24.1
master3.example.com   NotReady   control-plane   84s   v1.24.1
  • CNI(Container Network Interface) weave 설치 및 노드 확인
kubectl apply -f "<https://cloud.weave.works/k8s/net?k8s-version=$>(kubectl version | base64 | tr -d '\\n')"

kubectl get nodes

NAME                  STATUS   ROLES           AGE     VERSION
master1.example.com   Ready    control-plane   70m     v1.24.1
master2.example.com   Ready    control-plane   6m49s   v1.24.1
master3.example.com   Ready    control-plane   6m36s   v1.24.1
  • worker node1,2를 LB를 통해 master와 join
kubeadm join 10.100.0.106:6443 --token 4tj8em.7lmgpz3uiyjmqyle \\
        --discovery-token-ca-cert-hash sha256:96f4ede8a734c7eb21b99c866161216dc8223b3a781dbe55d9fbbdd1cb22100b

kubectl get nodes

NAME                  STATUS   ROLES           AGE   VERSION
master1.example.com   Ready    control-plane   96m   v1.24.1
master2.example.com   Ready    control-plane   32m   v1.24.1
master3.example.com   Ready    control-plane   32m   v1.24.1
worker1.example.com   Ready    <none>          24m   v1.24.1
worker2.example.com   Ready    <none>          24m   v1.24.1
  • kubectl command 자동완성 기능 추가 (계정마다 별도)
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

source <(kubeadm completion bash)
echo "source <(kubeadm completion bash)" >> ~/.bashrc

5. 설치 확인 및 Test

  • Pod 생성 및 확인
kubectl run webui --image=nginx

kubectl get pods -o wide

NAME                  STATUS   ROLES           AGE   VERSION
master1.example.com   Ready    control-plane   96m   v1.24.1
master2.example.com   Ready    control-plane   32m   v1.24.1
master3.example.com   Ready    control-plane   32m   v1.24.1
worker1.example.com   Ready    <none>          24m   v1.24.1
worker2.example.com   Ready    <none>          24m   v1.24.1

curl 10.42.0.1

```
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
```

 

 

참고 :

[따배쿠] Appendix1. 멀티마스터 쿠버네티스 클러스터(multi-master - HA Kuberenetes cluster) 운영

저작자표시 비영리

'INFRA > DevOps' 카테고리의 다른 글

[K8s] Taint & Toleration,Cordon& Drain  (0) 2023.02.13
[k8s] Pod Scheduling  (1) 2023.02.06
[k8s] Secret  (1) 2023.01.25
[K8s] ConfigMap  (0) 2023.01.16
[k8s] Canary Deployment  (0) 2023.01.09

관련글

댓글

티스토리툴바