策略引擎>> K3s-HA-kubernetes-cluster>> 返回
项目作者： densogiaichned

项目描述 ：
Installation instructions for a lightweight kubernetes cluster with high availability.
高级语言：
项目地址: git://github.com/densogiaichned/K3s-HA-kubernetes-cluster.git


K3s HA kubernetes cluster

Installation instructions for a lightweight kubernetes cluster with high availability.
Based on Adrian Goins instructions.
Using:

  • K3s, Rancher
  • Kube-VIP
    As a controlplane LoadBalancer.
  • MetalLb
    For load balancing Kubernetes services.
  • Traefik
    Ingress controller.
  • K3sup, Helm

Table of Contents

Prerequisites

Cluster

  • An odd number of server nodes, in this example three server nodes.
  • At least 1 worker / agent node, in this example four workers / agents.
  • Enable public-key authentication for ssh on each node.

  • Setup HostName for all nodes, for example

    1.   # network 10.0.0.0/24
    2.   # Server nodes
    3.   10.0.0.10   k3s-srv01.local
    4.   10.0.0.11   k3s-srv02.local
    5.   10.0.0.12   k3s-srv03.local
    6.   # Worker / Agents
    7.   10.0.0.20   k3s-wrk01.local
    8.   10.0.0.21   k3s-wrk02.local
    9.   10.0.0.22   k3s-wrk03.local
    10.   10.0.0.23   k3s-wrk04.local
    11.   # kube-vip
    12.   10.0.0.99   k3s-vip.local
    13.   # reserve for MetalLb
    14.   # 10.0.0.90-10.0.0.98
    15.   10.0.0.90   k3s-traefik.local   # if you want to use Traefik-dashboard

On your development machine

  • Install k3sup (dev machine): 

    1. curl -sLS https://get.k3sup.dev | sh
    2. sudo install k3sup /usr/local/bin/
    3. k3sup --help

  • Install Helm

    • From Apt (Debian/Ubuntu): 

      1. curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
      2. sudo apt-get install apt-transport-https --yes
      3. echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
      4. sudo apt-get update
      5. sudo apt-get install helm

    • Initialize a Helm Chart Repository, e.g. stable:

      1. helm repo add stable https://charts.helm.sh/stable

    • To fix warning Kubernetes configuration file is group-readable. This is insecure. Location: /home/<user>/.kube/config, run:

      1. chmod go-r ~/.kube/config

    For other installation options, see Helm install.

On each node in the cluster

  • Disable sudo password on the nodes with

    Note
    At the time of writing, K3sup only works if no sudo password is set.
    However, with ssh and public-key authentication enabled, this shouldn’t be much of a problem.
    You may enable password again, after k3s was installed.

    1. sudo visudo

    At the bottom of the file, add the following line: 

    1. $USER ALL=(ALL) NOPASSWD: ALL

    where $USER = username, e.g. yourusername ALL=(ALL) NOPASSWD: ALL

Setup K3s cluster and Kube-VIP (control plane load balancer)

  • Setup IP`s, Variables, … 

    1. export K3S_VERSION=v1.20.4+k3s1 
    2. export K3S_NAME=k3s-demo 
    3. export IP_TLS_SAN=k3s-vip.local   #10.0.0.99
    4. export USERNAME=<yourusername>
    5. export SSHKEY=~/.ssh/id_rsa
    6. export SSHPORT=22

  • Install via k3sup: 

    1. k3sup install \
    2.   --host k3s-srv01.local \
    3.   --user $USERNAME \
    4.   --ssh-key $SSHKEY \
    5.   --ssh-port $SSHPORT \
    6.   --k3s-version=$K3S_VERSION \
    7.   --local-path=config.$K3S_NAME.yaml \
    8.   --context $K3S_NAME \
    9.   --cluster \
    10.   --tls-san $IP_TLS_SAN \
    11.   --k3s-extra-args="--disable traefik --disable servicelb --node-taint  node-role.kubernetes.io/master=true:NoSchedule"

  • SSH into server, e.g. ssh yourname@k3s-srv01.local -p $SSHPORT

  • Switch user to root with su root

  • Get kube-vip manifest 

    1. curl -s https://kube-vip.io/manifests/rbac.yaml > /var/lib/rancher/k3s/server/manifests/kube-vip-rbac.yaml

  • Check / edit kube-vip-rbac.yaml 

    1.   - apiGroups: ["coordination.k8s.io"]
    2.     resources: ["leases"]
    3.     verbs: ["list", "get", "watch", "update", "create"]

  • Create alias 

    1. alias kube-vip="docker run --network host --rm plndr/kube-vip:0.3.3"

    Note
    We are using docker here, the documentation uses containerd. However, the alias command in the documentation didn`t work.
    See also kube-vip offical documentation.

  • Lookup interface name and export 

    1. ip a                      # --> e.g. ens18
    2. export INTERFACE=ens18
    3. export VIP=k3s-vip.local  # 10.0.0.99

  • Generate manifest 

    1. kube-vip manifest daemonset \
    2.     --arp \
    3.     --interface $INTERFACE \
    4.     --address $VIP \
    5.     --controlplane \
    6.     --leaderElection \
    7.     --taint \
    8.     --inCluster | tee /var/lib/rancher/k3s/server/manifests/kube-vip.yaml

  • Check / edit /var/lib/rancher/k3s/server/manifests/kube-vip.yaml

    1. tolerations:
    2. - effect: NoSchedule
    3.   key: node-role.kubernetes.io/master
    4.   operator: Exists

  • ping vip k3s-vip.local

    1. ping k3s-vip.local
    3. PING k3s-vip.local (10.0.0.99) 56(84) bytes of data.
    4. 64 bytes from k3s-vip.local (10.0.0.99): icmp_seq=1 ttl=64 time=0.024 ms
    5. 64 bytes from k3s-vip.local (10.0.0.99): icmp_seq=2 ttl=64 time=0.030 ms
    6. 64 bytes from k3s-vip.local (10.0.0.99): icmp_seq=3 ttl=64 time=0.059 ms

  • Exit ssh, switch back to your development machine.

  • Edit $HOME/config.$K3S_NAME.yaml, replace server-ip with vip-ip / vip-hostname:

    1. apiVersion: v1
    2. clusters:
    3. - cluster:
    4.     certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZ...
    5.     server: https://k3s-vip.local:6443  #replace ip with vip-hostname / vip-ip
    6.   name: k3s-demo
    7. #...

  • Add remaining server nodes 

    1. k3sup join \
    2.   --host k3s-srv02.local \
    3.   --user $USERNAME \
    4.   --ssh-key $SSHKEY \
    5.   --ssh-port $SSHPORT \
    6.   --server-host=$IP_TLS_SAN \
    7.   --k3s-version=$K3S_VERSION \
    8.   --server \
    9.   --k3s-extra-args="--disable traefik --disable servicelb --node-taint  node-role.kubernetes.io/master=true:NoSchedule"
    1. k3sup join \
    2.   --host k3s-srv03.local \
    3.   --user $USERNAME \
    4.   --ssh-key $SSHKEY \
    5.   --ssh-port $SSHPORT \
    6.   --server-host=$IP_TLS_SAN \
    7.   --k3s-version=$K3S_VERSION \
    8.   --server \
    9.   --k3s-extra-args="--disable traefik --disable servicelb --node-taint  node-role.kubernetes.io/master=true:NoSchedule"

  • Check cluster

    1. kubectl get nodes
    3. NAME        STATUS   ROLES                       AGE     VERSION
    4. k3s-srv01   Ready    control-plane,etcd,master   5m      v1.20.4+k3s1
    5. k3s-srv02   Ready    control-plane,etcd,master   30s     v1.20.4+k3s1
    6. k3s-srv03   Ready    control-plane,etcd,master   10s     v1.20.4+k3s1
    1. kubectl get pods -n kube-system
    3. NAME                                      READY   STATUS    RESTARTS   AGE
    4. coredns-854c77959c-t7rqd                  1/1     Running   0          107m
    5. helm-install-traefik-7spjg                0/1     Pending   0          107m
    6. kube-vip-ds-jzhvf                         1/1     Running   0          56s
    7. kube-vip-ds-l45gf                         1/1     Running   0          28s
    8. kube-vip-ds-srrt5                         1/1     Running   0          18m
    9. local-path-provisioner-5ff76fc89d-x9ppj   1/1     Running   0          107m
    10. metrics-server-86cbb8457f-gvzfx           1/1     Running   0          107m

  • Add worker node(s) 

    1. k3sup join \
    2.   --host=k3s-wrk01.local \
    3.   --user $USERNAME \
    4.   --ssh-key $SSHKEY \
    5.   --ssh-port $SSHPORT \
    6.   --server-host=$IP_TLS_SAN \
    7.   --server-user $USERNAME \
    8.   --k3s-version=$K3S_VERSION
    1. k3sup join \
    2.   --host=k3s-wrk02.local \
    3.   --user $USERNAME \
    4.   --ssh-key $SSHKEY \
    5.   --ssh-port $SSHPORT \
    6.   --server-host=$IP_TLS_SAN \
    7.   --server-user $USERNAME \
    8.   --k3s-version=$K3S_VERSION
    1. k3sup join \
    2.   --host=k3s-wrk03.local \
    3.   --user $USERNAME \
    4.   --ssh-key $SSHKEY \
    5.   --ssh-port $SSHPORT \
    6.   --server-host=$IP_TLS_SAN \
    7.   --server-user $USERNAME \
    8.   --k3s-version=$K3S_VERSION
    1. k3sup join --host=k3s-wrk04.local \
    2.   --user $USERNAME \
    3.   --ssh-key $SSHKEY \
    4.   --ssh-port $SSHPORT \
    5.   --server-host=$IP_TLS_SAN \
    6.   --server-user $USERNAME \
    7.   --k3s-version=$K3S_VERSION

MetalLB - K3s Service LoadBalancer Installation

Installation by manifest

  • Get default manifests and combine them in a single file metallb.yaml

    1. curl -s https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml > /var/tmp/metallb.yaml; \
    2. echo '---' >> /var/tmp/metallb.yaml; \
    3. curl -s https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml >> /var/tmp/metallb.yaml; \
    4. mv /var/tmp/metallb.yaml /var/lib/rancher/k3s/server/manifests

  • Edit manifest metallb.yaml

    1. nano /var/lib/rancher/k3s/server/manifests/metallb.yaml
    1.   kubernetes.io/os: linux
    2. serviceAccountName: speaker
    3. terminationGracePeriodSeconds: 2
    4. tolerations:
    5. - effect: NoSchedule
    6.   key: node-role.kubernetes.io/master
    7.   operator: Exists                      #<------ add

  • Create secret

    1. kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

  • create configmap config.yaml and assign a block of free addresses for MetalLb, here 10.0.0.90-10.0.0.98

    1. apiVersion: v1
    2. kind: ConfigMap
    3. metadata:
    4.   namespace: metallb-system
    5.   name: config
    6. data:
    7.   config: |
    8.     address-pools:
    9.     - name: default
    10.       protocol: layer2
    11.       addresses:
    12.       - 10.0.0.90-10.0.0.98

    and apply

    1. kubectl apply -f config.yaml

  • Verify MetalLb is deployed

    1. kubectl get pods -n metallb-system -o wide
    2. NAME                          READY   STATUS    RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
    3. controller-65db86ddc6-p8g74   1/1     Running   0          45s     10.42.6.10    k3s-wrk04   <none>           <none>
    4. speaker-fbr96                 1/1     Running   0          45s     10.0.0.12     k3s-srv03   <none>           <none>
    5. speaker-h5htt                 1/1     Running   0          45s     10.0.0.21     k3s-wrk02   <none>           <none>
    6. speaker-hqmpz                 1/1     Running   0          45s     10.0.0.23     k3s-wrk04   <none>           <none>
    7. speaker-ppbpn                 1/1     Running   0          45s     10.0.0.22     k3s-wrk03   <none>           <none>
    8. speaker-qmkjq                 1/1     Running   0          45s     10.0.0.11     k3s-srv02   <none>           <none>
    9. speaker-vdbqt                 1/1     Running   0          45s     10.0.0.10     k3s-srv01   <none>           <none>
    10. speaker-z2zbp                 1/1     Running   0          45s     10.0.0.20     k3s-wrk01   <none>           <none>

Install Traefik via Helm chart

  • Download example values.yaml file from GitHub
  1. mkdir ~/traefik
  2. wget https://raw.githubusercontent.com/traefik/traefik-helm-chart/master/traefik/values.yaml /traefik/values.yaml

  • Edit values.yaml file

    Note
    We want to modify some values, to run traefik as DaemonSet instead of Deployment.
    Running as Daemonset, a instance of traefik will be spawn automatically on each worker node (failover).

    1. nano /traefik/values.yaml

    • change kind: Deployment to kind: DaemonSet

      1. deployment:
      2. enabled: true
      3. # Can be either Deployment or DaemonSet
      4. #kind: Deployment
      5. kind: DaemonSet

    • Section providers set publishedService enabled = true 

      1. providers:
      2. #. . .
      3.   # IP used for Kubernetes Ingress endpoints
      4.   publishedService:
      5.     enabled: true

    • Check section service is enabled: true and type: Loadbalancer 

      1. # Options for the main traefik service, where the entrypoints traffic comes
      2. # from.
      3. service:
      4.   enabled: true
      5.   type: LoadBalancer

    • Check section rbac is enabled: true 

      1. # Whether Role Based Access Control objects like roles and rolebindings should be created
      2. rbac:
      3.   enabled: true

    Save modification and exit editor.

  • Create namespace traefik-system 

    1. kubectl create namespace traefik-system

  • Install traefik with helm

    1. helm repo add traefik https://helm.traefik.io/traefik
    2. helm repo update
    3. helm install traefik --values traefik/values.yaml --namespace traefik-system traefik/traefik

  • Check installation succeeded

    1. kubectl get all -n traefik-system -o wide
    3. NAME                READY   STATUS    RESTARTS   AGE     IP           NODE        NOMINATED NODE   READINESS GATES
    4. pod/traefik-hqc8c   1/1     Running   0          30s     10.42.5.9    k3s-wrk03   <none>           <none>
    5. pod/traefik-m7kn7   1/1     Running   1          30s     10.42.3.17   k3s-wrk01   <none>           <none>
    6. pod/traefik-pltvd   1/1     Running   0          30s     10.42.6.11   k3s-wrk04   <none>           <none>
    7. pod/traefik-wfj5g   1/1     Running   0          30s     10.42.4.11   k3s-wrk02   <none>           <none>
    9. NAME              TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE     SELECTOR
    10. service/traefik   LoadBalancer   10.43.82.54   10.0.0.90      80:31794/TCP,443:31030/TCP   30s     app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
    12. NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE     CONTAINERS   IMAGES          SELECTOR
    13. daemonset.apps/traefik   4         4         4       4            4           <none>          30s     traefik      traefik:2.4.7   app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik

    Note
    MetalLb will assign a external ip to the Traefik service, here 10.0.0.90 which is the first address of the block we reserved for MetalLb.

  • Optional Install Traefik Dashboard

    • Create traefik/dashboard.yaml file and paste/edit

      1. # dashboard.yaml
      2. apiVersion: traefik.containo.us/v1alpha1
      3. kind: IngressRoute
      4. metadata:
      5.   name: dashboard
      6. spec:
      7.   entryPoints:
      8.     - web
      9.   routes:
      10.    #- match: Host(`traefik.localhost`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))
      11.     - match: Host(`k3s-traefik.local`) || (PathPrefix(`/dashboard`)
      12.       kind: Rule
      13.       services:
      14.         - name: api@internal
      15.           kind: TraefikService

    Save modification and exit editor.

Install Rancher

Install Rancher on the HA cluster as described by Adrian Goins https://www.youtube.com/watch?v=9PLw1xalcYA&t=880s

Resources

K3s, Rancher

Adrian Goins on YouTube / GitLab

Helm

K3sup

Kube-VIP

MetalLB

Traefik