Kubernetes

From ArchWiki
Jump to navigation Jump to search

Kubernetes (aka. k8s) is an open-source system for automating the deployment, scaling, and management of containerized applications.

A k8s cluster consists of its control-plane components and node components (each representing one or more host machines running a container runtime and kubelet.service.

Installation

When manually creating a Kubernetes cluster install etcdAUR and the package group kubernetes-control-plane (for a control-plane node) and kubernetes-node (for a worker node).

When creating a Kubernetes cluster with the help of kubeadm, install kubeadm and kubelet on each node.

Both control-plane and regular worker nodes require a container runtime for their kubelet instances which is used for hosting containers. Install either containerd, cri-o or docker to meet this dependency.

To control a kubernetes cluster, install kubectl on the control-plane hosts and any external host that is supposed to be able to interact with the cluster.

Configuration

All nodes in a cluster (control-plane and worker) require a running instance of kubelet.service.

Tip: Read the following subsections closely before starting kubelet.service or using kubeadm.
Note: Disable swap on the host, as kubelet.service will otherwise fail to start.
Warning: Due to a long-standing bug in the Kubernetes code using kubelet.service on a nested btrfs drive or subvolume will lead to errors. A suggested fix for this issue is to create an explicit mountpoint for e.g. either /var/lib/ or /var/lib/containers/ and /var/lib/kubelet/ (added to fstab).

All provided systemd services accept CLI overrides in environment files:

  • kubelet.service: /etc/kubernetes/kubelet.env
  • kube-apiserver.service: /etc/kubernetes/kube-apiserver.env
  • kube-controller-manager.service: /etc/kubernetes/kube-controller-manager.env
  • kube-proxy.service: /etc/kubernetes/kube-proxy.env
  • kube-scheduler.service: /etc/kubernetes/kube-scheduler.env

Tango-view-fullscreen.pngThis article or section needs expansion.Tango-view-fullscreen.png

Reason:
  • Example for setup without kubeadm, using kube-apiserver.service, kube-controller-manager.service, kube-proxy.service and kube-scheduler.service.
  • Example for setup with kubeadm using configuration files.
(Discuss in Talk:Kubernetes#)

Networking

The networking setup for the cluster has to be configured for the respective container runtime. This can be done using cni-plugins.

Pass the virtual network's CIDR to kubeadm init with e.g. --pod-network-cidr='10.85.0.0/16'.

Container runtime

The container runtime has to be configured and started, before kubelet.service can make use of it.

CRI-O

When using CRI-O as container runtime, it is required to provide kubeadm init or kubeadm join with its CRI endpoint: --cri-socket='unix:///run/crio/crio.sock'

Note: CRI-O by default uses systemd as its cgroup_manager (see /etc/crio/crio.conf). This is not compatible with kubelet's default (cgroupfs).

Change kubelet's default by appending --cgroup-driver='systemd' to the KUBELET_EXTRA_ARGS environment variable in /etc/kubernetes/kubelet.env upon first start (i.e. before using kubeadm init.

After the node has been configured, the CLI flag can be replaced by a configuration entry for kubelet:

/var/lib/kubelet/config.yaml
cgroupDriver: 'systemd'

Running

Before creating a new kubernetes cluster with kubeadm start and enable kubelet.service.

Note: kubelet.service will fail (but restart) until configuration for it is present.

Setup

Tango-view-fullscreen.pngThis article or section needs expansion.Tango-view-fullscreen.png

Reason:
  • Example for setup without kubeadm, using kube-apiserver.service, kube-controller-manager.service, kube-proxy.service and kube-scheduler.service.
  • Example for setup with kubeadm using configuration files.
(Discuss in Talk:Kubernetes#)

When creating a new kubernetes cluster with kubeadm a control-plane has to be created before further worker nodes can join it.

Control-plane

Tip:
  • If the cluster is supposed to be turned into a high availability cluster (a stacked etcd topology) later on kubeadm init needs to be provided with --control-plane-endpoint=<IP or domain> (it is not possible to do this retroactively!).
  • It is possible to use a config file for kubeadm init instead of a set of parameters.

Use kubeadm init to initialize a control-plane on a host machine:

# kubeadm init --node-name=<name_of_the_node> --pod-network-cidr=<CIDR> --cri-socket=<SOCKET>
Note: Refer to #Networking and #Container runtime for <CIDR> and <SOCKET> (respectively).

If run successfully, kubeadm init will have generated configurations for the kubelet and various control-plane components below /etc/kubernetes and /var/lib/kubelet/. Finally, it will output commands ready to be copied and pasted to setup kubectl and make a worker node join the cluster (based on a token, valid for 24 hours).

To use kubectl with the freshly created control-plane node, setup the configuration (either as root or as a normal user):

$ mkdir -p $HOME/.kube
# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# chown $(id -u):$(id -g) $HOME/.kube/config

To install a pod network such as calico, follow the upstream documentation.

Worker node

With the token information generated in #Control-plane it is possible to make a node machine join an existing cluster:

 # kubeadm join <control-plane-host>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash> --node-name=<name_of_the_node> --cri-socket=<SOCKET>
Note: Refer to #Container runtime for <SOCKET>.

Tips and tricks

Tear down a cluster

When it is necessary to start from scratch, use kubectl to tear down a cluster.

 kubectl drain <node name> --delete-local-data --force --ignore-daemonsets

Here <node name> is the name of the node that should be drained and reset. Use kubectl get node -A to list all nodes.

Then reset the node:

# kubeadm reset

Operating from Behind a Proxy

kubeadm reads the https_proxy, http_proxy, and no_proxy environment variables. Kubernetes internal networking should be included in the latest one, for example

export no_proxy="192.168.122.0/24,10.96.0.0/12,192.168.123.0/24"

where the second one is the default service network CIDR.

Troubleshooting

Failed to get container stats

If kubelet.service emits

 Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"

it is necessary to add configuration for the kubelet (see relevant upstream ticket).

/var/lib/kubelet/config.yaml
systemCgroups: '/systemd/system.slice'
kubeletCgroups: '/systemd/system.slice'

See also