Kubernetes Cluster Security, Component by Component

If you know anything about Kubernetes, you know that it’s not a single tool but a multitude of distinct types of services and resources – such as an API server, node agents, container runtimes, and so on.

Each component within a Kubernetes cluster faces different types of security threats and requires different security configurations. In some cases, you can apply the same security technique to multiple components. For example, you can use Role-Based Access Control to manage permissions for various types of resources in Kubernetes.

Still, some components have unique security requirements and require unique security solutions. Securing the API server is different from securing Kubernetes nodes, for example. Kubernetes cluster security is a matter of understanding and addressing the possible security threats to each component.

With that reality in mind, here’s how to secure a Kubernetes cluster by securing each of its various components.

How to Secure a Kubernetes Cluster

Secure the Kubernetes API Server

The Kubernetes API server, kube-apiserver, is the heart of your Kubernetes cluster. The API server is what grants access requests, keeps workloads running, and so on.

The main tool for securing the API server in Kubernetes is the Kubernetes Role-Based Access Control, or RBAC, framework. With RBAC policies, you can define users and service accounts, then assign permissions that allow them to perform certain actions using the Kubernetes API.

For example, you can create an RBAC Role that allows a resource to get, watch, and list pods within the default Kubernetes namespace. The Role would look like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]Code language: PHP (php)

You would then associate the Role with a user or service group in Kubernetes by creating a RoleBinding or a ClusterRoleBinding with a command such as:

kubectl create rolebinding podreader-binding --user=john

This RoleBinding gives the user “john” the permissions to get, watch, and list pods that are defined in the Role above.

Admission Controllers and Kubernetes API Security

You can provide a second layer of defense for the Kubernetes API — and, by extension, your entire Kubernetes cluster — with admission controllers. In essence, an admission controller is code that intercepts requests to the Kubernetes API after the API has already authenticated and authorized a request, but before the API finishes fulfilling the request.

You can use admission controllers to limit the volume of resources that an object can request from the API, for instance, or prevent the execution of certain commands even if RBAC policies and security contexts allow them.

Using Audit Logs to Enhance Security

A third security resource to help prevent security problems related to the Kubernetes API is audit logging. Audit logging is an optional feature of Kubernetes that allows you to record and track requests as the API server processes them.

By creating audit logs and analyzing them continuously with a tool like Falco, you’ll receive automatic alerts when a user or service makes suspicious requests, such as repeatedly trying to access resources that the user or service is not authorized to access.

Secure and Backup Etcd

By default, Kubernetes uses etcd, a key-value store, to store all of the configuration data for your cluster.

Kubernetes doesn’t offer any native tooling for securing etcd, but you should be sure that you enable transport security. Refer to the etcd documentation for details.

As a cluster security best practice, you should also ensure that you back up etcd using the etcdctl snapshot save command. Even if you are running Kubernetes in a place where you have strong confidence in the reliability of etcd, etcd backups protect you against ransomware or other attacks where someone may try to disable your access to the critical configuration data stored in etcd.

Finally, a best practice when working with etcd is to avoid storing sensitive secrets data, like passwords and access keys, in etcd. Most Kubernetes distributions store secrets in etcd by default, but you should instead use an external secrets manager.

Protect the Master Node(s)

A Kubernetes master node is the node that hosts the Kubernetes API server, scheduler, controller, and etcd – in other words, all of the crucial services that bind together and manage a Kubernetes cluster. Some Kubernetes clusters may run multiple master nodes in order to increase cluster availability, in which case each master runs redundant instances of the core Kubernetes services.

Given the central role that master nodes play in the cluster, securing master nodes is vital to overall cluster security. There are several best practices to follow in order to protect master nodes:

• Provision master nodes with a minimalist Linux distribution, such as Alpine Linux. Avoid installing extra applications or libraries on the node unless they are strictly necessary.
• Use RBAC policies to restrict which actions can be performed on nodes. The less access to and information about your nodes that attackers have, the more secure the nodes are.
• Use enforcement frameworks like SELinux, AppArmor, and seccomp to restrict access controls at the kernel level on the master. These frameworks will help prevent the escalation of a security event by making it more difficult for a malicious process to access kernel resources or interfere with other processes.
• Continuously monitor node log files for signs of suspicious activity.

Secure Worker Nodes and Restrict Kubelet and Kube-Proxy Permissions

Worker nodes’ main job is to host running containers or pods. Each worker node communicates with the rest of the cluster via kubelet, an agent that runs locally on each worker node. Worker nodes also use a service called kube-proxy to manage network communications for pods that are hosted on the worker nodes.

By design, worker nodes in Kubernetes are replaceable. If a node fails, Kubernetes will automatically reassign the node’s pods to other nodes. In this sense, securing worker nodes is less critical than securing master nodes, because the failure of a few worker nodes due to security-related issues doesn’t pose a critical threat to your cluster as a whole.

Nonetheless, an attacker who manages to compromise a worker node can potentially access any pods hosted on those nodes. For that reason, it’s important to take steps to secure worker nodes as part of your overall cluster security process.

All of the best practices for securing master nodes, which we discussed above, also apply to worker nodes. In addition, consider restricting the permissions granted to kubelet and kube-proxy in order to reduce the potential damage that an attacker could cause if he or she manages to break out of a container and into the worker node that hosts the container.

Continuously Scan Container Images

Container images, which define the resources required to run containers, aren’t part of Kubernetes per se, but they are an essential part of most Kubernetes clusters, and shouldn’t be overlooked while securing the cluster.

The key to securing container images is to scan them continuously. In addition, you can minimize security issues associated with containers by:

• Only using images from trusted sources. Avoid the temptation to pull images from a random registry just because it’s convenient.
• Avoiding using the “latest” tag when pulling containers. Specify a version instead, which ensures that you know exactly which version of the container you are running and which security vulnerabilities may affect it.
• Taking a minimalist approach to designing container images. Instead of trying to pack multiple resources into a single image, create different images for each functionality that you need to implement.

Container Runtime Security

The container runtime is the service that actually executes containers. Kubernetes supports a variety of runtimes. The differences between runtimes tend not to be important for most use cases today because most runtimes are capable of running most containers.

A security flaw in your container runtime software may allow attackers to perform any number of exploits. For example, in 2019, developers discovered a major vulnerability in the Docker runtime that effectively allowed containers to gain root-level access to the entire host.

Kubernetes itself does nothing to monitor your container runtime for vulnerabilities or alert you to breaches that may result from those vulnerabilities. You therefore need to rely on external tools to secure runtimes, including:

• Enforcement frameworks, like SELinux and AppArmor, which you can use to restrict the actions that a malicious container can perform. When properly configured, these frameworks provide a second layer of defense in the event that an attacker exploits a container runtime security vulnerability to perform actions that shouldn’t be allowed.
• Auditing tools, like Falco, can continuously monitor audit logs and other data streams from your container runtime environment and alert you when they detect behavior that may be a sign of a breach. While auditing tools can detect a variety of security threats in addition to those associated with container runtimes, they are a primary means of identifying breaches that stem from a runtime-level vulnerability.
• Unless strictly necessary, use Kubernetes security contexts to prevent containers from running in privileged mode and to restrict the resources they can access. While security contexts don’t directly prevent container runtime vulnerabilities, they provide an additional layer of defense that can mitigate the impact of a runtime breach.

A Holistic Approach to Kubernetes Cluster Security

It would be nice if there were “one dumb trick” you could use to secure all components of Kubernetes all the time. But there’s not. Kubernetes security is as complex as the architecture of Kubernetes itself. While certain cluster-level practices, like auditing and monitoring, can provide broad protection against some types of threats, you also need to deploy security resources that can protect the various individual components that form a Kubernetes cluster, from the API server, to worker and master nodes, to etcd and beyond.