Managing Runtime Security Threats in Kubernetes

Most admins understand what runtime security is and why it’s important. And yet, they often overlook or underinvest in runtime security as part of their Kubernetes security strategy.

It’s easy to understand why: Kubernetes is such a complex system, and it offers so many different types of native tools to help access privileges and isolate workloads, that teams can become so engrossed in securing the cluster itself that they forget about Kubernetes runtime security.

To help close that gap, this article explains why runtime security in Kubernetes is so important and which tools are available for helping to secure Kubernetes against runtime threats.

What Is Runtime Security in Kubernetes?

Kubernetes runtime security is the protection of containers (or pods) against active threats once the containers are running.

Runtime security helps protect workloads against a variety of threats that could emerge after containers have been deployed, such as:

• The activation of malware that is hidden inside a container image.
• Privilege escalation attacks in which a container exploits security bugs in the container runtime, Kubernetes, or the host OS to access resources it should not be able to access (like storage volumes or external binaries).
• The deployment of unauthorized containers by an attacker who exploits a gap in an access control policy or a bug in Kubernetes.
• Unauthorized access to secrets or other sensitive information that a running container should not be able to read, but which it manages to access due to improper access control configurations or a security vulnerability that enables privilege escalation.

In a perfect world, none of these runtime threats would materialize because you would secure your application pipelines and clusters in such a way that threats never made their way into a live environment. You would also use rigid access controls to ensure that each workload was properly isolated, effectively preventing a compromised container from causing any harm to the cluster.

In the real world, of course, it’s impossible to guarantee that attackers won’t sneak malware into a container image by compromising your source code repository or build tools, for example, or that your Kubernetes RBAC policies and security contexts guarantee perfect isolation between workloads.

That’s why runtime security is so important. It’s the final layer of defense against threats that may creep into a Kubernetes environment. While you can, and should, take all reasonable measures to mitigate security risks within the pipeline and cluster architecture, you also need runtime security to help alert you to and control threats that sneak past your other defenses.

Kubernetes Runtime Security Tools

Kubernetes itself offers relatively little in the way of runtime security tooling. The only real resource it provides is auditing, which allows you to generate logs that track resource requests to the Kuberntes API.

However, while Kubernetes can record this information, Kubernetes doesn’t do anything on its own to analyze audit logs or alert you to activity that may be suspicious.

So, instead of turning to Kubernetes itself to secure your environment against runtime threats, you’ll need to lean on external runtime security tools. For Kubernetes, these tools fall into two main categories: enforcement tools and auditing tools.

Runtime Security Enforcement Tools

Enforcement tools allow you to define policies that restrict the access rights and permissions of resources within a runtime environment. While the tools can’t prevent a threat from materializing, they can minimize its impact by ensuring that, for example, malware that appears within one container can’t access resources external to the container.

Key enforcement tools for Kubernetes include:

• Seccomp: Seccomp is a Linux kernel-level tool that you can use to force processes to run in a secure state. When you use seccomp to place a process in a secure state, the kernel will prevent the process from making any system calls other than exit, sigreturn, and reading and writing to files that are already open.
• SELinux: SELinux is a kernel module that lets you define a broad array of access controls that the kernel enforces. You can use SELinux to set granular rules over which resources a container can access and which types of actions it can perform.
• AppArmor: AppArmor is also a kernel module that enables the definition of a broad set of access control rules. It’s very similar in most ways to SELinux; the debate about whether to use SELinux or AppArmor is like the one about vi vs. emacs: some folks just prefer one tool or the other, but you can achieve the same general results with either solution.

To an extent, you can think of Kubernetes RBAC policies and security contexts as also providing a certain level of enforcement control because they can be used to do things like prevent a container from running in privileged mode or disallowing access to kernel-level resources.

However, the difference between RBAC policies and security contexts on the one hand, and tools like SELinux and AppArmor on the other, is that the latter enforce access controls at the kernel level. That means that, even if Kubernetes itself is compromised in some way or has a security vulnerability, kernel-level enforcement tools can mitigate the impact of a security breach by preventing compromised containers from accessing external resources.

SELinux, AppArmor, and seccomp are also handy because they can be used to control access for any type of Linux workload. They are not specific to Kubernetes. If you are managing an environment that includes a mix of containers and VMs, then, you may find kernel-level access control frameworks useful because you can use the same tooling to manage all of your workloads.

Auditing Tools for Runtime Security

Auditing tools in Kubernetes help you detect and react to threats based on data collected from your cluster.

Again, Kubernetes provides the tooling for generating audit logs, but it doesn’t actually analyze the logs for you. For that, you’ll want to use a tool like Falco, an open source threat detection engine. Falco lets you define rules that will trigger alerts if the Falco engine detects the presence of certain conditions based on data like Kubernetes audit logs.

For instance, consider this Falco rule:

- rule: Example rule (nginx). This is the human name for the rule.
desc: Detect any listening socket outside our expected one.
condition: evt.type in (accept,listen) and (container.

image!=myregistry/nginx or proc.name!=nginx or k8s.ns.name!=”load-
balancer”)

output: This is where I write the alert message and I provide some
extra information (command=%proc.cmdline connection=%fd.name).
priority: WARNING

In this rule, any listening sockets that do not meet the following criteria will trigger an alert:

• The image is myregistry/nginx.
• The listening process is nginx.
• The namespace is load-balancer.

With a rule like this, you can detect containers running in your environment that may be trying to collect data or access unauthorized resources. You can also use a rule like this to detect unauthorized containers.

Falco has become the de facto auditing tool for Kubernetes and similar platforms for several reasons. In addition to being open source, Falco offers the ability to work with any type of cloud-native environment, not just Kubernetes.

At the same time, the Cloud Native Security Hub repository, which offers dozens of preconfigured Falco rules tailored to different types of workloads, makes it easy to set up Falco quickly without having to write a slew of auditing rules by hand.

Best Practices for Kubernetes Runtime Security

While deploying enforcement and auditing tools are among the steps required to address Kubernetes runtime security threats, you’ll want to go further than just setting up these tools. You should also ensure that you:

• Perform continuous image scanning: Although image scanning isn’t a security runtime operation per se (it falls within the scope of pipeline security), image scanning is important for preventing malware from reaching a runtime environment.
• Scan Kubernetes policy files: You can also help prevent runtime security issues by scanning Kubernetes RBAC files, deployments, and other resources to detect misconfigurations that could enable or exacerbate a runtime breach.
• Scan external policy files: You should also scan policy files or profiles that you create for SELinux, AppArmor, or other frameworks. Oversights within these files could weaken your runtime defenses.
• Don’t forget dev/test: When applying runtime security defenses, you’re likely to focus first and foremost on your production environment. After all, that’s where threats tend to have the greatest impact. But don’t forget to secure dev/test as well. Catching runtime threats in dev/test is a great way to prevent them from reaching production. Plus, even in a dev/test environment, a runtime security issue could have major consequences if, for example, an attacker is able to access secret data from within dev/test resources.
• Make a plan for incident remediation: Don’t wait until a breach occurs to start planning your response. Develop playbooks that will guide your team in handling different types of runtime security events. What will you do in the case of a privilege escalation attack, for example? What if one node is compromised but other Kubernetes nodes are not? Thinking through these issues beforehand will help you respond more quickly to live threats.

Runtime security is the only thing standing between your cluster and threats that have managed to elude the other defenses you erected within your application development pipeline and your Kubernetes architecture. But by continuously auditing your runtime environments, as well as using enforcement tools to segment runtime resources from each other, you can minimize the potential impact of runtime threats.