Guidance on Kubernetes Threat Modeling

By Brandon Niemczyk (Cloud Security Research Lead)

Kubernetes is one of the most used container orchestration systems in cloud environments. As such, like any widely used application, it is an attractive target for cybercriminals and other threat actors. There are three general areas that cloud administrators need to secure their deployments against as these can introduce threats or risks to their Kubernetes-driven containerization strategies:

1. External attacks: attacks coming from outside the organization (any unauthenticated attack would fall into this bucket)
2. Misconfiguration issues: issues arising from unsecure configurations
3. Vulnerable applications:issues arising from vulnerabilities in software or applications

External attacks

Figure 1 shows the components of a Kubernetes deployment or cluster, as laid out by Kubernetes in its official documentation.

Figure 1. Kubernetes component diagram

All communication is marshaled through kube-api-server, which is a component of the control plane that exposes the application programming interface (API). It is essentially a massive representational state transfer (REST) API that works as the front end for the control plane and allows the user to define and control all Kubernetes management functions. The easiest way to use the API is to use the command line interface command kubectl (or oc for OpenShift).

Attackers who succeed in accessing the API can carry out virtually any goal of their choosing. They can, for instance, install malicious containers to extract information from databases or to use up resources for their cryptocurrency-mining campaigns.

Making sure this API is reachable only by machines in a cluster and machines that need to administer the cluster will help a lot in keeping external attackers at bay.

To do that, cloud administrators need to keep in mind that, by default, kube-api-server listens on two ports:

• Port 8080 on localhost only

○ Any request to the API through this port bypasses the authentication and authorization modules.

○ The port can listen on other hosts when the --insecure-bind-address flag is used.

○ The default port can be changed with the --insecure-port flag.

• Port 6443 on a public IP address

○ This is the first non-localhost network interface by default, also called a “secure port.”

○ Requests are properly sent through authentication and authorization.

○ The default port can be changed with the --secure-port flag.

Focusing on tightening the security of ports 8080 and 6443, and being mindful of what not to do with them will help deflect intrusions.

Suggestions for protecting clusters against external attacks

• Use a firewall rule to ensure that only machines that need access to the API have access to it.
• DO NOT use the --insecure-bind-address option to open up the plain-text port on non-localhost.
• Use intrusion prevention systems (IPSs) with Secure Sockets Layer (SSL) decryption capabilities, such as the Trend Micro™ TippingPoint™ Threat Protection System solution, to allow monitoring of traffic to and from the API for zero-days and known vulnerabilities.

Misconfiguration issues

Kubernetes is a complex system that offers a great deal of flexibility through its many configuration options. But these configuration options should be understood properly and set in a secure way, lest the cluster be compromised by threat actors.

Even authentication configuration alone can already be complex in Kubernetes because there are multiple modes of authentication available (role-based, attribute-based, or node-based). To help manage this, cloud administrators can use the command kubectl auth can-i to query specific permissions. Kubernetes users and service accounts should be locked down to only the permissions they need.

The reason we emphasize this aspect of configuring Kubernetes is because we have observed an unfortunate number of instances of misconfiguration continuing to occur in the wild. For example, a quick Shodan scan, as shown in Figure 2, can indicate that there are almost 3,000 hosts (spread out globally) where etcd, a key value server used by Kubernetes and thus a service that should not be exposed, is publicly accessible.

Figure 2. A Shodan scan of exposed etcd services (as of March 23, 2020)

Another thing cloud administrators need to be particularly aware of is that if there is no network policy specified for a specific namespace, the default policy is to allow all traffic to and from all pods in that namespace. Every pod can talk to every other pod, so if an attacker can get into a public-facing one (with a web app, for instance), then the attacker can use it to connect to other pods. This makes lateral movement much easier in case of a breach.

The best practice is to deny access by default and allow traffic only explicitly. At the minimum, a default policy to deny ingress traffic should be in place. Cloud administrators can create this policy by implementing the specific code provided in Kubernetes’s official documentation.

Suggestions for protecting clusters against misconfiguration issues

• Perform a thorough audit of all Kubernetes users and service accounts to ensure that they have access only to the necessary operations. A regular review should be conducted thereafter to make sure this remains enforced.
• Consider using a distribution of Kubernetes that, while it may limit the choices in some way, provides secure out-of-the-box configuration, such as RedHat’s OpenShift.
• Check the guide with specific instructions from the cloud provider. (Depending on where a cluster is deployed, most cloud providers will have this guide.) Check the CIS Kubernetes Benchmark from the Center for Internet Security as well.
• Ensure all services used by Kubernetes are properly firewalled and not exposed publicly.
• Pay attention to the network policies and deny ingress traffic by default.

Vulnerable applications

As with regular servers and operating systems, no matter how much effort is put into securing a Kubernetes cluster, it is only as secure as the weakest service running and exposed. Containers and container orchestrators facilitate not only the deployment of a wide variety of applications in a heterogeneous environment, but also the combined use of apps in a significant number of permutations. But they do not solve everything. In fact, the nature of containers can make ensuring that updates are applied everywhere they need to be even more difficult because every app has its own copy of every library.

As illustrated in Figure 3, the same library may be installed in multiple images, from different base images. And these all need to be updated and have security patches applied. Integrating a solution like Trend Micro Deep Security™ Smart Check into the DevOps process can help in this regard.

Figure 3. Libraries get deployed across multiple images, many of which have different bases and may need to be updated by multiple parties.

Suggestions for protecting clusters against vulnerable apps

• At the very least, ensure that all container images used are up to date and are being downloaded from trusted sources.
• Use container-specific automated scanning technologies like those in Trend Micro™ Deep Security™ Smart Check – Container Image Security to scan images for apps as part of the continuous integration and continuous delivery (CI/CD) process.

Securing clusters with cloud-specific protection

Containers provide a step forward in both DevOps and security (DevSecOps) through better process isolation. Container orchestrators such as Kubernetes provide an important level of abstraction above them to make them more useful in a scalable environment. That said, deploying them can be complex and taking the steps above will help ensure that the cloud environment is secure. Cloud-specific security solutions such as Trend Micro™ Hybrid Cloud Security and Trend Micro Cloud One™ can streamline protection in these instances.

Hybrid Cloud Security provides threat defense for safeguarding runtime physical, virtual, and cloud workloads, and containers. It also scans container images during development phases.

Cloud One, a security services platform for cloud builders, provides automated protection for CI/CD pipeline and applications. It also helps identify and resolve security issues sooner, and improve delivery time for DevOps teams. It includes:

• Workload Security: runtime protection for workloads
• Container Security: automated container image and registry scanning
• File Storage Security: security for cloud file and object storage services
• Network Security: cloud network layer IPS security
• Application Security: security for serverless functions, APIs, and applications