Minding the Gaps: The State of Vulnerabilities in Cloud Native Applications

by Magno Logan

What does it mean to be cloud native? According to The Cloud Native Computing Foundation (CNCF), cloud native technologies help organizations grow and run solutions in cloud environments and on-premises architectures. In the current business climate, enterprises are quickly adopting these technologies, which include containers and infrastructure as code, especially since the pandemic has accelerated digital transformation. Cloud native security is of particular concern because these projects are being used worldwide across many different organizations. Businesses and users of these applications should be aware that a vulnerability in one of these dependencies may affect the entire system and, in some cases, lead to a complete cluster compromise. 

Apart from vulnerabilities, there are other causes for concern. For example, most cloud native projects rely on libraries and dependencies that are also open-source software. These libraries are usually incorporated during the development lifecycle and rarely get updated or checked against known vulnerabilities. This is another avenue that may lead to compromised cloud assets. 

Our report maps the vulnerable cracks in cloud native application security, and shows how this is a growing concern that enterprises should devote time and resources to secure.  

Charting the scope and severity of vulnerabilities across cloud native applications

Inside the CNCF is the Security TAG team (STAG), which facilitates the creation of security resources and controls for users across the cloud native ecosystem. One of the team’s tasks is to request and coordinate independent security audits on cloud native projects. A list of the 2018-2020 security audits performed on these projects can be seen in the Appendix. We also placed these reports and assessments in this GitHub project, which we will continue to update.

The CNCF has been hiring independent security firms since 2018 to help analyze projects: Cure53, Trail of Bits, Atredis Partners and AdaLogics. We collected and combed through the reports from these third-party consulting firms and analyzed their findings. We also researched their security audits and compared the types and severity of vulnerabilities, among other things. 

The first data point that we analyzed is the number of vulnerabilities reported in each project. It comes as no surprise that Kubernetes came out on top. It is the largest of those projects, the most adopted, and has more code, thus there are more chances of finding vulnerabilities in it. The etcd and Helm projects came in second and third place, respectively.

Figure 1. The number of vulnerabilities per project, according to the security
audit reports published by CNCF

We also analyzed the vulnerability type or classification occurrences. The Trail of Bits consulting company classified vulnerabilities into many different classes or categories: Data Validation, Denial of Service, Authentication, and many other issues. Here is the breakdown of what each of these types means:

Table 1. Description of different classes of vulnerabilities, according to the vulnerability
classification by Trail of Bits

A few reports didn't specify a vulnerability type, so we normalized them according to the description of the vulnerability and what it affected. Here is the breakdown of vulnerabilities per vulnerability type from our analysis:

Figure 2. The number of vulnerabilities per class category, according to the Trail of Bits vulnerability classification

Data validation came out on top and is a cause for concern here since we have almost 60 vulnerabilities of this type. When we talk about data validation, it mainly has to do with input validation. It also includes stack-based buffer overflows, heap overflows, integer overflows, and other data validation issues. Using untrusted data without validating is a severe problem and can cause all sorts of vulnerabilities. 

For example, let's examine this vulnerability named "Improperly patched directory traversal in kubectl cp" detected by Atredis Partners on their security audit, which they published together with Trail of Bits in May 2019. Keep in mind that this was found on Kubernetes v1.13.5 and has already been fixed.

Figure 3. Kubernetes vulnerability detected by Atredis Partners among with Trail of Bits

According to the vulnerability description on the report, Kubernetes allows privileged users to copy data between containers using the kubectl cp command. Kubectl uses tape archive (tar) files to consume one container and transfer them to another to implement this. However, kubectl does not fully validate the structure of the tar file during processing, allowing an attacker to write arbitrary files in the destination container.

Figure 4. Incorrect detection logic within Kubernetes v1.13.5 and v1.14.0 

The fix for this issue intends to test if the path referenced by the link is absolute or if the file contains path shortcuts. Instead, however, the logic checks if the path linked is absolute. As a result, there is neither an error nor a relative path, meaning that valid relative paths that do not cause an error will succeed (see thefull reportfor more details). The short-term recommendation was to validate the contents of the tar files correctly.

In second place, after the data validation vulnerability type, is configuration. Again, these are usually related to issues that are preventable if the proper security settings are implemented. As discussed earlier this year, misconfigurations are the primary cause of cloud security issues, so no wonder there would be similar issues for cloud native applications. 

One of the examples here is a critical Configuration vulnerability in CoreDNS. This is one of the few critical severity vulnerabilities listed in this report. Cure53 reported this on their security audit published in March 2018. The vulnerability is DNS Cache poisoning via malicious Response, and it affected the local DNS instance, the python_dns.py file. For more details, please see the full report here.

And in third place is denial-of-service (DoS), with precisely 20 vulnerabilities that could lead to a DoS of that project and sometimes the entire system. DoS vulnerabilities can also result from a lack of proper data validation, which would allow an attacker to send data larger than expected or send too many requests in a brief period.

There was a high severity denial-of-service vulnerability reported by Trail of Bits on etcd in the security audit report published in February 2020. The issue is titled "The gateway can include itself as an endpoint, resulting in resource exhaustion" and affects the gateway.go file on etcd version 3.4.3.

Figure 5. High Denial of Service vulnerability detected on etcd

In the assessment, Trail of Bits stated that "The etcd gateway is a simple TCP proxy to allow for fundamental service discovery and access. However, it is possible to include the gateway address as an endpoint. This results in a denial of service since the endpoint can become stuck in a loop of requesting itself until there are no more available file descriptors to accept connections on the gateway."

Figure 6. Evidence of the DoS vulnerability on etcd from the audit report

But what about severity? How severe were the vulnerabilities found in those projects? Are these projects reliable? Can they be trusted? For the most part, yes, these are very robust open source projects. Thanks to the support of the CNCF, they are well maintained and security is always a priority. From the collection of vulnerabilities reported, the highest number of vulnerabilities are of low severity, followed by medium. That means that only a few of these projects had high or critical vulnerabilities that would put the user in danger, which is excellent. Here's a breakdown of the number of vulnerabilities per severity:

Figure 7. The number of vulnerabilities per severity, according to the security audit reports published by CNCF

Vulnerabilities classified as low usually have a CVSS 3.0 score between 0.1 and 3.9 while the medium ones are between 4.0 and 6.9 — this generally means that they are harder to exploit or have a lesser impact on the system than high or critical ones. It shows that even with the third-party consulting firms' outstanding talent for performing security audits and finding vulnerabilities, very few projects had significant issues. Only 17 vulnerabilities were classified as High and seven as Critical across all the projects tested.

Looking at Open Source Security and the rising rate of vulnerabilities

In this section, we analyze the dependencies of these projects and look for outdated or vulnerable libraries and license risks that they might be using in their code. For this analysis, we used Trend Micro Cloud One - Open Source Security to scan and get the results of the software composition analysis. Here is the result:

Figure 8. The number of vulnerabilities in third-party dependencies of Cloud Native projects divided by severity detected by Cloud One Open Source Security powered by Snyk.

With great power comes great responsibility… and more vulnerabilities. Being an extensive project with over 4.6 million lines of code at the time of writing, again, Kubernetes takes the lead as the project with most third-party vulnerabilities. We saw over 500 medium and high vulnerabilities and licensing issues in its dependencies. Harbor, in second  place, has only 157 total vulnerabilities; and Vitess, in the third spot, has 144. The Notary project is the only project with a critical vulnerability in its dependencies, however, it is one of the projects with the lowest number of vulnerabilities in total. As we can see from the example below, the latest version of Notary has a critical vulnerability in one of its libraries called gorm. This is an SQL Injection vulnerability with a CVSS score of 9.8 (out of 10). Luckily, according to the results, there is no public exploit available for this vulnerability yet, but this shouldn't be a reason to ignore fixing this issue. 

Figure 9. Critical vulnerability (CVSS 9.8) found on a library from the Notary project

We can also see an increase in the number of issues over time in Figure 9. This is because new vulnerabilities are found every day. And as the project evolves, new vulnerabilities are created with the new code being added at every commit. The industry average is that for every 1000 lines of code, there are between 5 and 50 potential issues. So that is another reason why we should prioritize these third-party risks. 

Figure 10. The number of issues detected over time on the CNCF projects.

Security recommendations and process improvements 

Hopefully, this report provides awareness of the security vulnerabilities in these cloud native projects and at the same time showcases how robust they are as long as they are used properly and kept updated. We also want to highlight the need for more streamlined security audit processes by the CNCF and encourage more security professionals and researchers to get involved.

For end-users and security professionals

It may be cliché advice, but using the latest version of these projects is one of the best ways to stay secure. Most of these reported issues have been resolved since the projects must fix them to reach the graduated status at CNCF. But if you want to be sure, you can check the open issues related to security (label: security) for each of them. Here is an example for Kubernetes, but keep in mind that anyone can open issues in these projects, so you must check each of them:

Figure 11. List of issues in the Kubernetes GitHub repository marked with the label "area: security"

You can also keep an eye on the Security Audits. There are two ways to do this: first, keep an eye on a specific CNCF project and watch their repository publish an RFP or the Security Audit report. And the second method involves watching the TOC (Technical Oversight Committee) repository on GitHub for an update when they release a new audit:

Figure 12. A list of past security audits done on CNCF projects on the TOC GitHub repository

Enterprises should also consider using security solutions such as the Trend Micro Cloud One™ platform, which protects cloud-native systems by securing continuous-integration and continuous-delivery (CI/CD) pipelines and applications. The platform includes:

• Workload Security: runtime protection for workloads
• Container Security: automated container image and registry scanning
• File Storage Security: security for cloud files and object storage services
• Network Security: cloud network layer intrusion prevention system (IPS) security
• Application Security: security for serverless functions, APIs, and applications
• Conformity: real-time security for cloud infrastructure — secure, optimize, comply

For the Cloud Native Computing Foundation 

The Security TAG team has made efforts to improve and increase the frequency of security assessments of CNCF projects. However, such initiatives should be prioritized and the CNCF should establish and enforce more periodical assessments. The foundation should prioritize the most popular projects such as Kubernetes, etcd, Helm, and others. In addition, promoting the RFPs for security audits even further and having a more diverse pool of consulting companies would also be beneficial.

Another recommendation would be to establish a unique vulnerability disclosure program covering all CNCF graduated projects and, if possible, establish a bug bounty as a next step. In these programs, security professionals or bug bounty hunters look for vulnerabilities in many different sites and applications. Some of these CNCF projects, such as Kubernetes, have bug bounty programs that pay from $100 to $10,000 for found vulnerabilities depending on their severity. 

Most of the projects already have a SECURITY.md or some sort of avenue for reporting vulnerabilities. Unfortunately, that doesn't seem to be standardized across all projects. There is also little visibility on what gets disclosed and what gets fixed. Vulnerabilities that are confirmed should be tracked, addressed, and preferably have a CVE assigned to it if the issue is confirmed.

Last but not least, we should investigate and prioritize issues related to third-party components and secure them against supply chain attacks. Publishing the Supply Chain Security Whitepaper was a great team effort by the Security TAG team. Still, to drive change, we as a community should practice what we preach and lead by example by applying the recommendations and best practices provided by the whitepaper across the CNCF projects.

For more information on CNCF security audits, please see our Appendix.