Abusing Argo CD, Helm, and Artifact Hub: An Analysis of Supply Chain Attacks in Cloud-Native Applications

by Magno Logan

Cloud-native applications have become increasingly popular with the adoption of containers, Kubernetes, and microservices. The consequent boost in capabilities from these apps has led organizations to leverage DevOps processes to hasten code writing and deployment of their applications to production. However, this can be problematic when developers work with different tools, services, and architectures. While various tools for continuous integration and continuous deployment (CI/CD) are available to tackle problems that arise, heightened security awareness becomes crucial especially when taking a proactive stance to prevent supply chain attacks.

This report gives an overview of Argo CD, Helm, and Artifact Hub, three of the cloud-native tools that are widely used today to solve issues related to deploying cloud-native apps. We discuss their goals and, from a malicious actor’s perspective, analyze how they can be abused to compromise organizations through supply chain attacks. We also elaborate on some best practices and security recommendations for enterprises to protect their environments.

An overview of Argo CD, Helm, and Artifact Hub

The following section briefly discusses some basic information on and goals of each cloud-native tool.

Argo CD

Argo CD is a declarative GitOps continuous delivery tool for Kubernetes that uses Git repositories as the source of truth. The Argo Project, used by nearly 200 organizations, comprises four open-source tools that include Argo Workflows, Argo CD, Argo Rollouts, and Argo Events. It was created by Intuit and is currently maintained by many organizations with the support of the Cloud Native Computing Foundation (CNCF) as an incubation project. It is written in Go and underwent an independent third-party security audit in 2021. The CNCF Security Technical Advisory Group (TAG) has been performing security assessments on Argo CD, which is being tracked here.

Helm

Helm is the package manager for Kubernetes applications. This tool allows one to group a set of YAML files, configurations, and dependencies together into a single Helm chart and then to use that to install applications directly on one’s cluster. Helm is a graduated project from the CNCF, and it has become more robust in its third version as a result of security reviews and architectural changes.

Artifact Hub

Artifact Hub, a more recent CNCF project that is still in sandbox status, works as a centralized repository for Kubernetes artifacts. The artifacts consist mainly of Helm charts, kubectl plug-ins, and Falco rules, among others. One can also think of Artifact Hub as a Docker Hub for Kubernetes. While most of the hosted projects still trace back to a GitHub repository, Artifact Hub’s search functionality can be navigated with more ease. A look at the Artifact Hub Stats page shows information on packages, releases, and repositories.

Figure 1. Artifact Hub stats on packages and releases as of April 18, 2022

Threat modeling scenarios

To analyze risks from a malicious actor’s point of view, we devised some threat modeling scenarios to investigate the weaknesses that a cybercriminal might exploit and to determine security settings that organizations need to enable. We provide relevant insights in the following sections.

Compromising Argo CD

Guarding against unauthorized access to the Argo CD portal should be a priority. Therefore, the portal should not be exposed to the internet. Despite the need for authentication to access the portal, there is a possibility for someone to compromise credentials and use them to deploy malicious applications to a Kubernetes (aka K8s) cluster, such as cryptocurrency miners. A quick search on Shodan revealed dozens of Argo CD servers that are exposed to the internet and can be accessed directly.

Figure 2. The number of Argo CD servers exposed to the internet, as searched through Shodan on April 18, 2022 using the query, “argocd”

It’s worth noting here that as organizations pursue and shift to cloud-native applications such as Argo CD, there is reason to believe that this number will grow exponentially in the coming years unless deliberate measures are taken to minimize this exposure. In this case, since the number of results is relatively low and since the results are from a generic string search, we decided to use other search tools to gain a better sense of the risks involved.

A search on Censys.io using a string search for “argocd” revealed almost 4,000 exposed servers. Upon analysis, we learned that these are primarily Kubernetes clusters running Argo CD.

Figure 3. The number of Argo CD servers exposed to the internet according to a search on Censys on April 18, 2022 using the query, “argocd”

Like any other DevOps tool that is part of the pipeline, Argo CD should not be publicly exposed and should have strong authentication and authorization mechanisms. In some of the instances that we checked, the Argo CD server was exposed and only required a username and a password for gaining access. This is cause for concern since all access events should be thoroughly verified and policies be consistently enforced, just as the web interface should not be used to deploy applications to a Kubernetes cluster.

Figure 4. Exposed Argo CD server found on Shodan, retrieved on April 18, 2022

From the Argo CD documentation, the initial username is admin, and the password is randomly generated to consist of uppercase and lowercase letters, numbers, and dashes with a total count of 16 characters. It uses the crypto/rand library from Go to generate secure random passwords. Figure 5 shows the function that generates this initial password.

Figure 5. Initial password generation code for admin account, retrieved here

The admin’s initial password is then stored as cleartext in a Kubernetes Secret object inside the cluster. Kubernetes Secrets are objects designed to hold sensitive information. Unfortunately, the current default setup is far from secure. Kubernetes Secrets are only base64-encoded and are stored in cleartext inside the etcd, making it necessary to manually enable the encryption at rest for Secrets before using them.

This means that anyone with access to the K8s cluster can access the Argo CD admin password if they have the proper role-based access control (RBAC) permissions. The documentation warns that the initial password should be deleted after it has been changed, but it’s hard to tell if this is being done properly by cluster admins without accessing the clusters. In “The Basics of Keeping Kubernetes Clusters Secure,” we provide relevant information on how to protect the etcd.

Figure 6. Argo CD official documentation about the initial admin password, retrieved here

Downloading a malicious Helm chart into Argo CD

By default, Argo CD only has the admin as its built-in user with complete access to the system. Malicious actors can therefore try to grab the Argo CD admin user credentials by brute force to access the system through the web user interface (UI), as seen, for example, in Figure 4 since there are no native protections against these types of attacks. A malicious actor’s next steps will depend on the authentication method set up for the Argo CD server. An enterprise can either have local user accounts or set up single sign-on (SSO) with many different providers such as Microsoft, Okta, and Google, among others. Access to the Argo CD admin user credentials also allows RBAC configuration to facilitate access control and restrict access to different Argo CD resources.

When a malicious actor has access to Argo CD, it also means that they can access the environment that it manages. Naturally, to create, modify, or delete any containerized applications on Kubernetes clusters, malicious actors also need to have the proper permissions. If malicious actors deploy a malicious Helm chart or GitHub repository that they control, they can then direct Argo CD to inject a harmful application to the cluster. In the next section, we examine the implications of this in a real-world example.

Let us assume that the malicious actor has already compromised the Argo CD instance with the intent to mine Monero cryptocurrency. Even if they know little about K8s and Helm charts, they can still exploit a premade application using Artifact Hub. A quick search for cryptocurrency on Artifact Hub only yields a few results, so we used XMRig instead to mine Monero in a Kubernetes cluster.

Figure 7. Monero cryptocurrency miner Helm chart from Artifact Hub, retrieved here

At this point, the malicious actor has set the stage for an attack and can now deploy malicious applications through the compromised Argo CD as a path to the Kubernetes clusters with this information. By logging in to Argo CD and creating a new application, cybercriminals can point the new application source to the GitHub repository where the Monero Helm chart is located. After syncing it, the miner pods will be automatically deployed to the target cluster.

Figure 8. Deployment of the Monero application through Argo CD

Figure 9 shows the objects being created for the Monero application after syncing Argo CD with the Kubernetes cluster.

Figure 9. Kubernetes objects being created for the Monero application

Figure 10. Creation of Kubernetes objects for the Monero application

This scenario demonstrates a potential attack based on the aforementioned vulnerabilities. Although there are many red flags that might alert cluster admins that something is awry — and these flags can be found anywhere from the deployment of an application through a web portal to the application namespace where it is located on the cluster — malicious actors can easily configure these to avoid detection.

Malicious actors can also conceal these red flags as legitimate applications, as we elaborate on next.

Taking over a GitHub repository used by Argo CD

Since Argo CD relies on a Git repository for the source of truth, we ventured further to see what would happen if the repository was compromised by someone taking over one of the maintainers’ account credentials. Interestingly, this is not unheard of: It has been reported that thousands of npm accounts use email addresses with expired domains. In addition, according to GitHub's Security Incident Response Team (SIRT), many developers’ GitHub credentials were targeted in phishing attacks in 2020.

Under these circumstances, malicious actors can take over a GitHub repository that is being used by an organizations’ Argo CD tool to deploy an application to a K8s cluster. The malicious actor then updates the repository with malicious code that will be deployed later to the cluster through Argo CD. That malicious code could be the cryptocurrency miner from the scenario discussed previously (an application used to steal Kubernetes Secrets and other credentials) or even a backdoor into the K8s cluster itself. We detail how this could pan out here.

Figure 11. Taking over a GitHub repository used by Argo CD

In this case, we created a new GitHub repository for this example with the assumption that this would be a legitimate and well-known repository for use in deploying applications to a K8s cluster through Argo CD. Here, it is relevant for us to ask: How well do project maintainers implement proper security measures to ensure the protection of their accounts and sufficiently protect the integrity of their code?

Figure 12. Fake Helm Chart created on GitHub

In Figure 13, we can see the Helm chart being used to deploy the application on Argo CD. We used the Helm charts for Argo CD to illustrate an identified flaw, although it could have also been any other Helm chart. In this case, we deployed Argo CD. But what if it wasn’t Argo CD and instead was a malicious application to steal Secrets from the cluster? This then begs the question of how consistently developers verify the source and provenance of Helm charts. One thing is for certain, however. When developers don’t verify these, Argo CD could deploy an application to the cluster without any issues as long as the Helm charts are properly created.

Figure 13. Adding a fake Helm chart to Argo CD

Another possible attack scenario involves someone hijacking a GitHub account to make changes to a fake Helm repository, aimed at modifying the resources monitored by Argo CD. This action causes the application to be out of sync with the repository, thus forcing one to synchronize it again to deploy the changes.

Impersonating or typosquatting Helm chart repositories or maintainers in Artifact Hub

In Artifact Hub, anyone can register a new artifact such as a Helm chart. We explored two situations to investigate possible avenues for exploitation:

• First, someone registers a malicious Helm chart with a similar name to another legitimate chart. This typosquatting trick can cause users to unwittingly download an infected chart instead of the legitimate one.
• Second, a malicious actor registers a username that is similar to that of another Artifact Hub user who is known for registering many different Helm charts. This could cause users who don’t see the slight difference in usernames to assume that the Helm charts from the malicious actor are also from the legitimate Artifact Hub user.

Scenario 1

Which of the two SonarQube Helm charts in Figure 13 is safe to download? How can one measure the trustworthiness of Helm charts? The following are signifiers of trustworthiness when using Artifact Hub.

• Verified publisher. The publisher owns the repository. This means that whoever published that artifact on Artifact Hub is the actual owner of the GitHub repository where the artifact is located. This process is done automatically and uses a YAML file called artifacthub-repo.yml to match the repository owner with the publisher. More details on this process are available here.
• Official. The publisher owns the software deployed by the packages. This process is done manually and requires a few prerequisites such as being a verified publisher. More information on the official tag can be found here.
• Image security rating. This refers to the security rating provided by the image scanning tool used on Artifact Hub. This is generated by an open-source tool called Trivy. The scans run twice every hour and the security reports are generated often. While it is important to pay attention to the image security rating, one needs to weigh it along with other information and be on the lookout for any critical security vulnerability. According to our findings on cloud threats and security recommendations in our Linux Threat Report 2021 1H, all the 10 most popular official Docker images from Docker Hub had at least one critical security flaw. More information on the image security rating is discussed here.
• Ownership claim. Since users are free to add any artifact that they want to Artifact Hub, they can add even artifacts that others own. If users see that their repository is already added there and seeks to claim ownership, they may do so through the ownership claim process by adding the artifacthub-repo.yml YAML file with an owner’s section. According to Artifact Hub, the user requesting ownership must be in Artifact Hub’s list and the email should match the one used for logging in to Artifact Hub. More details are given here.

Figure 14. Different SonarQube Helm charts available on Artifact Hub using the query, “sonarqube” on April 18, 2022

The “Official” tag on the first Helm chart in Figure 13 might lead one to think that it is the legitimate one. But the first Helm chart got an image security rating of “F” and received fewer stars than the third chart, so these two indicators should flag developers about the chart’s legitimacy. Having these many options for the same tool can complicate and render the decision-making process risky and inefficient, thus creating a situation that is conducive for a cyberattack. Similarly, one cannot disregard the possibility of malicious actors creating their own modified version of a SonarQube Helm chart and adding it to Artifact Hub as part of their scheme. Incidents like this commonly happen on GitHub, Docker Hub, and other package managers such as Maven and npm, among others.

The discernment required to navigate Docker Hub safely is just as important in this instance. Developers should carefully determine the trustworthiness and legitimacy of the files they download and install in their environments. Checking for specific tags such as “Official” and “Verified Publisher” is necessary to mitigate the risk of downloading infected artifacts.

Scenario 2

The case of a user named “joaquinito2051” is most apt in this scenario. User “joaquinito2051” had purportedly published thousands of Helm charts on Artifact Hub. However, none of those charts belong to him. This means that he is neither the owner nor the maintainer of the Helm charts on GitHub. Worst of all, a user by that name never existed on GitHub.

Figure 15. Search results on Artifact Hub for a spurious publisher named “joaquinito2051”, retrieved here on April 18, 2022

Figure 16. GitHub shows that the user "joaquinito2051” does not exist on GitHub, retrieved here on April 18, 2022

Despite the absence of a direct correlation between an Artifact Hub publisher or an Artifact Hub user and a GitHub user, we discovered that there is a way to log in to Artifact Hub using GitHub credentials.

Figure 17. Artifact Hub login page showing options to sign in with a GitHub or Google account

We explored the route of registering a new GitHub user as joaquinito2051 and tried to log in to Artifact Hub using the new GitHub user credentials we created.

Figure 18. New GitHub user joaquinito2051 registered to illustrate this scenario

Our first attempt at using the new login credentials did not work, however, as the username “joaquinito2051” had previously been registered on Artifact Hub. Therefore, we had to slightly tweak the username that was first registered on GitHub by changing the zero in “2051” to an uppercase “O”, a change that is barely perceptible to most users. Our second try went through.

Figure 19. New “joaquinito2O51” user registered on Artifact Hub

At this stage, the next step that malicious actors need to do is add some repositories and populate them with malicious Helm charts. Afterward, all that’s left for them to do is wait for unsuspecting users to download the infected files to gain access to their environments, aside from being able to use their resources. This is also where, in our attempt, we abuse the trust that users have on this seemingly popular yet unverified publisher to trick other users into downloading malicious artifacts. This point was the furthest we could go in the threat modeling process. To pursue it any further might cause issues that negatively affect legitimate users who rely on Artifact Hub.

Security considerations

Our analysis of the different threat modeling scenarios related to the three cloud-native app tools provides the context for outlining the considerations and security best practices to keep in mind when using the aforementioned tools.

Argo CD security audit and self-assessment

As previously mentioned, the Argo Project, of which Argo CD is part, underwent an independent third-party security audit in March 2021. The technical details, along with the final report, are publicly available here. There is also a security assessment being performed by the CNCF Security TAG team. The team at Argo also dealt with its first zero-day CVE in early 2022. This goes to show that security is a significant concern that the Argo community takes seriously.

With that said, a few documented security issues need to be highlighted so that DevOps teams take these into consideration when they use Argo CD in their environment:

1. Redis communication is HTTP by default. In a previous article, we discussed our discovery of how over 8,000 unsecured Redis servers were exposed in the cloud and were being abused through remote code execution (RCE) for cryptocurrency mining. Argo uses Redis as its cache service. Compromising Redis can therefore lead to issues in Argo CD as well.
2. Permissions required are cluster admin. After organizations deploy Argo CD and create additional users, we recommend that they disable the admin user as outlined in the Argo CD documentation.
3. There is no IP address logging into the API endpoint.

It is worth reiterating that enterprises can keep threats at bay if they take these into consideration when deploying Argo CD.

Helm Security Audit

The latest Helm security audit took place in August 2020. With regard to the use of third-party Helm charts, it is important to check the provenance and integrity of these charts. Helm has also made it easy to sign and check signed charts. The Helm documentation discusses this topic at length. Enterprises should not run code in their environment that they neither own nor trust. Additionally, Helm charts should only be deployed after their trustworthiness and legitimacy are established.

Artifact Hub

As the newest project of the three and given its sandbox status, Artifact Hub carries similar concerns to Docker Hub when it was first launched. As shown earlier,finding the right Helm chart and ascertaining the legitimacy of its origin are security measures that need to be taken. While the tags are helpful, they don’t eliminate the problem completely. Malicious actors can take advantage of these flaws to create charts with similar names or use typosquatting to trick users into downloading malicious versions of well-known charts. The onus to thwart cyberattacks therefore rests mainly on the users, and due diligence is a must before using charts made available on Artifact Hub. Artifact Hub’s package security report is a helpful resource that guides users to navigate this tool safely. For now, we urge developers to only use charts marked with the “Official” and “Verified Publisher” tags. Likewise, publishers need to ensure that their artifacts are being scanned.

Other recommendations

One of the cybersecurity hygiene practices that enterprises can implement is to apply signing and verification mechanisms along the pipeline. These can be done through signing commits or signing container images. Developers can adopt the Helm chart signing process to keep the integrity of the artifacts. Docker Content Trust and cosign from sigstore are two promising tools to consider. The goal is to ensure that only authorized users can make changes to these repositories and establish a proper chain of custody from provenance to the end users.

Trend Micro Solutions

Trend Micro Cloud One™ – Container Security delivers automated build pipeline container image and registry scanning. Designed for developers and operations teams, Container Security enables earlier and faster detection of malware, secrets or keys, compliance violations, and vulnerabilities, including those found in open-source code dependencies. Additionally, it provides the ability to detect threats in package manager-installed apps, as well as directly installed apps using Trend Micro’s industry-leading rules feed. Container Security helps developers extend even further to the left with Snyk’s open-source vulnerability database, offering early detection and mitigation of vulnerabilities in open-source code dependencies. With Container Security, DevOps teams are enabled to continuously deliver production-ready applications and meet the needs of the business — without making an impact on build cycles.