GitHub Action Runners: Analyzing the Environment and Security in Action

In 2019, GitHub released its own CI tool called GitHub Actions. According to GitHub, the feature helps developers automate tasks within the software development life cycle. Workflows are event-driven, meaning that after a specified event has occurred, the developer can trigger one or more commands to run on their repository. One advantage of GHA is that developers do not need a separate CI tool since GHA runs directly from GitHub, where the project code is generally located. This article covers some security risks and best practices about using GHA as your primary CI tool.

Actions are formed by a set of components. These are the six main components of a GHA:

• Workflows: Automated procedure added to the repository, and is the actual Action itself
• Events: An activity that triggers a workflow; these can be based on events such as push or pull requests, but they can also be scheduled using the crontab syntax
• Jobs: A group of one or more steps that are executed inside a runner
• Steps: These are tasks from a job that can be used to run commands
• Actions: The standalone commands from the steps
• Runners: A server that has the GHA runner application installed

Figure 1. An example of GitHub Actions components 

Since there are some articles published around the risks of secrets, injections, and using third-party actions, we analyzed the runner itself and study the possibilities for attacks such as lateral movement and privilege escalation. This article focuses on the Ubuntu runner provided by GitHub.

As mentioned earlier, the runner is a server provided by GitHub to run your workflows (also known as Actions). They are deployed on Azure and are terminated after your automation is completed. Although there are some limits regarding this service, users do not pay anything to use it, even with a free GitHub account.

Early in 2021, there have been some reports of threat actors and malicious users abusing this service to trick vulnerable repositories and users into deploying coinminers and mining cryptocurrency for them. This involved forking a repository with a GHA enabled and then submitting a pull request to trigger the action. Since then, GitHub has placed some security measures to prevent this from happening, which will be discussed further.

A quick search on GitHub will show hundreds of repositories with GHA files, anything inside .github/workflows in a YAML format, running cryptominers. That would be fine if they used their servers for the mining process, but that's not the case. They use GitHub's provided runners to do that for free (just look at the runs-on tag). Here are some examples:

Figure 2. GHA code for running a Monero cryptocurrency miner inside the Ubuntu runner

Figure 3. GHA code for running a Monero cryptocurrency miner inside the Windows runner

For most of these repositories, GitHub has flagged them and disabled the GHA feature, which blocks them from doing that again in this repository. Users can see that the GHA is disabled by going to the Actions tab on the repository; here's the message shown:

Figure 4. Message stating that the GHA is disabled for malicious repositories mining cryptocurrencies

We decided to do reconnaissance and gather more information about GHA's environment by creating a new repository and running basic recon commands. We created this small YAML script to run some basic commands:

Figure 5. GHA used to perform reconnaissance inside the GitHub Ubuntu runner

First, from the id and uname –a commands. Nothing out of the ordinary: on line 5, we have the result from the id, which shows that the userID is a runner that belongs to the group ID docker, and it belongs to these other supplemental groups, adm and systemd-journal. The following line is the output of uname –a, which shows that the runner is a 20.04 Ubuntu Linux running inside Azure, which is also not surprising since Microsoft owns GitHub.

Figure 6. Output of the commands id and uname –a run inside a GHA Ubuntu runner

To check the running processes and ports, we added other commands in the action, such as ps –ef and netstat –nl. Under the assumption that everything were in containers, we were surprised with the results: 148 processes were running in the latest Ubuntu runner at the time of writing. The following is a trimmed output of the ps command:

Figure 7. Trimmed output of the ps –ef command ran on Ubuntu on Oct. 14, 2021

On the netstat command output, we observed port 22 with the secure shell (SSH) service running. But what caught our attention was a service running on port 8084, listening for connections.

Figure 8. Trimmed output of the netstat –nl command ran on Ubuntu on Oct. 14, 2021

Reading the documentation GitHub provided on Ubuntu, and also after installing and running a nmap scan on the runner, we found that the Mono XSP web server is active by default and runs on port 8084.

Figure 9. List of web servers installed on the GHA Ubuntu 20.04 runner

Figure 10. Trimmed nmap scan results on the GHA Ubuntu 20.04 runner

Before digging a little deeper, we decided to run a container introspection tool called amicontained by Jess Frazelle, which shows what container runtime is being used as well as features available. We wanted to confirm that the runner was not operating as a container and proved that the GHA runner is not inside a container.

Figure 11. amicontained showing that the Ubuntu runner was not in a container.

We observed that it does not have an AppArmor profile, seccomp is disabled, and it has 30 blocked syscalls. On the other hand, we observed interesting capabilities such as net_admin, net_raw, and sys_admin, among others, giving the user almost root-like permissions for its processes.

Since we were investigating the inside of a virtual machine (VM) on Azure, we also looked into querying the Azure Instance Metadata Service (IMDS). The output shows details about the instance regarding the name, region, and resource group associated with it.

Figure 12. Script for querying different IMDS endpoints from the GHA runner

Figure 13. Trimmed output from the IMDS endpoint request inside GHA

Since there was no identity tied to the GHA runner, there were no tokens to be retrieved from the query. This could have allowed a user to request more information about the Azure services and even deploy new instances depending on the permissions provided.

Figure 14. Curl request output to the IMDS identity access token inside GHA for the instance’s metadata information

We decided to test some scripts to scan for any security issues or vulnerabilities with Docker inside the runner. We started with a container enumeration and exploit script using Deepce often used by pen testers, hackers, and developers alike. By adding the following commands to the GHA workflow, we were able to run Deepce inside the runner.

Figure 15. Workflow commands for running Deepce in the GHA runner

The enumeration provided similar but more detailed results than the id command, such as the Docker version and the list of container tools (like Podman and Kubectl) installed, which are also listed in the documentation linked earlier.

Figure 16. Trimmed deepce enumeration output on the GHA Ubuntu runner

The second part showed some version information, system architecture, potential CVEs, and a list of running containers.

Figure 17. Trimmed enumeration output on the GHA Ubuntu runner.

Since we are not on a container but an actual VM, we tried another script for escalating privileges in Linux environments, called Linux Privilege Escalation Awesome Scripts (linPEAS).

Similar to earlier tests, we observed basic system information such as the OS, version, users, groups, and hostname. We also found more details about the CPU information of the Ubuntu runner:

Figure 18. Basic information gathered with linPEAS

Figure 19. CPU information of the Ubuntu runner

We found it interesting that linPEAS also checked for CPU-related vulnerabilities. For the most part, the runner was updated and had mitigations for gaps except for Speculative Store bypass, which is the vulnerability CVE-2018-3639 reported by Microsoft Security Response Center (MSRC) and Google Project Zero. Unfortunately, although the script mentioned it was vulnerable, we were unable to validate this issue.

Figure 20. CPU vulnerabilities on the GHA Ubuntu runner checked with linPEAS

Another detail we found through the linPEAS script was seeing which Linux security modules were enabled on the runner, and it appeared to be few since this is supposed to be an ephemeral host for CI/CD pipelines. Unfortunately, according to linPEAS, only address space layout randomization (ASLR) was enabled on this machine.

Figure 21. List of security modules enabled on the GHA Ubuntu runner

The process of testing and researching GHA can be overwhelming and requires detailed attention as any changes made needs to be triggered every time: deploying a new runner, setting it up, and running action commands. Although the speed at which the runners would start is impressive (which made us initially assume these were containers) and after running almost 100 different workflows, we decided to set up a reverse shell where we could run commands interactively on the runner while it was alive.

We checked if the runner had Netcat installed by default and saw a different version of it that did not have the -e/--exec option, which is commonly used to execute a specified program when a client connects to it. It is also handy for setting up backdoors.

Figure 22. Output of Netcat help command from the Ubuntu runner

We improvised by downloading and installing Netcat from the source with additional commands added to the action. This allows us to execute a specified program such as a bash when a client connects to it.

Figure 23. Installing netcat on the runner

Figure 24. GNU netcat installed with the –e option

We set up a command and control (C&C) server and used an AWS Elastic Compute Cloud (EC2) instance and started listening to a specific port. We started with 5555, but that didn't work, probably because of firewall rules from Azure and changed the port to 443. Then we configured our GHA to connect to that instance on the specified port 443. Once we reran the workflow, we received a shell on the EC2 listening on that port:

Figure 25. GHA installing and running netcat and connecting to EC2 instance via port 443

Figure 26. EC2 instance listening to the specified port and sending commands to the runner.

After seeing that the listening and C&C server setup is working, we used Socket Cat (SOCAT) and used the GHA runner interactively to send any commands. While a free plan offers a limited time window of six hours, that is more than enough time for an attacker to try escalating privileges, move laterally, or even use this runner as a pivot for attacking other sources. It is also possible to put this backdoor hidden in a custom action code and call it from another repository such as the GitHub Marketplace. By creating a fake GHA on a repository, we were able to call it from another repo and get a reverse shell on the EC2 instance:

Figure 27. A repository calling a composite action from another location

Figure 28. A composite action from a third party that does a reverse shell to a C&C server

We go back to the Mono XSP web server as it raised some red flags for us. First, because it is an old web server. According to the documentation, “XSP is a standalone web server written in C# that can be used to run your ASP.NET applications with minimal effort.” So why is that installed and running by default on our runner? It does not have many reported vulnerabilities, with the latest one being from 2010. Checking Shodan to see who else runs this web server today, the results also showed that few were using it. It seems that these started popping up on Shodan in 2020:

Figure 29. Shodan scan results for Mono.WebServer.XSP/4.7.1.0 Linux on Nov. 24, 2021.

According to the results, most of the Mono Webserver XSP runs on port 8084 and returns an error when accessing the main page:

Figure 30. Mono.WebServer.XSP/4.7.1.0 inside the Ubuntu runner returns an error notification.

We decided to investigate this application further. Searching online for some of these strings, we found some issues reported in the Mono repository and found no fixes.

We downloaded the Mono XSP code from GitHub to check the code. The exact version used in the Ubuntu runner is version 4.7.1 from January 2020, which is considered the latest version. We tried a few static application security testing (SAST) tools and code scanners to see if we could find anything, and found that being written primarily in C# made things more difficult since there are few options that can scan this language correctly. XSP also requires that you build it first, which made things even more complicated so we decided to focus on command line (CLI) tools that did not require us to build at first.

After running a couple of different tools such as Semgrep, GitLab, Flawfinder, and Horusec, we yielded a few results but nothing notable that could lead to a compromise from our perspective. Most of the issues found were related to the shim.c and shimtest.c files located inside the shim folder. Others were either false positives or related to test files. But this could still be a possible entry point for compromising other users’ GHA.

Figure 31. Semgrep scan results for Mono XSP using C# rules "r/csharp"

Figure 32. Some of the Horusec scan results using Flawfinder for the Mono XSP repo.

Aside from putting a backdoor on the runners and running interactive shells, we wanted to see what else was available on the network our runners were in. To summarize the previous test results, we know that they run on Azure, they are not containers, and they start fast. Is there a set of runner VMs previously created and waiting to run different jobs and workflows? According to the Nmap scan results, either that or there were other users’ runners running simultaneous to the scan since there were other servers that resolved their hostname while others only returned the IP address. As shown in the results in Figure 33, our runner (10.1.0.33) has a hostname that was a bit different from the others, and it also has ports 22 and 8084 open since we ran it from the runner itself while the other ones were filtered.

Figure 33. Nmap scan results from inside the Ubuntu runner resolving the hostname

Out of the 65,536 hosts scanned, only 34 of them resolved the hostname. And 33 of those were fv-az224-611.internal.cloudapp.net. The other was our runner, which had a longer hostname such as fv-az224-611.bi4zmy1qo3tuniz3adueuvvcza.cx.internal.cloudapp.net. The other hosts only returned the IP addresses:

Figure 34. Nmap scan results from inside the Ubuntu runner unable to resolve the hostname

While we could not access someone else's GHA runner information, we did get ideas on what we could do to try. This research gives us an overview of how GHA works and some exciting details about Azure.

GitHub Actions is an excellent service provided by GitHub. If users are already using GitHub to store their codes and projects and are currently considering GHA, users should remember that they are trusting every software that comes installed on the latest Ubuntu runner by default with the GitHub Runners provided.

On the flip side, developers would have to make sure their respective projects and workflows are safe from threats and attacks for the development’s entire life cycle. While GitHub continuously improves the security platform for every security incident documented by researchers, cybercriminals have taken notice of the ubiquitous opportunities for illicit activities. As discussed, malicious cryptominers in GHA can be used as a jump off point to spread it to other servers and projects. Moreover, cybercriminals' interests in embedding malicious cryptominers in GHA may simply be a start and symptom of more malicious attacks and malware that can abuse the platform. For businesses and supply chains, vulnerabilities and other openings in workflows, components, and projects can be used for reconnaissance, information and data theft, and operations disruption, among other risks.

GHA offers security customizations that all developers can enable. Users can set some permissions and security settings for their workflows in the Settings > Actions menu of each repository. First, there are Actions Permissions that can be set to either allow or disable actions, or at least restrict to run only on local or select ones.

Figure 35. Options for Actions Permissions for each repository on GitHub.

Developers can also require approval for actions when it comes to pull requests from forked projects. We believe this was created to prevent cryptomining attacks mentioned earlier. This prevents outside collaborators from triggering workflows following the options provided:

Figure 36. Approval requirements for outside collaborators to trigger GHA

Here are additional best practices users can follow to avoid having their GHA exposed or compromised:

• Protect your secrets: Any token or secret added to the GHA can be retrieved inside the runner and exposed in the logs, although GitHub does an excellent job of trying to mask those. Make sure that whoever has permission to modify Actions can also can see the secrets. Also, be careful with external pull requests that might allow unauthorized users to read and compromise secrets.
• Be aware of command injection attacks: GHA uses a lot of event context data that a malicious user could manipulate, which should be treated as an untrusted input. Most of them are located inside the github.event.* context and can be used to perform command injection attacks.
• Examine and research on the listed GHAs on the Marketplace: As with other open source marketplaces, extensions, or plugin stores such as WordPress, Jenkins, and VSCode, anyone can create and upload a new plugin or extension on the GHA Marketplace. The GHA Marketplace benefits organizations by allowing them to share their actions for customers to use their products, but it can also allow malicious users to share their malicious GHA code. With that in mind, users would have to trust whatever code is being used if they leverage the marketplace. Currently, there are no verification or security checks for GHA published in the marketplace.

Aside from customizing the security features and permissions of developers’ GitHub repositories and GHA runners, users can also reinforce the security for their project’s entire life cycle. Trend Micro™ Cloud One™ - Open Source Security by Snyk connects with repositories and CI/CD pipelines to scan projects for potential open-source components with potential vulnerabilities and weaknesses. It automatically finds, prioritizes, and reports vulnerabilities and license risks in open-source dependencies used by applications, and provides ongoing security insight to help developers and organizations identify, manage, and resolve open-source risks by delivering visibility into the overall state of security compliance posture, benefitting SecOps with consolidated guidance and risk mitigation.