HTTPS over HTTP: A Supply Chain Attack on Azure DevOps Server 2020
We provide the technical details of a supply chain attack on an improperly configured Azure DevOps Server 2020, specifically in the continuous integration and continuous delivery (CI/CD) Pipeline Agent communicating without TLS.
Save to Folio
The need for data encryption during transmission has paved the way for organizations to rely on TLS — not just for sending data through the internet, but even within trusted corporate environments. Without the use of TLS or SSL, the authenticity of transmitted data and the identity of endpoint can’t be verified.
In this blog, we provide the technical details of a supply chain attack on an improperly configured Azure DevOps Server 2020, specifically in the continuous integration and continuous delivery (CI/CD) Pipeline Agent communicating without TLS. We have reached out to Microsoft prior to the publication of this blog. We feature their best practice recommendations to help mitigate this risk.
Azure DevOps Server 2020: A Brief Overview
Azure DevOps Server, previously called Visual Studio Team Foundation Server, is an on-premises software that is used for reporting, requirements management, project development, automated builds, project management, and CI/CD.
This article focuses on the CI/CD pipeline step within the supply chain process. It is common for the CI/CD pipeline to use distributed architecture and build agents running on multiple environments, such as containers or machines, to perform a build task. Azure DevOps Server is no different, providing its own pipelines agents available for multiple platforms. This not only prevents unwanted code execution on the primary machine or on the main server that contains configurations and secrets, but also provides the ability to build code for different platforms and perform more builds simultaneously.
When a build task is executed, it requires a process spawn, which will configure and compile the source code. This can be referred to as remote code execution (RCE) on the build agent. It should be noted that this is a normal process — as long as the process is not malicious or is able to perform tasks as intended. It is also important to mention that during this process, the confidentiality of the source code is transferred to the build agent. This highlights the crucial need for the establishment of a secure communication channel in order to keep the whole process running securely.
Let us focus on the communication channel that is used between the build agent and the Azure DevOps server. The Azure DevOps server runs as a web service on the Internet Information Software (IIS) web server. In order to register a new agent, a user has to install it inside the “Build Environment” option and execute a configuration script. This script will then connect to the server and register the agent, which will require authentication for this action to go through. So far, so good, it seems. However, the devil is in the details as the default settings of the Azure DevOps server relies on Hypertext Transfer Protocol (HTTP) and would need to be manually configured to use Hypertext Transfer Protocol Secure (HTTPS).
Attack Scenario on Azure DevOps Server Using HTTP
In this section, we will describe an attack scenario that a malicious actor can execute if HTTP is kept as a communication channel, as opposed to configuring it to use HTTPS. It’s important to note that in order for this attack to be successfully carried out, the attacker needs to be already inside the network — a risk in and of itself. This also includes infected devices such as routers that can sit between the server and the agent and are not within our control.
Let us start with the agent registration. When a configuration script is executed on the agent, which happens just once, it starts a connection to the server. The server then requires authentication, which is done using the Microsoft New Technology LAN Manager (NTLM) protocol.
The agent generates a new RSA public-private key pair. The agent sends the public key together with other agent metadata to the server, which then stores it in the database. As becomes evident, there is an attempt at security even when an unsecure channel is utilized. However, the server doesn’t generate a key pair for the agent to be able to verify its responses, which will play a significant role later on.
After this, the agent service still needs to be executed, which is done via a provided run script. The agent will use the configured settings to connect to the server through a series of HTTP connections that are kept active, and the agent will wait for the server’s reply with job definitions.
However, when the agent connects to the server, a session is established, during which the server generates an AES key, which is encrypted by the agent’s public key by default. This means that only the agent will be able to decrypt the content of the encrypted message, which contains the build definition. As this might seem like bulletproof security at first glance, it is imperative to point out the existence of the “encrypted” property that is present in the session initiation response.
This property is connected to the AES key generated by the server. Upon further analysis of the agent binaries, we discovered that it is possible to send an unencrypted version of the AES key.
The AES key is then used for encryption of job specifications, including RCE commands.
The Supply Chain Attack
After our discovery, we checked to see if it is possible to perform an RCE attack on the agent when an attacker is on the same network segment or has access to a network device that is on a path between the server and the agent.
We have confirmed that one of the supply chain attack scenarios would consist of the following steps:
- Performing Address Resolution Protocol (ARP) spoofing, a type of attack that allows a malicious actor to send fraudulent ARP messages over a local area network for the purpose of linking the malicious actor’s media access control address with the IP address of the server. This will allow a malicious actor to intercept connections and pretend to be a legitimate server.
- Simulating server replies on agent requests, which forces the agent to execute arbitrary commands.
After performing the aforementioned steps, we were able to execute a successful attack on an Azure DevOps Server agent, which allowed us to spawn a reverse shell.
What Makes the Attack Possible?
Several platform design features allow the possibility of a supply chain attack on the system:
- The agent does not verify if the reply is coming from the legitimate server or otherwise.
- The AES key that is used for encrypting a job specification can be successfully replaced by a custom AES key by setting the “encrypted” property to “false,” forcing the agent to accept the unencrypted custom version of the AES key. Even if this feature isn’t present, the attacker still has the option to sniff out a public key transmission while the agent is being configured and use it for encrypting its own AES key.
- TLS is not configured by default. The user needs to manually configure it.
As build agents are generally used to transform the software’s source code into an executable format, the impact of a successful attack on an improperly configured Azure DevOps Server should be considered as similar to the supply chain compromise in the SolarWinds case, which gave malicious actors unhampered access to the targeted organization’s network.
Similar to how attackers made use of the Sunburst backdoor in the SolarWinds attack, an attacker would also be able to implant a backdoor into a build environment, which would allow the attacker to interfere with the build process itself. A successful attack would also mean that an attacker would have access to the build artifacts, which typically consist of source code and executables.
To mitigate attack scenarios similar to this, users should remember not to rely on default settings and to use TLS.
These recommendations also mirror the ones that Microsoft shared when we reported this attack scenario to them:
“In the testing and reproduction of this issue, it seems to rely on communications via HTTP rather than HTTPS. While HTTP is an available option, it is meant for build environments that have been secured via network abstraction layers or other preventative/restrictive security measures because there are known, inherent weaknesses in HTTP. In most customer use cases, we recommend following best practices which include deploying the build agents/environments per the guidance published here."