Unleashing Chaos: Real World Threats Hidden in the DevOps Minefield

Download Unleashing Chaos: Real World Threats Hidden in the DevOps Minefield

By Alfredo de Oliveira and David Fiser

Application development rapidly changed with the increased use of containers, which has become a crucial part of the DevOps methodology. Developers got used to configuring our applications using environment variables instead of configuration files within the application directory, a benefit allowing more flexible deployments. 

We use them so much that we inadvertently pass secrets using environment variables without thinking of their consequences and forgetting the underlying implementation entirely or, worse, ignoring it completely. Everyone must be aware that environment variables are inherited in every child process. We proved that it is a dangerous feature, especially inside managed cloud services as the information can leak to more sophisticated attacks, including  remote code execution.

Developers took it a step further. Convenience empowered us to aggregate these environment variables into a single file, .env, which combines harmless application settings with highly sensitive cloud service provider (CSP) access tokens, unconsciously returning to the roots. Developers have adopted .env files so profusely that they have forgotten its sensitivity and left it available to the public.  

Our honeypot data proves that threat actors are actively looking for those exposed .env files. Thus, these files are a ticking bomb deeply rooted inside DevOps practices. The consequences of such exposures are massive data breaches (consisting of hundreds of terabytes of data), supply-chain attacks, and economic fines. 

Our research paper uncovers the hidden dangers in DevOps using real-world examples, including the popular web application framework Laravel, as well as modern AI and LLM applications. The report focuses on the adoption and use of .env and our discovery of multiple malicious payloads harvesting and exploiting these neglected credentials. Specifically, we found more than 600,000 secrets exposed in the wild, including CSP access keys that could lead to account takeovers.  

We also discuss the use of honeypots to simulate a realistic DevOps project environment, leaking credentials through .env files to monitor unauthorized access attempts. The data collected showed a noticeable increase in .env file requests, suggesting a growing interest among attackers. 

The analysis of the honeypot data revealed various techniques used by attackers to locate and access .env files, including directory traversal attacks and automated scanning tools. The requests originated from a diverse set of IP addresses, indicating a broad interest across different attacker profiles and geographies.  

Here are some of the most interesting findings discussed in the full report:   

• The lack of DevSecOps policies representing real word danger for modern AI and LLM applications, ultimately exposing highly sensitive customer data. 

• We found over 480 exposed and writeable container images that are clearly related to AI. This includes LLM applications with all its prompts and API configurations. 

• A honeypot simulating a misconfigured cloud environment was set up to observe attack patterns, common entry points, and the types of resources most frequently targeted. The honeypot recorded 24,488 unauthorized access attempts in one month. 

• 18,680 incidents were unauthorized reading and writing to cloud storage accounts with primary interest in accessing and potentially exfiltrating data. 

• A significant majority of the 18,747 requests originated from a single IP address, suggesting the operation of a dedicated adversary or a botnet controller. 

• We conducted searches for .env files in production-ready codebases hosted and accessible over misconfigured platforms, leading to the discovery of secrets across 1,754 unique hosts. 

• The 1,754 hosts combined contained 710 GB of data, and sensitive information was found in 82,687 files. The .env file was the most popular by a large margin. 

• A total of 677,426 secrets were found among 1,754 different sources. The most common types of secrets were generic-api-key, jwt, and cloud access tokens. The secrets also included OpenAI API keys. 

• The exposure of private keys, cloud access tokens, and Slack legacy tokens was identified as high-security risks. 

• A high number of database and email configurations were found inside the .env files, representing a significant portion of the secrets found in the generic-api-key category stored in plain text.  

Our report also tackles the proactive measures organizations need to take against misconfiguration vulnerabilities, such as scanning for .env files and the presence of secrets inside container images and production environments, preventing committing secrets inside content management systems (CMSs) and image repositories, as well as rotating secrets in case of exposure.  

Read our full report for more information.