Cloud
Secure Secrets: Managing Authentication Credentials
Secret management plays an important role in keeping essential information secure and out of threat actors’ reach. We discuss what secrets are and how to store them securely.
This entry aims to introduce and impress upon organizations the importance of proper secret management in today’s environments, as a leaked password could lead to data breaches with heavy consequences. Such important access information has taken on a greater significance as more workload is being pushed towards the cloud.
Secret management plays an important role in keeping essential information secure and out of threat actors’ reach. Well-kept secrets, such as passwords and other authentication credentials, ideally allow the right people access to important assets and minimizes the possibility of exposing organizations to attack. However, secrets can also be double-edged, especially when they are not as well-hidden as organizations would like to believe.
We have already discussed weak parts of the supply chain in an earlier article. One thing that resonated throughout every aspect of the chain was the importance of having a proper secret storage. This article will focus on what these secrets are, their role and impact in security, and what kind of problems they may entail.
What are secrets?
First, let us define what “secret” means. In this discussion, the term “secret” pertains to sensitive information used to access systems that require authentication. This includes login credentials such as usernames, email addresses, and passwords, as well as access tokens and private keys. The SolarWinds breach provides a good example of how tenuous secrets can be since the attack involved the use of a weak password — a supposed secret.
The value of strong passwords
With that example in mind, let us discuss password strength. Password strength can be measured by its length and the size of the used alphabets. Using these values, we can compute the number of maximum guesses needed for a brute force attack. In theory, the maximum number of guesses is defined as xy, with x denoting the number of characters in the alphabet and y the number of letters used in the password. The result would be hundreds to billions of possible combinations.
However, there are exceptions to this computation. When ordinary dictionary words are used, passwords become easier to guess. Generally, when the password uses more random characters from a richer alphabet, it becomes more secure. In addition, it becomes less secure each time the password is reused.
| Alphabet | Alphabet size | Number of possible combinations for 6 characters | Number of possible combinations for 10 characters |
| A to Z |
26 | 308,915,776 | 141,167,095,653,376 |
| A to Z, a to z | 52 | 19,770,609,664 | |
| A to Z, a to z, 0 to 9 | 62 | 56,800,235,584 | |
| All printable | 126 | 4,001,504,141,376 |
Table 1. The number of maximum brute force attempts required to crack a password of a given length
Unfortunately, lengthy and strong passwords require memorizing a long sequence of random characters, which is naturally challenging for users, especially when considering multiple accounts.
Even if a user memorizes one lengthy password, the tendency to use it everywhere diminishes its strength. Password reuse is another security risk. In case of a breach, a reused password compromises other services, leading to greater impact.
While strong passwords are still important, we have to question if these types of secrets are our only reliable means of safeguarding access to mission-critical assets and systems. Is the password authentication method the only way to log in?
Alternatives to passwords
Fortunately, alternatives to passwords are available. For example, it’s been a standard for Secure Shell (SSH) services to use asymmetric cryptography and private keys for authentication; this limits the risk of secret guesses significantly and enforces the need for proper secret storage.
While this alternative still needs a token to authenticate users, it is much less reliant on human memorization. For instance, we can also use a vault, which will store credentials and provides tools for generating secrets. These tools also prevent the creation of weak passwords and password reuse.
While it avoids the pitfalls of using standalone passwords, keeping secrets in a centralized storage solution introduces a risk of all credentials and passwords being compromised in a single breach and highlights the necessity of treating such storage as a vault. Securing access to this vault is crucial, wherein biometrics or multi-factor authentication (MFA) authorization are highly suggested additional safeguards.
MFA itself significantly decreases the chance of compromise, even if the credentials of an account have been exposed, as it requires another authentication typically done through a mobile app. The probability of a user’s cellphone being compromised or stolen together with their secret being exposed is significantly lower.
Expiring secrets
If you have been working with computers, you have probably been forced to change your password after a certain period. While some consider this an inconvenience, letting secrets expire has a good reason. Secrets can be cracked after some time — the longer they are valid the higher the chances that some threat actor could successfully find a way to steal and leak it upon a data breach.
Types of secret storage
Let us now talk about the different ways to store secrets. As we have mentioned earlier, secret storage should be treated and therefore secured as a virtual vault, because cracking it would allow threat actors access to passwords and the accounts linked to them.
The number of systems in workflows that have to communicate with each other is increasing. They must do so securely, and each request from its users must be authenticated first for that to happen. In case users don’t want to enter credentials every time or it’s not practical for credentials to simply be memorized, they must be stored.
There are multiple ways credentials can be stored:
- Plaintext: the most unsecure type, wherein read access to the file would allow access to the system.
- Hashed: the type mostly used by servers to store a fingerprint of a secret, preventing intruders from obtaining the plaintext password. For this type, we emphasize the usage of a salt, or another hashing layer on top of an encryption algorithm to make decryption more difficult for potential attackers.
- Encoded: another form of plaintext, an example of which are base64-encoded credentials, because of non-readable characters.
- Encrypted: the credentials are encrypted with another password, decreasing the probability and impact of an unauthorized leak. A person who can see an encrypted file would not be able to decrypt it and get the needed credentials, without first knowing another secret.
From these four types, it is easy to surmise which would be the most advantageous and secure for storing secrets. However, aside from these four types, organizations have to grapple with other considerations that can affect their plans when it comes to storing their credentials.
Other applications to consider
One way to tackle secrets storage is to use an external application such as vaults. The main advantage of secret vaults lies in the way it stores secrets in encrypted form and its ability to change the secrets from one place and reflecting that change to multiple applications without a code change.
However, an access token to the vault has to either be stored on the machine or obtained dynamically. In the case of the latter, dynamic access token via MFA confirmation would be the best fit. Hence, MFA would not be the best option in all cases, especially for highly automated systems.
The problem with plaintext
Not every secret storage application stores credentials in encrypted form by default. Documentation should verify if stored secrets are encrypted; they should also be checked to prevent exploiting misconfiguration issues.
In some cases, storing secrets in plaintext is considered a vulnerability. Generally, plaintext is an unsecure and poor design decision, and should be reconsidered. Much of our discussion on DevOps and cloud problems later on also revolves around plaintext.
High level of trust
Finally, even if the secrets are stored in encrypted form, organizations would still have to achieve and maintain a certain level of trust and shared responsibility among their personnel. This is because certain individuals would still have to know or hold the decryption key.
Organizations should never forget that secrets must only be accessible to a select few. This translates to keeping read permission only for users or applications with a proper reason to access this storage or systems. Encrypting the secrets would then add additional security, with malicious actors having to work much harder to crack the credentials.
Secret transport
Aside from stored secrets, security would also be necessary for services that have to be configured and require transfer. It is important to emphasize that moving secrets is normal and therefore should be accounted for and done through a secure (i.e., encrypted) channel.
A secure channel means using a protocol that is secure by default, examples of such protocols are SSH or HTTPS. Using plain text-based protocols such as HTTP, SMTP, FTP to transfer secrets can allow malicious actors to intercept the communication and leverage the leaked secret.
It should be mentioned that putting secrets into a source code management (SCM) system is also not a good idea as it gives access to more people than necessary. Cases involving public repositories adds another layer of danger to SCM. Should an organization insist on using SCM, they should use tools that will encrypt the secret, (e.g., git-secret) to protect against the possibility of having the SCM used as an entry point in a supply chain attack, as we have discussed in a previous article.
Impact of secrets
This section looks into the implications of using unsecure secret storage in the context of today’s developing technologies, namely that of DevOps and the cloud. We also briefly describe a real-world scenario in which a threat actor goes after credentials in infected devices.
Secrets and DevOps
Today, most have gotten used to considering development and “local” environments as inherently secure. This is reflected by practices such as storing secrets in their unencrypted form, designing software to run in secure environments only, and not hardening security features by default. This provides additional comfort for users as it removes the need for a complex security configuration. The trade-off, however, is how these practices increase the risk of incurring a security incident. We have already seen the repercussions of supply chain attacks, weak VPN security, VPN vulnerabilities, etc.
It is also important to note that “local” environments are not that local anymore, as we are seeing more companies moving into the cloud. A plaintext secret that has been stored on a device that uses hardware-based (HW) encryption is simultaneously stored on a device connected to the cloud. In short, it is not secure to store a secret in plaintext whether in the cloud or not.
To give a real example from the world of DevOps, Visual Studio Code has only recently been able to provide support for secret storage in its release of version 1.5353, released in January 2021. Before this update, no official API was available for storing secrets in a more secure manner and, thus, it was the developers’ responsibility to implement it securely.
After conducting a follow-up research we found out that other 338 extensions have the same problem – storing secrets in plaintext. These third-party extensions included a binding for multiple cloud service providers (CSPs) and SCM systems.
Compounding to these issues is the reality of threat actors focusing on credential harvesting that would make Visual Studio Code users easy targets for such attacks.
Secrets and cloud threats
As more companies are adopting cloud technologies, threat actors are continuously evolving their payloads and performing targeted searches for cloud service credentials. It is important to consider how security can be enhanced by using different storing mechanisms — especially since secrets are often stored in plaintext — because users are taking having a secure environment for granted. Leaving secrets exposed in plaintext makes it much simpler and easier for threat actors to steal them. When operating in the cloud, organizations must be contingent of the shared responsibility model. The security of their used credentials remains their responsibility even in the cloud.
As we mentioned before, achieving a high level of trust among all those involved in keeping secrets secure is also essential. Cloud providers always refer to the shared responsibility model, which is applicable especially in addressing cases of user misconfiguration. This stance of shared responsibility is essential because no software is 100% secure. For example, an earlier blog entry covered how attackers, after compromising an instance, requested the AWS metadata service to obtain credentials or secrets. This highlights how attackers could access credentials on a cloud instance once they have compromised it to further their attack.
Also, in a more generic approach, CSP command line tools generate authorization tokens upon first configuration and store those tokens in plaintext format either in files or as system variables. Knowing this functionality, threat actors can develop malware to search for those credential tokens once they break into an instance. Depending on the level of access of the stolen credential tokens, the threat actors can keep the attack going and spread more payloads internally.
A behavior that called our attention in one of our investigations was how attackers started using native CSP tools at different points of their attacks instead of developing their own or using exploits. In such cases the attacker would have the same authorization as the service, as configured by its user. This behavior means they are learning all the possible options and parameters of native tools and abusing them. Figure 7 shows that their scripts have commented lines with native tools documentations, which led us to think the attackers are learning on the fly. In internal environments like Lambda, the usage of native tools are pre-authorized so if the attacker succeeds to upload the tool there, they would have the same permissions as the service.
A real scenario
Here’s a real-world scenario that highlights the importance of proper secret storage. We have seen threat actors seeking credentials inside infected devices with a malware capable of infecting other devices should the initial infected device have credentials of the others stored in its system. This is after they have gained the ability to modify a cloud service.
A more recent investigation also showed that threat actors are adapting to the newest technologies and are actively trying to exploit infrastructure as code (IaC) software, such as Ansible, Chef, and SaltStack tools, to distribute their payload to as many devices as possible.
Containers
Finally, organizations should be made aware of how threat actors have learned to adapt to the increased usage of containers. A typical container is nothing more than another process with limited permission and capabilities; however, this is not the case for privileged containers, as we had warned before.
Recently, we have also witnessed treat actors incorporating container exploit features into their arsenal. Running a service inside a container often needs a secret to obtain access to another service, so the same principle for secrets and its storage also applies to this technology.
The following video shows how a potential attacker can break out of a Docker container using tools from the TeamTNT toolset.
Securing secrets
Organizations have to recognize by now how secrets have taken a deeper significance in their role as gatekeepers. This is why it’s best to assume that threat actors are always on the lookout for ways to reveal these secrets and prepare accordingly.
Securing secrets also demonstrates the relevance of adopting the shared responsibility model. Organizations should not assume that data stored in the cloud is already protected from risks, thus allowing them to take additional security measures such as employing cloud-specific security solutions to protect their cloud-native systems.
As always, there is no silver bullet that will address all of the issues we have presented here. However, organizations can start by reviewing the services they are using. By having a firmer grasp of the type of secrets they are storing, and how they are storing and transporting them, they can create an accurate risk matrix and be better prepared for data breaches.