Apache APISIX In-the-wild Exploitations: An API Gateway Security Study

Previously, we published a study introducing Application Programming Interfaces (API) gateway security. This time, we will focus on the Apache APISIX API gateway and its security implications.

Apache APISIX API gateway components

To help us gain a better understanding of the Apache APISIX API gateway, let’s take a look at its components and information flow.

Figure 1. Apache APISIX API gateway information flow

The APISIX API gateway is built on top of the nginx web server and the OpenResty extension framework, which serve as the gateway’s core components. The API gateway also consists of the following components:

• APISIX core, Lua-based plugin(s), multi-language plugin runtime, and the WASM plugin runtime
• Built-in plugins that add features for observability, security, and traffic control
• An optional dashboard that communicates directly with the administration API to provide a graphical user interface (GUI) for configuring the gateway

APISIX core handles functions such as route matching, load balancing, service discovery, configuration management, and the provision of a management API. It also includes APISIX plugin runtime that supports Lua and multilingual plugins (e.g., Go, Java, Python, and JavaScript).

APISIX also offers an optional web dashboard that communicates directly with the administration API, which provides a more convenient way of configuring the gateway using a graphical user interface (GUI).

The risks associated with not changing APISIX’s default master API token

Misconfigurations are among the most prevalent software deployment threats, regardless of deployment location. It's important to note that secure configurations remain the user's full responsibility, including configurations in cloud environments. Users should be extra careful in deploying unmanaged services.

Evaluating security defaults and their implications plays an essential role in user education: Even when all the security features are available, they still need to be enforced and used appropriately. By default, using more secure patterns can make an application secure by design and prevent users from committing unintentional mistakes.

One example of an explicit security mistake is not changing the master password for the administration interface. When the master password is left unchanged, it provides threat actors administration access if the default password used is well-known or typical. 

Unfortunately, APISIX has a default master API token for accessing the administration API. It is irresponsible to keep using the default API key, and users should change it immediately.

It may be more convenient for software developers to not generate random tokens per instance (e.g., on its first run) and instead use the hardcoded API token, making it easy to follow the documentation examples.

We would not have the default credential problem if APISIX generated a randomized API token, prioritizing security.

Based on our observation, multiple exposed instances with default administration tokens are in the wild, just waiting to be exploited by threat actors, ignoring an essential security concept that administrators should permanently hide management API from non-local environments.

But why do we still see exposed administration plane instances?

Although some of the deployments we saw in the wild may be honeypots created by security researchers, others are potential misconfigurations or, even worse, misconfigured defaults, or default passwords and tokens that are publicly known and remain unchanged. There are several layers to this problem.

First, as we already mentioned, the hardcoded master password is publicly available.

Second, the administration plane is exposed. At this stage, it’s important to note that users should be extra careful of their containerized applications’ default configurations and runtime port exposures. Figure 2 shows container image examples from Docker and what can happen when network access isn’t restricted.

Figure 2. Docker documentation that shows what can happen to ports when network access isn’t restricted

Bypassing IP restriction limitations via vulnerability exploitation

More secure deployments will likely be protected by at least a web application firewall (WAF) that uses IP restriction rules that can prevent users from accessing the administration plane. However, as previously seen, this IP restriction rule has only been applied at APISIX’s API gateway level. In 2022, CVE-2022-24112 was disclosed. This vulnerability, one of the most-used flaws by threat actors in 2022, allows malicious actors to bypass IP restrictions using a custom header. When this vulnerability is exploited in an environment where the default administration password is left unchanged, malicious actors can perform remote code execution (RCE) on the API gateway.

Clearly, we can see that leaving default API keys and restricting access solely via network rules inside the configuration file are not enough, as threat actors can find ways to bypass these security deployments. Consequently, making the admin plane available to the entire private network with multiple devices connected to it is not the best idea. The network-connected devices can get compromised, possibly allowing for the gateway to be compromised.

There are a few lessons to take away from the disclosure of CVE-2022-24112. The root cause of the vulnerability is solely inside the IP limitation. The fact that the default configuration of Apache APISIX is vulnerable to remote code execution remained disregarded.

During our research, we saw the active exploitation of default credentials that resulted into the RCE.

Threat actors first identified exposed and vulnerable instances using default credentials. After that, they created a route using the Lua scripting capabilities of the APISIX gateway, resulting in RCE on the gateway when triggering the route.

Figure 3. APISIX RCE exploitation found in the wild

There are different ways to execute Lua code within the APISIX gateway, and none of which is sandboxed. Thus, it gives malicious actors executing Lua on the APISIX gateway access to all available modules without limitations. This allows threat actors to interact with the gateway in various malicious ways, such as deploying malware and crypto-jacking payloads, performing denial-of-service (DoS) attacks on the gateway, or worse, leaking sensitive data that can eventually lead to cloud account compromise or user impersonation.

Figure 4. Example of exploiting the APISIX gateway for RCE in a private network environment

Potential APISIX dashboard risks

The optional APISIX dashboard, which allows for user-friendly gateway configuration, has similar issues. This dashboard has direct access to gateway configuration, so users should be extra careful in cases where the admin API is shielded from the outside world, but the dashboard remains accessible. By default, the dashboard application requires credentials to grant access. However, the default username and password are set to “admin”, especially inside containerized environments.

It’s important to emphasize that when the APISIX dashboard is left exposed, it will have the same RCE implications.

The question remains — how prevalent are exposed APISIX services?

Figure 5. The number of exposed APISIX services found on Shodan

Exposed APISIX services in the wild

In January 2024, a Shodan search revealed 622 exposed unique IP addresses running some version of APISIX services in different ports.

When we checked the content of these exposed instances using default tokens, at least 39 were entirely open, exposing sensitive data.

Figure 6. Graph of exposed APISIX instances

Last year, we published a report about API gateway use cases, which include the following:

• Secrets management for accessing upstreams
• Token issuing and gateway authentication mechanism, including secure identity provider IdP settings
• Logging and secret stripping
• Plugin and script creation

In a previous section of this report, we discussed the missing sandbox for executing Lua code. Users should be extra careful when using Lua-based plugins or scripts and relying on user input, as these might introduce a vulnerability with an RCE capability on the gateway. For example, if a user writes a vulnerable code parsing HTTP headers, threat actors can exploit it and execute their payloads without restrictions.

The API gateway can be used as a central access point for accessing the organization’s entire API infrastructure, which makes the API gateway an ideal target for malicious actors. When malicious actors gain access to the gateway, it could lead to information disclosure of multiple secrets, such as all backend authentication tokens and IdP configuration secrets. Disclosing the IdP configuration could lead to user impersonation, token harvesting attacks, and misuse of the entire API ecosystem.

When discussing the APISIX default secret management system, we must mention its default storage, a place where the gateway configuration is stored. APISIX relies on etcd — an open-source key-value database used to manage critical information and configuration data on distributed systems — which is popular with Kubernetes users. The configuration is stored unencrypted, allowing everyone who has access to the etcd instance to read secrets stored therein. A static api-key for an API endpoint is an example of an etcd secret.

Figure 7. An API endpoint protected by an API key

Figure 8. An example of a plaintext API key

The good news is that since APISIX version 3.1, users can use encrypted storage of plugin configurations. Sadly, it is not a default setting and additional configuration is still needed.

Fortunately, there are other ways to store secrets; APISIX also supports storing secrets inside a vault and referencing them within the configuration. We want to encourage users to use vaults for storing secrets. However, users should note that they most likely need at least one secret to access the vault and thus ensure its security.

 
Figure 9. Vault types according to APISIX’s documentation

However, we generally don’t recommend storing secrets inside environment variables. In a report we published in 2022, we analyzed the dangers of using environmental variables for keeping secrets. Environmental variables are often misused and misunderstood, which leads to the introduction of security issues with tragic consequences.

We described this in detail in a more recent report where we talked about exposed container registries – hard-coding secrets stored inside environmental variables into container images.

We acknowledge that there is never a 100% secure system. The developers can use environmental variables in a “safer” manner; however, that requires understanding environmental variables’ underlying implementations and ensuring that no security boundary will be crossed (e.g., injecting them on runtime only, no hardcoding).

We provide an example of removing unnecessary environmental variables posing security risks and enhancing the security of Azure serverless solutions using custom container images.

Figure 10: An example of a plaintext API key for Azure Functions

The risks of excessive logging

Logging activities could be a lifesaver when troubleshooting an issue. It allows administrators and developers to simulate and understand where a problem lies. However, excessive logging fills your storage and serves as an additional attack surface, particularly when users log sensitive information. It is also typical for users to discuss pending issues with peers or upload logs for consultation.

But what could happen when sensitive information, such as access tokens, is logged and accessed by an unauthorized entity?

The answer to that question is dependent on multiple factors. How is the sensitive information passed through the request (whether inside the URL, the header, or the body), how thoroughly are activities logged, and, of course, how sensitive is the information? Let’s assume we pass a token as a parameter in an HTTP GET request. Our questions would be: Is it a valid token, not an expired one? What are the token’s permissions? Overly permissive tokens with extended validity are ticking bombs waiting to explode. 

Figure 11. An example of a token logging in URL
Source: Grafana Labs

We have tested an http-logger plugin with default settings; we only configured the http logging server URL. All the headers, including the authorization header, were present by default.

Figure 12. An example of a default setting for APISIX http-logger

However, what does this mean in terms of security? We could say that with great power comes great responsibility, as the configuration is entirely in the user’s hands. First, we should ask ourselves the following questions: What do we want to monitor? Do we genuinely need to log tokens? If so, can we strip, substitute, or extract sensitive information so that it’s not valuable for threat actors?

For instance, if we follow an example from APISIX’s documentation, the “supposed” default provides much less information. As we have proven again, relying on defaults is a dangerous practice. Users should know what they are logging and for what purpose.

Figure 13. Log based on APISIX’s default documentation

In any case, if we are logging sensitive information for whatever reason, we should secure the sensitivity of that information by ensuring its secure transfer via Transport Layer Security (TLS) and securing its storage (which might problematic especially on logging platforms). Otherwise, we are just increasing the risk of sensitive information being leaked.

Enforcing TLS is crucial when using third-party IdP services, such as Keycloak and AWS Cognito, and storing configuration secrets within the database.

The risks associated with APISIX API gateway plugins

Community plugins can be a great source of vulnerabilities, especially when they are managed by multiple parties with different levels of experience. In some cases, community plugins can even be integrated into a product together with their security issues. As such, plugin usage should be considered not only for its given functionality but also for its security.

APISIX’s API gateway split its plugin set into several categories: transformation, authentication, security, traffic, observability and serverless. Each of these categories has different risks associated with them. For instance, the transformation plugin’s risk is related to processing user input in the form of API requests, their responses, and the associated parsing issues. Meanwhile, authentication plugins are prone to unsecure sensitive information storage and incorrect verification implementations. On the other hand, observability plugins can have issues related to excessive logging of sensitive information, as previously described.

One of the authentication plugins is a basic-auth implementing RFC-7617, which conducts authentication using usernames and passwords. Even though this mechanism is deprecated in serious use cases nowadays, it perfectly demonstrates the security posture of API gateways related to secrets. After setting up this plugin, a valid username and password that are encoded in an authorization header are required for successful authentication.

Figure 14. An example of basic-auth plugin credentials’ storage

Figure 14 clearly demonstrates that credentials, including passwords, are stored in plaintext. Even though it is possible to enable encryption or vault reference on the password, the real question should be why the password is even stored and why the API gateway stores such secrets. Storing secrets makes sense if their raw form is necessary for third-party service authentication and when they are not passed from the initial request.

Storing a raw password is unnecessary for authentication purposes on the API gateway level. In fact, it is a security vulnerability that allows threat actors to exploit the password when a breach happens.

A secure way to store credentials for verification purposes is to create a salted hash and to store the hash instead of storing a raw password. When proper hashing is used, in case of a breach, threat actors will only have access to the salted hash. Given the salted hash’s properties, they won’t be able to recover the plain-text password needed for authentication, making the whole ecosystem more secure.

How threat actors abuse API gateways for cybercrime

How could threat actors target API gateways? We will use the APISIX API gateway as an example to answer this question. The first thing we should consider is a threat actor’s motivation — how they could leverage API gateway security misconfigurations for nefarious purposes.

Clearly, abusing a vulnerable computing resource that’s free of charge is a valid motivation and reason to deploy crypto-jacking malware that can generate revenue, including a bot joining a large-scale network used in distributed denial of service (DDoS) attacks.

Another motivation can be malware distribution. This case may be more complicated given the fact that API gateways serve as an API/backend resource and it’s unlikely to serve as a pure HTML resource. However, we can imagine an API service that’s returning software download links. In such cases, exploited API gateway instances can be configured to serve as malicious links, making it a supply chain attack.

In more targeted attacks, threat actors might use API gateway functionality to gather credentials, perform user impersonation, or get critical company information that they could sell or exploit in future targeted attacks.

In-the-wild perspective with honeypot data

Our honeypot, which was active from October 2023 to January 2024, was designed to monitor and analyze attacks on an API linked to a website. The website, hosted on port 80, was a front to attract attackers to the API, which was hosted on a higher port and linked to the website's source code.

Figure 15. The honeypot accesses data from the webpage to the API

The website received about 24,000 requests during this period, while the API endpoint saw around 15,500. Excluding the root "/" requests primarily from bots, we identified approximately 11,500 attempts using well-known generic exploits.

Figure 16. APISIX gateway-specific requests

The setup gave no clear indication of being managed via APISIX, requiring attackers to interact with the API to discern if it is or otherwise. Notably, there were 14 targeted attacks on APISIX, characterized specific URLs, additional API request parameters, and attempts to access the administration dashboard using common username-password combinations.

Figure 17. An example of an HTTP request that targets an exposed APISIX dashboard with default credentials

Conclusion

Misconfigurations are one of the most prevalent security issues today. Relying on supposedly secure defaults is a significant security risk, and changing default security settings, especially hardcoded and master credentials, is imperative. Updating software can be considered a best practice cliché, but its importance cannot be understated. Ensuring that your APISIX gateway is not vulnerable to any disclosed vulnerabilities, especially CVE-2022-24112, should be a substantial priority —  this vulnerability gives entire control of the API gateway upon exploitation and, at the same time, offers malicious actors an entry point to the centralized system that presumably aggregates your entire API ecosystem.

Given the nature of the vulnerability and the core features of APISIX, users should be cautious not only in deploying up-to-date API gateways but also in using customized Lua-based scripts, as these could introduce a vulnerability that threat actors can easily exploit.

Securing storage such as etcd and access to every admin plane and dashboard is equally essential. Administrators should consider applying network protection mechanisms other than WAF (for anomaly detection), as individual backends linked to the API gateway, as we described, can also have vulnerabilities.

It is important for system administrators to enforce security best practices as many modern software application deployments are not designed with security in mind. Aggregating access to multiple backends into a single point is becoming more popular, especially with hybrid cloud solutions. This is where applications’ backends are distributed among cloud service providers (CSPs) or on-premises platforms. However, before implementing this approach, users should weigh the additional security risks of aggregating to a single access point. If it is not adequately secured, the single access point can quickly become the organization’s single point of failure upon compromise.

API gateways aggregate access to on-premises backends and provide a functional way to integrate multiple cloud serverless frameworks and support identity provider configuration. However, it is important to remember that misconfigurations and excessive token permission could lead to full cloud account compromise or user impersonation.