Leaky Labels: Bypassing Traefik Proxy Leveraging cAdvisor Metrics

By Nitesh Surana

In our previous article, we detailed how misconfigured deployments of container advisor (cAdvisor) might inadvertently expose information stored in environment variables that contained credentials, tokens, and the like in the form of Prometheus metrics. Apart from the environment variables, we observed that the metrics contained all the container labels for each running container exposed by default. Generally, container labels are not considered as a secret — or are they? In this article, we share our findings around the risks of exposed container labels in cAdvisor metrics with a case study of a widely used cloud native application proxy known as Traefik proxy.

Introduction

While examining the exposed information in the metrics endpoint of cAdvisor, we noticed that it exposes container labels for all the running containers on the “/metrics” endpoint. Container labels are key value pairs that are stored as a string. They are used to store metadata such as licensing information, maintainer information, versioning about container objects such as container images, running containers, volumes, daemons, and networks. For instance, one can use the LABEL instruction in a Dockerfile at build time to create a container image with a specific label as we see in Figure 1:

Figure 1. An example of the LABEL instruction being used

For best practices on using container labels, one can refer to the Open Container Initiative (OCI) image-spec guidelines on GitHub.

Let’s think about these container labels. Will these container labels be considered a secret? They are supposed to be just metadata about the object eventually. For example, if a container image is publicly available for anyone to pull, the metadata specified in the “LABEL” instruction is expected or assumed to not contain secrets. Our candid conversation with ChatGPT also confirms our assumption, as shown in Figure 2:

Figure 2. Even AI thinks (pun intended) that using container labels to store secrets or credentials is unadvisable.

Fetching metrics from /metrics

In our last article, we shared how environment variables from running containers could get exposed in the “/metrics” endpoint (Figure 4). To get the metrics in a much better readable format from an exposed “/metrics” endpoint, one can use an open-source tool from the Prometheus developers called “prom2json.” This tool is used to scrape Prometheus metrics and convert them to JSON files. To fetch the container labels from the “/metrics” endpoint, one can use the below command in Figure 3:

Figure 3. Command used to fetch container labels

Figure 4. Example of container labels exposed on the “/metrics” endpoint

Now that we have a definite way of querying container labels from the “/metrics” endpoint on exposed cAdvisor instances, we were interested to see what these container labels contained in publicly accessible cAdvisor instances. We noticed certain container labels beginning with “container_label_traefik_” that contained what seemed to be usernames and password hashes.

Enter Traefik

To better understand the container labels beginning with "container_label_traefik_”, we need to understand what Traefik proxy is. Traefik proxy is an open-source, cloud-native application proxy that can function as an ingress controller, reverse proxy, web application firewall, and load balancer. As we see in the official documentation, it uses service discovery to dynamically configure routing. It supports all major protocols such as HTTP/2, HTTP/3, TCP, UDP, WebSockets, and gRPC, leveraging middleware for load balancing, rate limiting, circuit breakers, mirroring, authentication, and more. Traefik proxy also supports SSL termination and can be used to generate certificates automatically using providers like Let’s Encrypt.

With native support for services and platforms like Kubernetes, Docker Swarm, etcd, Azure Service Fabric, Amazon Elastic Container Service (ECS), Azure Kubernetes Service (AKS), Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and OpenShift, Traefik proxy is widely used and quite popular with over 48,000 stars on GitHub. Traefik’s extensive features and capabilities make it the comprehensive gateway to diverse sets of applications.

Traefik components

Before we create a sample deployment with Traefik proxy, let’s understand its different components and corresponding functions:

1. Providers: Supply routing information to Traefik via APIs to dynamically update routes
2. EntryPoints: Network entry configurations that contain port and protocol information
3. Routers: Connects incoming requests to the service that can handle them
4. Middleware: Process requests or responses before they are sent to the services from the routers
5. Services: Configures how the request or response reaches the actual handler for the service

Of these components, EntryPoints and providers are static in nature; that is, they are needed only at startup and don’t change after Traefik proxy starts up. The remaining components process and route incoming requests and are completely dynamic in nature.

Figure 5. Traefik intercepts incoming requests and processes them before sending to a service
Source: Traefik

In Figure 5, Traefik proxy routes the incoming requests to the back-end services based on the URLs /path1, /path2, or /path3. The routes are dynamically fetched from providers by Traefik proxy. These providers can be orchestrators, container engines like Docker, cloud providers, or key-value stores. If there are any changes detected in the routes from these providers, Traefik dynamically updates the routes.

The configurations that decide whether the incoming request will be acknowledged or not are defined as rules that process the requests and match them based on host, header, URL values, and so on. If the rule is validated, the router is activated which in turn calls the middleware. The middleware processes the incoming request before the request is forwarded to the service. Figure 6 shows the list of rule matchers.

Figure 6. List of rule matchers and their functionality

Particularly, we are interested in how the configuration discovery works for Docker containers (Figure 7), as described in the documentation. While using Docker, Swarm, or Amazon ECS as a provider, Traefik proxy can dynamically populate the routing configuration by either using container labels or by using dynamic configuration specified as files. For docker-compose, Traefik proxy fetches the routing configuration from the “labels” directive from the “services” objects. To get dynamic configuration from the Docker engine, Traefik Proxy requires access to the docker socket.

Figure 7. Traefik routing incoming requests before sending to the desired container
Source: Traefik

Here’s an example of Traefik proxy using Docker as a provider and an EntryPoint named “web” listening on port 80. The flow (Figure 8) is as follows:

1. The proxy fetches the service routes from the Docker provider.
2. It listens on port 80 and uses the EntryPoint named “web”.
3. The container labels for the “nginx” service help update the proxy router’s configuration automatically. Hence, all incoming requests on port 80 are sent to the “nginx’” service.
4. This is followed by the router rule that matches the value of the “Host” header to be “nswashere.local”. If it is true, then the request is successfully received by the “nginx” service:

Figure 8. Traefik Proxy configuration to allow nginx access with a custom host header value

When we create the two services “reverse-proxy’” and “nginx”, we try to access the nginx container via the reverse proxy service, listening on port 80. Note that we haven’t exposed any port on the nginx container. Without setting the host header, we receive a “404 page not found” in the response. As expected, we see the nginx welcome page only when the incoming request has the host header value set to “nswashere.local” (Figure 9).

Figure 9. Incoming request validated only when the host header matches “nswashere.local”

Since we specified the Traefik router configuration for the “nginx” service in the labels, we see them in Figure 10 by inspecting the container labels using the Docker CLI.

Figure 10. Traefik configuration in the labels of the running nginx container

Adding authentication

Using middleware, one can modify, process, or tweak requests before sending pieces of middleware to the service. One can add authentication, URL prefixes and do much more. The list of available HTTP middleware can be found in the documentation. Now we examine a piece of middleware that adds basic authentication (RFC 2617) called BasicAuth.

The BasicAuth middleware allows authorized users to access services. To use BasicAuth in Traefik proxy deployments with the Docker provider, username and password information is required. The password must be hashed using MD5, SHA1 or BCrypt. This information can be specified in the following ways:

1. The “users” option. The username and hashed password are specified in the container label directly.
2. The “usersFiles” option. The username and hashed password are stored in an external file and the path is specified.

Figure 11 shows an example leveraging the BasicAuth middleware:

Figure 11. Traefik proxy configuration to allow nginx access with a custom host header value and authentication

Figure 12. Incoming request validated only when the host header matches “nswashere.local” and request is authorized

As we observe in Figure 12, the nginx welcome page is only accessible when the host header is set to “nswashere.local” and the credentials “test:test” are supplied. Like our previous observation, we see that the container labels contain the username and password hash that was specified for the Traefik proxy to pick up dynamically from the nginx service.

Figure 13. Traefik configuration in the labels of the running nginx container

To recap, while specifying the Traefik configuration where the provider is Docker, we used container labels for Traefik proxy to dynamically discover the service and route the requests based on the host header matcher and the BasicAuth middleware. This configuration information can also be fetched from the labels of the running container using “docker inspect”. This is an obvious but interesting observation that container labels for the target service contain Traefik rule configuration.

Enter cAdvisor

In a default configuration, cAdvisor exposes container labels as Prometheus metrics. Since this is a default configuration, we were interested to see whether there were real deployments that had Traefik proxy and cAdvisor running with cAdvisor metrics exposed. It’s worth noting that the cAdvisor metrics endpoint cannot be protected natively using cAdvisor. To get started with how a vulnerable configuration would look like, Figure 14 shows a sample docker-compose YAML for reference:

Figure 14. Docker Compose YAML for observing Traefik configuration being logged as metrics in cAdvisor

The preceding docker-compose YAML creates an additional cAdvisor service that monitors the reverse-proxy and nginx containers. The dashboard is exposed on port 8080. While inspecting the metrics endpoint, we can see the container labels visible (Figure 15):

Figure 15. Container labels from cAdvisor containing Traefik routing configuration

ITW exposure

We came across 16,374 unique container label key-value pairs related to Traefik proxy. The container labels contained the following:

1. Middleware configurations:

• BasicAuth: username and hash of the password (BCrypt and MD5)
• Headers: content security policy values
• IPWhiteList: sourceRange values
• RedirectRegex: regex matching on URL

Figure 16. Amounts of unique pieces of middleware

2. Router rules using following matchers:

• Host
• HostSNI
• PathPrefix
• Hostregexp
• Path

Figure 17. Amounts of matchers

The information could be leveraged by an unauthorized user to access the endpoints that were being protected with Traefik proxy. From the BasicAuth middleware configuration, we found 117 unique hashes, of which we could crack 25 hashes using publicly available wordlists, indicating weak passwords being used by end users. The router rules revealed information about valid values of host headers, the regex processing the incoming request, and the back-end service where the request is finally received.

Exploit scenarios

Since Traefik proxy configuration is exposed due to how cAdvisor exposes container labels in metrics, this can be leveraged by attackers to completely bypass Traefik proxy and gain unauthorized access to the services protected by Traefik in the following manner:

1. The attacker could fetch container labels from the “/metrics” endpoint of an exposed cAdvisor instance. Since the “/metrics” endpoint is enabled by default, this cannot be protected using cAdvisor natively and exposes all the container labels for all running containers.
2. From the container labels, attackers could figure out the target service that is being protected by Traefik proxy. For instance, they could use the “Host” header matcher or the BasicAuth middleware configuration containing username and password hashes, as they are stored as container labels. Attackers could attempt to crack these hashes offline.
3. Using the cracked credential pair and/or by abusing the reverse proxy configuration, an attacker could completely bypass the Traefik proxy and access the target services that are being protected.

Apart from Traefik configurations, we came across container labels with Amazon ECS metadata that reveals the name of the ECS container, ECS cluster, and ECS Task ARNs. This information can equip an unauthorized user with information about the AWS environment such as AWS account IDs. Although AWS account IDs are not considered as a secret (as per their official documentation), there have been insightful discussions in the research community that shed light on why they might be secrets, as they could aid in performing reconnaissance of an environment.

Defensive measures

It might seem that container labels are the only way to move forward with Docker, ECS, and Swarm providers. However, there is a secure way of deploying Traefik without using container labels. Figure 18 provides an example of how one can specify dynamic configurations in files instead of container labels.

Figure 18. Mounting configuration files as volume mounts on the Traefik container

We specify and mount the static (“./traefik.yml”) and dynamic (“./dynamic.yml”) configurations used by Traefik in files. Traefik configuration can be passed using the flag “—configFile” and by default, it looks for configuration in the following sequence:

1. /etc/traefik/
2. $XDG_CONFIG_HOME/
3. $HOME/.config/
4. . (the current working directory)

The static configuration will contain the EntryPoint, followed by the file provider, as shown in Figure 19:

Figure 19. EntryPoint and provider defined in the static configuration file

Finally, the dynamic configuration contains the router, middleware, and services defined (Figure 20):

Figure 20. Dynamic configuration defined in a file

The dynamic configuration is identical to the configuration we described earlier in Figure 12. Now, when we try to view the container labels using cAdvisor’s “/metrics” endpoint, we observe that the Traefik configuration is not visible even when we still see all the container labels.

When we think of defensive measures to safeguard from such unintended exposures, defenders can do the following:

1. Does your metrics endpoint really need to be publicly accessible? If not, protect them.
2. Threat model your edge routers by thoroughly examining the features and the risks posed. For instance, use “usersFile” instead of “users” configuration option in BasicAuth middleware to avoid leakage of credential hashes in case the container labels are leaked by components like cAdvisor. Reading a file off the disk when compared to viewing metrics makes exploitation difficult for attackers.
3. Monitor public exposure of your environment continuously to avoid such unintended exposures.
4. Proactively scan your codebases (docker-compose YAMLs, Kubernetes object YAMLs, and similar templates) for instances of providers where Traefik relies on container labels such as Amazon ECS, Docker, and Swarm configurations being used with cAdvisor. Avoid using container labels for storing credentials or sensitive information and use file providers instead.
5. The cAdvisor deployment can be secured by using TLS and the BasicAuth middleware and preventing exposure of cAdvisor to the public.
6. To prevent cAdvisor from inspecting all container labels, one can use the ‘--store_container_labels” or create an allowlist of container labels and use “--whitelisted_container_labels” options.
7. Prevent cAdvisor dashboards from being exposed to the public. We detail how and what an attacker could leak from an exposed cAdvisor instance in our previous article.

Conclusion

On its own, Traefix proxy comes pretty much secure right out of the box. However, because of the features and design decisions on how it can use container labels for Docker, ECS, and Swarm providers, things get blurry when other components like cAdvisor come into play. We greatly value Traefik’s insights on the findings of this research, which prompted its proposal of additional security guidelines that have since been integrated in this article. Upon review, Traefik concurs that the use of Docker labels is not a safe way of storing sensitive information such as credentials, underscoring our shared security concerns. Although labels can still be conveniently used for storing routing rules, Traefik recommends more secure means of storing confidential data, like using files or secrets.

In this article, we observe that due to interdependency and potential disconnect in how certain features are used versus how they are meant to be used, such silent discrepancies eventually help attackers win and make things difficult for defenders. Additionally, when there are multiple services being used, the features these services provide can turn dangerous if they are not taken care of. Using container labels to store metadata about an object is reasonable; however, processing them for a functionality can go unnoticed, as container labels are not quite visible to a user upfront (unless users inspect the object’s labels manually). Securing our applications requires a deep understanding of the complex relationships between different components and services. It is important that we are aware of public exposure to components like cAdvisor and the risks posed while using it with applications.