336K Prometheus Instances Exposed to DoS, ‘Repojacking’

Reseachers have discovered hundreds of thousands of servers running Prometheus open source monitoring software on the open Web are exposing passwords, tokens, and opportunities for denial of service (DoS) and remote code execution.

As a leader among open source observability tools, Prometheus is used widely by organizations to monitor the performance of their applications and cloud infrastructure. But it comes with a catch: As noted in its documentation, “It is presumed that untrusted users have access to the Prometheus HTTP endpoint and logs. They have access to all time series information contained in the database, plus a variety of operational/debugging information.”

Apparently, a whole lot of users either aren’t aware of the ways in which Prometheus is exposed by default, or don’t realize the value of the data that’s exposed along the way. Using Shodan, researchers from Aqua Nautilus discovered more than 40,000 exposed Prometheus servers, and more than 296,000 exposed “exporters,” which the program uses to collect data from monitored endpoints. The researchers found sensitive data in those servers and exporters, and opportunities for “repojacking” and DoS attacks.

What Prometheus Exposes

On first impression, the data Prometheus collects might seem rather bland: application performance metrics, metrics associated with particular cloud tools, CPU, memory, and disk usage, for example.

“We think that it’s only statistics — it’s only information about the health of the system. That’s the problem,” says Assaf Morag, director of threat intelligence at Aqua Nautilus. Probing the data from the perspective of an attacker reveals all kinds of information that could lubricate cyberattacks.

“We noticed that we can actually see plaintext passwords and tokens, and API addresses of internal locations that should be kept hidden,” Morag says. For example, he found one exposed and unauthenticated instance of Prometheus belonging to Skoda Auto, the Czech automobile manufacturer, which revealed some of the company’s subdomains, and Docker registries and images.

Besides exposing secrets, open Web Prometheus servers and exporters also carry a risk of DoS. There’s the ‘/debug/pprof’ endpoint, for example, which helps users profile remote hosts, and is enabled by default by most Prometheus components. In their testing, the researchers demonstrated that they could overload the endpoint to disrupt communications or outright crash Amazon Web Services Elastic Compute Cloud (AWS EC2) instances or Kubernetes pods.

“The result was conclusive: We ended up stopping virtual machines each time we ran our script,” Morag reports. To drive home the significance of such an attack scenario, he jokes, “I read somewhere that Kubernetes clusters run in fighter jets. I don’t think that they are exposed to the Internet, but [it goes to show] we run Kubernetes in lots of places today.”

Repojacking Opportunities in Prometheus

Users can protect their Prometheus servers and exporters by taking them offline, or at least adding a layer of authentication to keep out prying eyes. And, of course, there are tools designed to mitigate DoS risks.

Less easily solved is a third issue in the platform: Several of its exporters were found vulnerable to repojacking attacks.

The opportunity for repojacking can occur whenever a developer changes or deletes their account on GitHub and doesn’t perform a namespace retirement. Simply, an attacker registers the developer’s old username, then plants malware under the same title as the developer’s old, legitimate projects. Then any projects that reference this repository but aren’t updated with the correct redirect link can end up ingesting the malicious copycat.

Prometheus’ official documentation referenced several exporters associated with freely claimable usernames, meaning that any attacker could have stepped in and taken advantage to perform remote code execution. Aqua Nautilus reported the issue to Prometheus, and it has since been addressed.

Repojacking opportunities are likely far more widespread than is realized, Morag emphasizes, so organizations need to be monitoring any discrepancies between the projects they rely on and the links they follow to access them. “It’s not that difficult,” he says. “But if you’re doing it for millions of open source projects, that’s where the problem starts. If you use an automated [scanning tool], you could be safe.”