A while back, a researcher had reported that the Instance Metadata feature in public cloud platforms makes them a very effective exploitation target. Essentially, an instance’s metadata can be queried via an API to obtain access credentials to the public cloud environment by any process running on the instance. This appears to be an emerging threat vector for account compromises, as we discuss in the RedLock CSI Trends Report - May 2018 edition.
I sat down with the RedLock Cloud Security Intelligence (CSI) team to dig a little bit into this topic to get their thoughts and hear some of their recent discoveries surrounding this. The team reported that they had found a couple of new ways this feature can be exploited which we will detail later in the blog.
This blog provides some background on the Instance Metadata feature, takes a deep dive into a couple of new ways that the RedLock CSI team has discovered to exploit it, and provides some recommendations on how to defend against it.
Instance Metadata Primer
AWS Identity and Access Management (IAM), MS Azure Managed Service Identity (MSI), and Google Cloud Cloud IAM are features that allow for the creation of users and grant/deny permissions. They simplify the task of creating and distributing credentials and are popular features with developers.
When a virtual machine is created, one can query the instance metadata API to find out information such as name, security group, and other metadata. It’s not necessarily bad - and is actually a very important, useful feature - virtually all third party tools make use of this feature. IAM role credentials can also be exposed to applications via the instance metadata API. By using a simple CURL command to http://169.254.169.254/ IAM role credentials are freely available for programs to obtain.
As with any cloud service, such powerful features also leave potential for abuse if not utilized by the end-user per the cloud providers’ recommendations. Credential management has always been a big headache and users know it’s bad practice to leave credentials lying around in files, environment variables, etc. AWS was the first to offer the convenience of using this API to obtain these credentials, and subsequently provided a layer of protection by rotating these keys every six hours. It also warns about the risk in its documentation about instance metadata. However, it is up to organizations to understand the risk and take steps to defend against it.
Figure 1: Amazon Warning Regarding “Retrieving Security Credentials from Instance Metadata”
RedLock’s CSI team came up with a couple of ways hackers might exploit this API. We are not aware if these methods have been exploited yet, but similar to the Spectre/Meltdown vulnerabilities the potential impact has a very large blast radius in public cloud.
Exploit#1: Vulnerable Reverse Proxies
Reverse proxies are common in public cloud environments, especially amongst organizations that are moving on-premise applications to the cloud. The RedLock CSI team had a hypothesis that some reverse proxies in AWS, MS Azure, and Google Cloud environments are set up such that anyone can set the host header to call the instance metadata API and obtain credentials as described earlier. To test this hypothesis, the team wrote a little script which finds such proxies and sure enough, they found dozens of such vulnerable servers.
Figure 2: Example of a bad NGINX proxy configuration
In the above example, the NGINX proxy will forward the request to the server represented by the $host variable. This variable is controlled by the user and if the address of the instance metadata API endpoint is provided, access credentials are returned as demonstrated below.
Figure 3: Demonstration of Accessing Access Credentials from a Reverse Proxy
Exploit#2: Malicious Docker Images
The second exploitation method is even scarier and potentially more far-reaching.
In the old world, developers had to worry about dependencies and compatibility across different operating systems versions while building applications. It was basically a tedious distraction that took away from creative, innovative development time.
Docker solved this problem very nicely. It is an open source tool that can package an application and its dependencies in a virtual container that can run on any Linux server. Developers share Docker images with the public using stores such as Docker Hub. There are so many useful services and helpful images to save developers’ time and this in turn allows them to focus on their areas of expertise and value add. As a result, developers access, download and extend these images all the time, without blinking an eye.
Now suppose some crafty developer creates a super helpful, free-to-download docker image called “X” and posts on Docker Hub along with millions of other popular resources. Then one fine day, after thousands or millions of downloads of the free service have been deployed, what if this malicious developer modifies and uploads an updated version of X (this happens all the time and others pull the latest version or make “calls” to it) now containing the nefarious command: “ONBBUILD -<malcious_script.sh>”. Utilizing the instance metadata API, every application built upon the “X” docker image will run this script (malcious_script.sh) unbeknownst to the dependent program and will request IAM role credentials. And here lies the risk.
Figure 4: Developer Extended Base Docker Image with Malicious OnBuild Script Which is Re-Used by Acme Corp
Most developers have long been aware of the potential dangers of malware creeping its way into third party services that are being packaged and utilized in their applications. But frankly, nobody really cared. The thinking has been that since these programs are running in a container, that even if this container was compromised, things would be fine since any potential attack would be limited in scope. However, leaking access credentials is a whole different type of risk altogether; the scope of the attack broadens from just the compromised container to now providing hackers with access to an entire public cloud account for them to perform all sorts of illicit activities as discussed earlier.
Ways to Defend Against Such Compromises
￼There are so many theoretical attack vectors discovered and research is published about them all the time. Frankly, until something headline-grabbing happens, nobody seems to care. Such was the case with Spectre/Meltdown; while the theoretical vulnerabilities were documented decades ago, nobody could explain exactly how they could be exploited. Similarly, it is still to be determined if any organizations have suffered a breach where the instance metadata API was used as a trojan horse. That notwithstanding, organizations should consider the following recommendations to defend against such compromises:
- Monitor User Behavior: Operate under the assumption that access credentials will be leaked one way or another, and implement solutions in your public cloud environments to monitor for potential account compromises. The RedLock Cloud 360 platform baselines “normal” behavior for each user in an environment and alerts on deviations as illustrated below.
- Monitor Network Activity: Again, operate under the assumption that access credentials will be leaked, and implement measures to detect suspicious network activity such as cryptojacking. The RedLock Cloud 360 platform monitors north-south as well as east-west traffic in public cloud environments to detect nefarious activity as illustrated below.
- IP Tables: Use this feature to limit access to the instance metadata API to privileged processes only (principle of least privilege access).
- Docker Image Audit: Audit third party docker images being used, and review the ONBUILD instructions
- Google Cloud Platform: Use the metadata concealment feature in Google Kubernetes Engine to prevent user pods from accessing Kubelet credentials and VM instance information. Specifically, the feature protects access to kube-env (which contains Kubelet credentials) and the VM's instance identity token.
Figure 5: RedLock Cloud 360 Platform Detects a Potential Account Compromise
Figure 6: RedLock Cloud 360 Platform Detects a Cryptojacking Activity
Get the NEW Cloud Security Trends - May 2018 - Anniversary Edition
This edition of RedLock’s Cloud Security Trends marks the report’s one year anniversary, and it’s been a sobering year in terms of public cloud breaches, disclosures and attacks. Download the latest Cloud Security Trends - May 2018 report to get 14 tips to fortify your public cloud environment.