IoT
VPNFilter Two Years Later: Routers Still Compromised
We look into VPNFilter, an IoT botnet discovered over two years ago, to see why there are still routers infected by the malware and what else can be done to minimise its potential risks.
With the internet of things (IoT) gaining more popularity, common IoT devices such as routers, printers, cameras, and network-attached storage (NAS) devices, are becoming more frequent targets for cybercriminals. Unlike typical operating systems such as Windows and macOS, users are less likely to patch IoT devices. This is because users find the task more difficult and inconvenient since, in comparison, the operating systems of these devices have no auto-update feature and some manufacturers rarely even issue security updates at all. These are the kinds of systems that users log on to once in order to set them up and then never to do so again, unless they encounter a big problem. It also is not rare to find an outdated router — one that has been running for as long as the system has.
As a result, many systems are left wide open to known vulnerabilities, which can lead to successful attacks even years after the first infection. While looking at these types of infections by known malware families, we found that one of the biggest reported malware families was from 2018’s VPNFilter.
VPNFilter is a malware type that affects routers and storage devices by using backdoor accounts and exploits of several known vendors. In May 2018, Cisco Talos released the first report on the malware, which showed how VPNFilter was designed to gain a foothold into networks and look for Modbus traffic. However, it should be noted that this was not the only plug-in but rather merely one of the plug-ins that could be deployed. Modbus is a popular industrial control system (ICS) communication protocol that is specific to these types of systems. While the malware tends to focus on compromising consumer-grade IoT devices, it is also true that consumer devices are often found as part of ICS systems for various reasons. Some reasons include the fact that these devices are easy to deploy, provide remote vendor access, or simply because they are mistakenly added by the administrator. That the malware potentially targets control systems could be the reason that the FBI has attributed the first reported attack to the work of nation-state actors.
VPNFilter operates in multiple stages that include initial infection, command-and-control (C&C) communications, and the third stage, in which the payloads are deployed. These payloads perform the tasks that the malware has been intended to do. This is also where the Modbus portion of the malware is found. The first stage of the malware involves gaining access to specific devices from over 12 vendors. Once this is done, the second stage of the malware involves an attempt to connect to Photobucket, an image-hosting site, to download an image that has the IP address of the C&C server embedded in the GPS coordinates of the exchangeable image file format, also known as Exif. If this fails, then it will try to reach out to the domain toknowall[.]com to download an image with the C&C server also embedded in the file. Finally, if both these two attempts fail, it will open a listener to monitor all incoming packets for a specially crafted TCP packet that would contain the IP address of the C&C server.
While VPNFilter gained considerable attention and became a threat when it was first discovered, this happened back in 2018. This means that several mitigation tactics have already been used to render VPNFilter essentially offline. With domain seizures and every action taken to stop the malware, therefore, it is worth asking why there are still infections out there. The FBI’s statement at the time of the malware’s discovery advised users to restart potentially affected devices to temporarily disrupt the malware. The statement also meant that by restarting the router users would essentially remove any current first-stage and third-stage malware based on the initial findings from Cisco Talos. However, this leaves leftover infections, which is the original listener setup by the first-stage malware.
In our recent paper titled “Worm War: The Botnet Battle for IoT Territory,” we covered the aspect of botnets removing the infections of other botnets and the ease with which these IoT devices can be compromised. At times, it seems that threat actors are the few people who have access to some of these systems and are the ones removing malware types like VPNFilter. It’s worth noting that this kind of access might not be available to all users in the first place. As an example, when the FBI took down part of the botnet’s network infrastructure, they recommended end users to restart their devices. However, many users have routers that were provided by their ISP. Often, this type of end users would therefore not have login access to the router. As a result, without their ISP’s permission, they cannot update the firmware to get the latest vulnerability fixes. As we will discuss here, this is precisely the reason that the FBI worked with Shadowserver to add the extra safeguard of sinkholing the botnet.
Still, we did find examples where vendors published updates and guidance to remediate the infection. Netgear and MikroTik, for instance, have stated that upgrading the firmware would remediate the malware’s effects and prevent further infections.
As of writing, however, we have not encountered any one organisation in the IoT space that has committed to clean up vulnerabilities and infections. Due to the lack of a standard group or mechanism for updates, the owners of these IoT devices often need to manually go to their vendor’s website, download a firmware file, then upload it to the device. Very few seem to have automatic firmware update procedures in place. These observations have spurred us to ask the following questions with regard to the remaining VPNFIlter infections:
- How many victims have updated their router’s firmware?
- How many victims have had their infected routers replaced in the last two years?
- Was VPNFilter one of the infections that was being removed by other malware actors?
- Are there any infected devices still out there?
Shadowserver Sinkhole
The Shadowserver Foundation is a nonprofit security organisation whose mission is to “... make the Internet more secure by bringing to light vulnerabilities, malicious activity and emerging threats." Recently, Trend Micro has partnered with Shadowserver to provide funding for their cause over the next three years. Given this partnership and the work that Cisco Talos, the FBI, and the US Department of Justice (with the support of Shadowserver) had already done in the past to sinkhole VPNFilter’s second domain (toknowall[.]com), we wanted to work with Shadowserver to collect data from any available stats that they might have, such as how many infections are still out there. We also wanted to suggest ways to clean up any leftover VPNFilter infections. This is of particular importance, as cleaning these devices is far from trivial for end users. This is precisely the reason that it was so important for these organisations to carry out the sinkhole in the first place.
First, we needed to get a better understanding of the malware, find out how it worked, and from there determine if anything else could be added to the takedown processes that were already being done.
![Figure 2. The number of requests since the sinkhole of toknowall[.]com started, as provided by Shadowserver](/content/dam/trendmicro/global/en/research/21/a/vpnfilter-two-years-later--routers-still-compromised/figure_2_vpnfilter_two_years_later.jpg)
As shown in the image above, when Shadowserver started the sinkhole, they saw an initial spike of over 14,000 networks infected in the first two months; over time, that has been reduced to 5,447. This shows that even after over two years, there is still a sizeable number of infections left. Most notably, at this rate, the infections will likely still be around for years to come, until perhaps these devices are physically swapped out — a common trend in IoT botnets. Not only does this tie to our main point that IoT botnets are to some degree nearly “uncleanable,” it also makes VPNFilter a botnet that is ripe for taking over by another threat actor for them to utilise, as we will explain later on.
Technical details
As stated in the original publication by Cisco Talos, the first stage seeks to download the second-stage malware from an IP address received using the following scheme:
- Get the second-stage malware from an image file uploaded to Photobucket[.]com.
- If the above fails, it tries to download the image from the domain toknowall[.]com.
- If both fail, it starts listening to all TCP packets on the affected device to receive the IP address from a specific TCP packet sent by the attacker.
This logic is presented in Figure 4.
We wanted to verify the effectivity of the solutions against the first and second phases of the first stage. Mainly, this means that to see the effectivity of the sinkhole, it is necessary to check for victims that are still infected. In case there are still infected victims, it is necessary to try to think of a solution to permanently clean up these devices.
One key thing that we discovered from the outset was that even though the first phase (the image in Photobucket) was already taken down and the second phase (alternative domain) sinkholed, unless the malware receives a valid Image file from the sinkhole, it would still enter its third phase where attackers could potentially regain control. Ironically, while this might seem like bad news, this showed us an opening for a solution, as we will discuss later on. Here we detail our findings on the first and second phases for this initial stage.
Photobucket[.]com C&C
All the URLs pointing to Photobucket[.]com were already taken down, which means that the pictures hosted there were also removed. Therefore, the only two possible options for currently infected devices would be to reach out to toknowall[.]com or to start listening to TCP packets while expecting a specially crafted packet containing the IP address of the second stage server. For our next step, we verified how many real victim devices were left communicating with the sinkhole and how many of these had already moved to listening mode.
toknowall[.]com C&C
We then verified if it would be possible to inject our own image with a C&C IP address controlled by us, to see how many hosts would respond. For this, we had to dig deeper in the malware logic to extract the C&C IP address from the image downloaded from either Photobucket[.]com or toknowall[.]com. The algorithm calculates each of the four octets of an IPv4 address out of the GPS information contained in the Exif header of the downloaded image file. The two values used are GPSLatitude and GPSLongitude. Each of these consists of three values (degrees, minutes, and seconds) stored as rationals (64-bit numbers split into two 32-bit integers). The first 32-bit number is called a numerator and the second is called a denominator. The theory behind this is better explained Soufiane Tahiri’s book, “Mastering Mobile Forensics.” In the case of VPNFilter, these coordinates are used like this:
As can be seen from Figure 5, the denominator part of all coordinates is ignored by the malware. The numerator, however, is used to generate the values of all octets.
We wrote a programme to replace these bytes in the image that we created. The code that does the job can be seen in Figure 6.
In the preceding code, the first coordinate is located at position 4500 (decimal) in the image. We could have parsed the whole Exif header, but we needed something quicker. Also, in lines 48 and 51, we set the numerator part of the degree value for both latitude and longitude to zero so that we could leverage the negative integer values in C to write the exact bytes that we needed.
By using our programme, we were able to inject an IP address that points back to the sinkhole owned by Shadowserver to see how many infected devices are still actively looking for a second stage download server and more importantly, to prevent the still-infected networks from moving to stage one’s third phase (listening to all TCP packets) in the future.
Figure 7 shows the resulting image that we used. This image was then supplied to Shadowserver who are in the process of deploying it. While this is being done, we went to look for victims in a specific stage of the infection.
Listening mode
If the malware does not get a valid image from previous stages, it will then enter into listening mode. This allows the attackers to regain control over infected victims if the sinkholed alternative domain does not serve a valid image.
Figure 8 shows the source code that makes this third option possible.
The above code lets the malware listen to all packets and expects one TCP packet with the SYN flag that is at least 8-bytes long. The magic sequence expected is a hexadecimal 0C 15 22 2b.
After verifying the logic, our next step was to generate the packet. The easiest way that we could think of doing this was to use hping3 since this tool is capable of sending any type of network packet. We used the code seen in Figure 10.
We then asked Shadowserver to scan the infected devices that they have logged throughout the years to see how many would respond. Here are the results for the unique 5,447 devices that were found still connecting to the sinkhole on a single day. It is important to remember that because these are routers and other similar types of devices, this number also represents thousands of infected networks, not simply individual machines. This means that the reach and visibility for attackers with a botnet like this can be substantial. Also, the number of infected hosts might be higher than what Shadowserver sees on their sinkhole, as the domain toknowall[.]com is a well-known C&C and can be blocked at the DNS level by DNS providers or other security products.
Here we summarise our main findings and show the geolocation data.
- 1,801 (33.1%) networks responded to our packet sent by hping3 on TCP port 80.
- 363 (6.7%) networks reached back to our sinkhole on port TCP 443.
Infections seem to be led by USA, followed by Russia, according to geolocation data from the hosts that reached back to our sinkholed server.
Although only 363 networks connected back to our sinkhole, we cannot assume that the 1,801 networks that gave us an initial positive response are clean. They might still be infected by VPNFilter, but the connection to our sinkhole could have been blocked if they are behind a firewall. These 363 networks can be taken over by anyone with an understanding of how the malware works. From a technical perspective, there was nothing to prevent the takeover of these devices. Additionally, at any point in time the original actors could take control of the devices that are already infected.
Conclusion and recommendations
Even though solutions have been deployed to lower the effectivity of VPNFilter (which has been known for over two years), for end users restarting is still not enough to protect their devices from reinfection. To reiterate, this is why the initial sinkhole was so important. While it's not likely to have the malicious actor still on infected systems, the malware can still have a potential negative impact. With just a bit of understanding, another malicious actor can have the botnet reactivated. This is exactly why we worked with Shadowserver to upgrade the second stage sinkhole with a valid image to prevent the malware from moving forward to its listening mode, which occurs in the third phase of the first stage.
A firmware update would also remediate this problem. However, as mentioned earlier, firmware updates become problematic and in most cases, are not as easy as in PC ecosystems. The major hurdles involve verifying that the firmware file is legitimate and understanding how to apply the updates to the system. This would be assuming, however, that users even have access to the router to perform upgrades in the first place, as well as that their device’s vendor has an upgrade available for their model. In many cases, someone else was responsible for setting up the device for the user, such as the company that they bought it from, or their ISP. To compound all of this, getting a new router might also be problematic if users don’t own the router, so they might have to wait for their ISP to provide a new router.
These limitations and challenges that are faced by the users on their side bring us to where we are and why the numbers of VPNFilter infections are still at the levels that are being seen on the internet. Moving forward, the approach would be to keep the sinkhole running with the fixes that we have implemented. This limits the ability of this specific malware to become active again. It also gives time for devices to naturally go offline with a normal life span. The need for such an approach emphasises the importance for the existence and operations of organisations like Shadowserver, as they serve as custodians of the internet. This is precisely why Trend Micro made the commitment earlier this year to support Shadowserver.
Ordinary end users can also do their part in making sure that the IoT remains safe from the hands of cybercriminals thereby preventing the success of their future campaigns. One of the best ways to minimise potential infection from IoT-malware is to limit the number of exposures one has to the internet. This means, in most cases, disabling any remote management options in the configuration. Also, disabling Universal Plug-and-Play (UPnP) if the option is available, as this setting might expose services to the internet without the awareness of their users. Updating devices as regularly as needed to the latest firmware also provides the latest security features and bug fixes that might be used in attacks against systems.
Applying general security practices can also minimise the chances of IoT-malware infecting routers and other devices. At present, given the prolonged work-from-home (WFH) setups, users should reexamine their current security measures as home devices and systems now have a heavier influence on corporate networks.
In our paper “Worm War: The Botnet Battle for IoT Territory,” we show how these devices are both actively attacked and are relatively simple to compromise. We would now like to extend our messaging to say that these router infections are never cleaned and can be active even years after a botnet has supposedly been offline. Although we have used VPNFilter to elaborate, it’s important to emphasise that this is true of any other IoT botnet. Indeed, this will keep happening unless users take care of their router frequently and router manufacturers start adding auto-update features to their devices’ operating systems. In the meantime, the security industry faces an uphill battle against infections that cease to be a threat only when the router is finally replaced.