So, read the title! Someone managed to exploit my site and install some malware. Ok, ok.. not something that I’d want to be sharing publicly, you’d think, but I disagree. Granted, I might not be proud of the fact that my site had exploitable code in it, but I also think that it’s important to tell folks – who host sites as a hobby or for a living – that it’s possible for things like this to happen. And so, now I share with you just how my site got compromised, and how I found out about it, and how I dealt with it.
Today I was doing some maintenance on my server cluster. I was installing some security software on one of my auxiliary servers and I was testing out a scan script.
Now, a contextual thing to keep in mind is that this particular auxiliary server is my failover for all of the sites I host. If I were expecting my primary content server to go down for whatever reason, I can failover to this particular secondary just by running a simple script. I have scheduled cron jobs that run every day, and they sync up all of the content that I have on my primary content server to this secondary server so that everything matches.
Anyway, I was testing out my scan script. Lo and behold, the scan turns up a single malicious file that ought not to be there. My security admin training from HostGator suddenly kicks in, and I check out the file and look at its contents. Sure enough, it’s malicious! Well! If the file is on my secondary server, it has to be on the primary server from which it’s synced, so I went over there and sure enough, there it was. I looked through my scan logs and found evidence there as well.
So, my site got compromised. Now it’s time where I go over how I dealt with it. This is a bit advanced, I think, and I’m not tailoring this precisely for new users, but it might help some folks to find their way if they need to.
First thing that you do in a situation like this, is you run the ‘stat’ command on the malicious file that your scanner found. I did this, and here is what I got.
File: `ajax-upload.php’
Size: 177962 Blocks: 352 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 134650 Links: 1
Access: (0644/-rw-r–r–) Uid: ( 500/ nope) Gid: ( 500/ nope)
Access: 2015-03-22 00:05:54.445132850 -0500
Modify: 2014-10-16 13:03:52.000000000 -0500
Change: 2015-02-04 21:10:28.178217693 -0600
What does this output say? It says that this file was last Accessed on the web or on the local system at 2015-03-22 00:05:54.445132850 -0500. This is the month, day, year, second, and timezone of the server (-0500 is 5 hours behind GMT). It also says that the file was last Modified on 2014-10-16 13:03:52.000000000 -0500, and last Changed on 2015-02-04 21:10:28.178217693 -0600.
What’s the difference between Modify and Change? Aren’t they the same? Well, yes and no. In this particular context, Modify refers to the last time that the file contents were changed, and Change refers to the last time that the meta data of the file changed (permissions, location, and such). So, essentially, the last time that the file contents were modified was on October 16, 2014, long before it ever made it onto my system. That’s probably when the programmer last modified the file before some script kiddy decided to make it a part of their h4ck0r bot. The Change date is what we want to pay attention to then, since this indicates when this file’s metadata (on my system) last changed.
I go through and, using the change date as a reference, look through my http access logs. I find the following:
91.217.82.160 – – [04/Feb/2015:21:10:19 -0600] “GET /wp-admin/ajax-upload.php HTTP/1.0” 404 2514 “http://google.us” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”
91.217.82.160 – – [04/Feb/2015:21:10:24 -0600] “POST /wp-admin/admin-ajax.php?action=settings_upload&page=pagelines&pageaction=import&imported=true HTTP/1.0” 200 27 “https://prannon.net” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”
91.217.82.160 – – [04/Feb/2015:21:10:26 -0600] “GET /wp-admin/ajax-upload.php HTTP/1.0” 200 114 “http://google.us” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”
91.217.82.160 – – [04/Feb/2015:21:10:27 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 2654 “http://google.us” “Mozilla/4.0 (compatible; MSIE 8.0.1; Windows NT 6.1)”
91.217.82.160 – – [04/Feb/2015:21:10:28 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 3486 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
91.217.82.160 – – [04/Feb/2015:21:10:29 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 5522 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
91.217.82.160 – – [04/Feb/2015:21:10:33 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 16233 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
91.217.82.160 – – [04/Feb/2015:21:10:46 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 23382 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
91.217.82.160 – – [04/Feb/2015:21:10:48 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 3583 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
Note how the time lines up in these log entries and the Change time in the ‘stat’ that I ran above? Since the logs line up with the time stamps, it means that this is where the file came from. If I follow this set of logs back to the original spot, I can see exactly what happened.
91.217.82.160 – – [04/Feb/2015:21:10:19 -0600] “GET /wp-admin/ajax-upload.php HTTP/1.0” 404 2514 “http://google.us” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”
The malicious user checked to see if their shell existed. It doesn’t. If you look, you can see the 404 “Not Found” error code in this log entry.
91.217.82.160 – – [04/Feb/2015:21:10:24 -0600] “POST /wp-admin/admin-ajax.php?action=settings_upload&page=pagelines&pageaction=import&imported=true HTTP/1.0″ 200 27 “https://prannon.net” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”
Seeing that their shell didn’t exist, they used a newly published theme exploit against the theme that I use on my site. I actually found some links for this exploit that pretty much confirmed that this is exactly where my malware came from.
http://www.pagelines.com/themes-requiring-updates/
http://blog.sucuri.net/2015/01/security-advisory-vulnerabilities-in-pagelinesplatform-theme-for-wordpress.html
And now I know where the malware came from and how my site was exploited. I can take action. I immediately updated my theme, and then looked through my entire site directory structure for any and all malware that might have been left behind. I used a ‘find’ command to do it.
find -type f -ctime +1 -ctime -90
Since I know when the original compromise happened, and since I just updated my site, I can create a narrow time frame during which the malicious user uploaded or modified files to contain their malicious code. Finding these, I can delete the files or I can clean them of bad code.
And then finally, I took action to make sure that my wp-admin folder was restricted to unknown users. In the past, I’d had .htaccess restrictions in place that were supposed to prevent unknown users from being able to access these sites. If this were working, it would have prevented this sort of compromise from ever happening. Remember from the logs above, the malicious user used a file within wp-admin to perform their exploit.
At some point, though, these restrictions stopped working properly, and… well here we are. Since my .htaccess restrictions aren’t working anymore, I enacted the restrictions at my front end layer instead, nginx. I’m weird! I use two http servers to do my bidding. I have nginx as a front end and apache as a back end. It’s sad that .htaccess doesn’t work 100%, but I can live with having the restriction in place at the nginx layer. Ultimately, however the site is secured, I’m happy.
So, that’s the tale of my rather sudden and random malware event of the day. Hopefully it’s somewhat interesting. Anyone is free to hit me up for questions on this point, and I’ll be glad to help out!