Board index General discussions hacked? question on site generation

Random stuff about serendipity. Discussion, Questions, Paraphernalia.
thagenesis
 
Posts: 4
Joined: Sun Feb 09, 2014 12:07 pm

Postby thagenesis » Sun Feb 09, 2014 12:50 pm

Hi,
I guess my sy9 installation has been hacked but to confirm this I need some information on the internals of s9y. First off the facts. I have my own dedicated server with evrything set up by hand. A few days ago the server went unreachable and only a hard reset via the admin interface of my hoster did the job. did not find anything unusual in the system logs (didn't look in the apache logs though). Acouple days later I opened my blog in the browser and everything looked fine. my last entries were displayed on the start page but then I discovered that clicking on an entry didn't work anymore. I always got an "not found error". I didn't think of a hack at that point but that my installation was broken in some way so I updated it to the most current version and the update itself seemed to work. My entries were still displayed on the start page but I weren't able to access them. so I made a fresh install into a new directory which left me quiete confused. when I accessed it I only saw ancient entries from 2010. first I thought s9y was sorting ascending instead of descending but after logging in and accessing the admin interface I saw that my more current entries were gone. So why did they appear on the front page of my old installation? Is s9y only generating the start page when entries are edited from the admin-interface and the attacker "only" got access to the mysql?I'm going to dig into my mysql backup from last week as soon as there is time which may help clearing some things up but at the moment I'm just confused. I'm not 100% sure which version I had before I updated but the files in my www directory indicate 1.6:

Code: Select all
 # ls -ltr | tail -n 7
-rw-r--r--  1 root   root    5357394 Oct 27  2011 serendipity-1.6.tar.gz
-rw-r--r--  1 root   root    5685683 Jul 26  2013 serendipity-1.7.2.tar.gz
-rw-r--r--  1 root   root    5665896 Aug 28 12:41 serendipity-1.7.3.tar.gz?download
drwxr-xr-x  2 root   root       4096 Feb  7 20:37 tmp
drwxrwxrwx 14 nobody nogroup    4096 Feb  7 20:55 serendipity_old
drwxrwxrwx 14 nobody nogroup    4096 Feb  7 22:04 serendipity
drwxr-xr-x  2 root   root      32768 Feb  9 07:14 logs

Thanks for any hints/help

User avatar
garvinhicking
Core Developer
 
Posts: 29966
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany

Postby garvinhicking » Sun Feb 09, 2014 6:04 pm

Hi!

s9y doesn't store any entries inside files, what you might have seen is only cached files in templates_c/ or maybe files from the simplecache plugin. All entries are stored in MySQL in the serendipity_entries database table.

I would suggest you to download the current files of your installation into one directory. Then download a fresh version of eactly the serendipity version you had installed into anotther directory. Then use a tool like diff/windiff zu compare those two directories and you should see files that might have been changed/hacked. Usually files that get attacked are index.php and .htaccess, as well as any possible index.html or index.tpl files inside the templates_c/ and templates/ directories.

Other than that, it's currently hard to make a guess. Serendipity is pretty a small system, most hackers usually use larger hacks (like against the system kernel, ftp servers, php, apache itself, ssh backdors, exploits of tools like plesk/cpanel, or even against your Client so that they can get FTP/SSH access from your stored passwords). So this is a scenario you should also inspect, because usually if your whole server went down, this isn't causeed by a PHP application like serendipity, but a broader hack against the whole system...

HTH,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

thagenesis
 
Posts: 4
Joined: Sun Feb 09, 2014 12:07 pm

Postby thagenesis » Sun Feb 09, 2014 6:59 pm

Thanks. Caching is a good point. totally forgot that. I think you are probably right and something else is going on. As I wanted to open phpmyadmin to take a look(and I know what you want to say now but phpmyadmin is protected with http basic auth and was updated in december :) ) the connection timed out but I had a ssh session open and I was able to tcpdump a little bit. As a networking guy this is what I usually do first. I caught an ongoing SYN flood:
Code: Select all
16:25:13.517805 IP 88.198.157.194.80 > 5.135.135.42.31520: Flags [S.], seq 3070412371, ack 404543019, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
16:25:13.517819 IP 88.198.157.194.80 > 5.135.135.42.11653: Flags [S.], seq 3076087331, ack 567001033, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
16:25:13.517837 IP 88.198.157.194.80 > 5.135.135.42.58408: Flags [S.], seq 3076577347, ack 1251476323, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
16:25:13.517841 IP 88.198.157.194.80 > 5.135.135.42.39442: Flags [S.], seq 3073451480, ack 2124195077, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
16:25:13.530378 IP 5.135.135.42.38642 > 88.198.157.194.80: Flags [S], seq 2092456200, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
16:25:13.530414 IP 88.198.157.194.80 > 5.135.135.42.38642: Flags [S.], seq 3220843779, ack 2092456201, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
1

Now this can't be some random traffic but it is a directed atteck against my server. when I'm able to access phpmyadmin again I'll dig into the DB and let you know if I find anything. Furthermore the attacked IP88.198.157.194 is a secondary IP. No domain name points to it. It is only used for vpn connections to serve the right certificate All vhosts/domains on the server for legit www traffic are pointing to the main IP.Maybe the crash was also related to a SYN flood. At least some explaination but I still have some work to do

User avatar
kleinerChemiker
Regular
 
Posts: 765
Joined: Tue Oct 17, 2006 2:36 pm
Location: Vienna/Austria

Postby kleinerChemiker » Sun Feb 09, 2014 8:07 pm

if you have problems with attacks, you could try hiding your server behind https://www.cloudflare.com/

thagenesis
 
Posts: 4
Joined: Sun Feb 09, 2014 12:07 pm

Postby thagenesis » Mon Feb 10, 2014 12:48 pm

Warning: the rest of this text will contain barely anything s9y related. if you are not interested just skip it. I just want to write this down in case anyone gets here while googling and maybe garvin wants to know.
I've now put some time into thinking about the problem and here are my conclusions:
This can't be a sophisticated hacker. These don't want to leave obvious traces onn the compromised system but turn it into a c&c server for botnets/a spam relay/anything else which can be sold or used to their advantage. Vandalizing the site won't go unnoticed and only attract the attention of the admin. So if it was a hack it can only be a script kiddie. But then: why only deleting these very specific recent entries and leaving the old stuff intact? this makes no sense in my opinion. And then the next logical step: why only delete stuff and leave the front page intact. I would expect a defacement from a script kiddie so they can prove the successful hack to their friends. There is a saying in medicine: When you hear hoofbeats, think of horses not zebras". I think the horse here might be some kind of DB corruption and I can think of a way how this happened. I remember the root partition of the server being full some time ago. it helps at this point to know that mysql (when using my ISAM format) splits databases over several files as the DB grows. So the current files were probably corrupted because mysql had these open while the partition ran full. I checked the table serendipity_entries and sorted by the timestamp field. the entries from the last 4 weeks (I wrote a couple in december and some in january) are gone indeed. Now I'm trying to get anything from my backups but I'm afraid the rotation is already to old. Due to lack of space I can only keep 3 generations of backup. additionally it takes quite some time to extract the mysql dump files from the large fullbackup archives. regarding the SYN flood: Now I think this is completely unrelated. I configured apache not to listen on this IP anymore but even after this was done and the reachability was restored the flood did not cease. I think this was a backscatter attack. The IP generating these SYN packets was spoofed so my server would send back 3 SYNACKS for every spoofed SYN and therefore congest the connection of the spoofed target. only a small amplification factor of 3 but my server probably wasn't the only one participating in this attack.As a secondary IP was used I suppose a scan of the IP ranges my hoster has been carried out to detect open ports which can be used as reflectors /amplifiers. My first intention of posting here was to make sure no mass-hack of s9y installations is going on or there are untetected exploitable bugs in s9y but I think at this point one can safely say this is not the case, Also I wouldn't be the only one posting here.

User avatar
garvinhicking
Core Developer
 
Posts: 29966
Joined: Tue Sep 16, 2003 9:45 pm
Location: Cologne, Germany

Postby garvinhicking » Mon Feb 10, 2014 3:43 pm

Hi!

Hehe, this reads like a detective story. I agree with your thoughts.. good look on trying to repair and further analyze what's been done, and thanks for sharing.

Regards,
Garvin
# Garvin Hicking (s9y Developer)
# Did I help you? Consider making me happy: http://wishes.garv.in/
# or use my PayPal account "paypal {at} supergarv (dot) de"
# My "other" hobby: http://flickr.garv.in/

thagenesis
 
Posts: 4
Joined: Sun Feb 09, 2014 12:07 pm

Postby thagenesis » Thu May 15, 2014 10:31 am

So I wanted to restore some entries from Browser/Google Cache but I wasn't able to log in. My authors table must have been corrupted too. I wrote a little script to reset my password:
Code: Select all
   

     # cat reset_pw.php
    <?PHP
    $newpass="changeme";
    echo "<h1>s9y password reset</h1>";
    require_once("serendipity_config_local.inc.php");
    //$serendipity['dbName']
    $DB=mysql_connect($serendipity['dbHost'],$serendipity['dbUser'],$serendipity['dbPass']);
    if (!$DB) {
        die('connection to database failed: ' . mysql_error());
    }
    echo 'successfully connected to database<br/>';
    mysql_select_db($serendipity['dbName']);
    $tablename=$serendipity['dbPrefix']."config";
    $result = mysql_query("SELECT value FROM `$tablename`
    WHERE `name` LIKE 'hashkey'");
    $row = mysql_fetch_array($result);
    $hashkey=$row[0];
    echo "hashkey:$hashkey<br/>";
    $tmppass="$hashkey$newpass";
    $tablename=$serendipity['dbPrefix']."authors";
    $result = mysql_query("UPDATE $tablename SET password = sha1('$tmppass') WHERE authorid=1");
    if ($result){}
    echo "error while updating password<br/>". mysql_error();
    mysql_close($DB);
    ?>



obviously it lacks in error handling and the code isn't that pretty but I thought this might be useful for others and it did the job for me

User avatar
Timbalu
Regular
 
Posts: 4598
Joined: Sun May 02, 2004 3:04 pm

Postby Timbalu » Thu May 15, 2014 11:06 am

You might also have found the fixlogin.php script, occasionally announced here in the forum for such cases.
Regards,
Ian

Serendipity Styx Edition and additional_plugins @ https://ophian.github.io/styx/ @ https://github.com/ophian



Return to General discussions

Who is online

Users browsing this forum: No registered users and 2 guests