Aftermath of a Hack

This site was hacked. While it’s still unclear exactly how it happened, or precisely when, sometime in the past 6 weeks my blog, at least 2 other websites and possibly my DreamHost shell account were all hacked. I’m generally a pretty security conscious person, but even I get lazy from time to time. It wasn’t clear to me just how dangerous that laziness could be until this week. I’m going to outline a bit below some of the issues which may have led to my problems, and talk about the steps that have now been taken to help prevent them from occurring again in the future.

The Problem

In retrospect I can see five things I did wrong, and all of them can be traced back to laziness or perhaps, to be less forbidding, they can be traced back to actions taken (or not taken) for the sake of convenience.

Error #1 - Out-of-date Software

Many of us take the time to make sure our operating systems and browsers are up-to-date and fully patched; but do we take the necessary time to make sure that all of our software is patched? Particularly things which don’t reside on our home computers? If you run your own blog, forum or other website and are responsible for your own updates can you say unequivocally that you are currently running the latest and greatest version? Software that is out of date by as little as one revision may have critical vulnerabilities which could allow for disruption of your site, or even execution of commands on your web server.

([aside: If you don't use Secunia's PSI product on your home PC at least once a month, you should.])

Error #2 - Abandoned Web Properties

This goes hand-in-hand with the out-of-date software but is, in some ways, a bit trickier to prevent. It is far easier to remember to update software on sites which you update and monitor on a regular basis. It’s far more difficult to monitor sites which have been, for lack of a better term, abandoned. In my case there were three separate sites on my account which were running versions of their software which were more than 12 months out-of-date. The reason was that I was no longer maintaining these sites and had, in essence, forgotten they were still there. I had hidden a couple of them by renaming the homepage which made it look (to the casual observer) like the sites weren’t there but of course all of the other pages were still in their normal locations and were full of holes.

Error #3 - Shared User Accounts

Sharing is good, right? Not in this case... I have a several different domains hosted under a single hosting account. DreamHost is really generous allowing customers to register any number of domains and attach them to the account. I host sites for myself, for family and for a couple of organizations I’m affiliated with. This in and of itself does not cause a problem. The security hole in my plan was that most of these domains were hosted on a single user account. This means that if that shared user account gets compromised, all of the domains which are run on that user account are potentially at risk.

Error #4 - Lack of Backups

The websites had no viable backups. Because no regular backups were being run of the account, it was virtually impossible to trace when the hack initially occurred. If there had been regular full or differential backups being made of the various websites it may have been possible to determine when the initial attack took place and roll all of the sites back to the way they were before they were compromised. In addition, if there had been any data loss (there does not appear to have been) the lack of backups could have meant the loss of many hours of work.

Error #5 - Reused Credentials

We hear it all the time – do not reuse usernames and passwords on your various accounts, particularly accounts you care about or are important. Account reuse increases the chances that a hack on one site can do more wide-spread damage than the initial compromised password should really allow. My main SSH credentials were a username and password that I had used on over 100 different sites and services. I know for sure that one of the web properties I use had these particular credentials released into the wild. Why didn't I change the password? I don't know. If that was the entry vector, it is quite possible that a number of other accounts of mine have also been compromised.

Overall Impact

The impact was (thankfully) minimal. Only two sites of value were compromised, and it appears that all of the data for those sites is undamaged. A number of other obsolete sites were compromised as well but as they are no longer actively used they are of no great loss.

It also appears that some sort of mass-mailing script was being run from the account as well. My server-side user account had received over 27K “Message Undeliverable” replies from various web servers. I hate to think how many it was able to send successfully.

The Cleanup

The cleanup had to be done in phases, addressing each of the five defects individually. Some of them were very easy to change, others required quite a bit more effort to implement and verify. However before any of the remediation could begin, the site needed to be cleansed.

The very first step was to ensure that my local machine had not been infected or compromised. I was pretty sure that it was clean as scans are run every night, but it would be like trying to wash a car with mud. No amount of scrubbing with the muddy sponge would get it clean. The machine checked out.

The second step was to change the passwords for all of the users on my hosting account, and change the main password for the account itself.

Next, data from the websites that needed to be saved was exported. None of the code for the software running those sites was saved, only the data. There was no way to tell if the software was clean or compromised so I decided to take no chances. The application software is not that difficult to install, and I was willing to take the hit on setting up modules, components and themes anew.

Once the data was backed up I wiped out all of the data on the user accounts which were being preserved. This meant a full wipe from the file-system from the operating-system shell on the server. All files and directories including “hidden” and “special” folders were wiped out. Some of these operations required the assistance of a DreamHost technician.

Step #1 - Remove all unused or obsolete websites

This was taken care of as part of the cleanup activities mentioned above. Simply removing the affected websites greatly decreased the attack surface of the account and reduced the number of attack vectors which could be used to attack the websites and/or the account.

Step #2 – Remove all un-needed user accounts

In the case of any obsolete sites, test accounts or test databases, these were removed directly from the hosting provider’s control panel as they would no longer be needed. Much like response #1, there is no sense in keeping any old files or data hanging around where they might later become a liability.

Step #3 - Change the passwords again

Once all of the files, scripts, data, databases, directories, logs and anything else I could think of were removed from the sites, the passwords were rotated again. This was done in the off-chance that there were cached credentials or some other form of persistent authentication lurking somewhere in the ether.

Step #4 - Create new per-domain user accounts

For each of the domains that would be remaining active, a new user account was created specifically for that user. These accounts would be used to connect to and install the necessary software on the websites, as well as to run backup and maintenance scripts. Passwords for these accounts were set to extremely long strings of random characters as they would not be required for day-to-day access and maintenance.

Step #5 - Set up public key authentication

For regular access to these sites, I decided to go with public key authentication. By requiring a private key (stored in an encrypted volume on my main desktop) and a lengthy but easy-to-remember passphrase I could fairly safely rely on the same public/private key pair to secure access to all of the websites. I found out during this step that both PuTTY’s puttygen application and my hosting provider’s implementation of OpenSSH have an upper-limit on the length of the passphrase. It is still a very long upper limit, but I was surprised to find it. If you share access to a website keys can be installed for each trusted user using the same method.

Step #6 – Change passwords again (optional)

Once the public-key authentication is in place the account passwords can be changed at will without affecting the state of the affected keys. This means that I have effectively made the public keys the only viable way of accessing the site over SSH short of having access to the main hosting provider account to do a password reset. Admittedly this step is for the very security conscious (read: paranoid) as I was quite certain at this point that the passwords on the system at this time had not been compromised. This however is to be the first step in a regularly scheduled series of password rotations that the system will handle on my behalf as a part of standard system maintenance.

Step #7 - Reinstall all server-side software

Once all of the base security measures was in place and tested, I set up the application software I wanted to run on the web server. The key here is to do the set up using copies of the software obtained only from trusted sources. What a trusted source is will vary from software package to software package, but typically the main project site for an open-source project (not a mirror) or the vendor website are good places to start. In this case downloading the latest stable WordPress release from the main website <link>. I made sure not to rely on previously downloaded installation packages, getting the newest most up-to-date version I could lay my hands on.

Step #8 - Configure server-side software

Each software package is different, but going through all the configuration steps for your software package is important: don’t try to short-cut the process. In the case of WordPress we have to set up a MySQL database, set a number of hey/hash values which are used for authorization and cookies and finally set up the user accounts. I wanted to make sure that any passwords, keys or salt values were set using long randomly-generated strings. In my case I used the password generation function in LastPass. Other options would include tools like 1Password, RoboForm or Perfect Paper Passwords <Links>. The longer and more random the string is, the more difficult it will be to crack. I have been using values from 24 to 64 characters in length depending on the purpose. If you have a system that assigns default passwords for new user accounts, be sure to change those default system-generated passwords and replace them with your own strong credentials at this stage.

Step #9 - Set up extensions and themes for server-side software

Once I got the base configuration is in place it was time to add in the additional features I required for these sites. In my case it was a collection of WordPress plugins and themes. It is easy to forget that each extension, plugin or theme that you add to your website’s software package is in fact additional software that will be executed when the website is used. Just as with the base software package it is important to trust the source of your plugins and themes. If you are suspicious as to the origins of the software, choose something else. I also added the plugins and themes one at a time confirming after each step that there were no immediately visible defects.

Step 10 – Automated backup

The next step was to add a backup script for both the website and the associated database. By building this as a shell script it was possible to schedule full backups of the various sites and have them run on a set schedule. For now the script is very simple:

  1. Extract the contents of the database
  2. Zip the website and extracted database into a single archive
  3. Send that file over SFTP to a location off-site from the server There are other ideas for automation as well, but this post is long enough as it is. I will save those for later.

Lessons Learned

This could have been much worse. In many ways I count myself very lucky. I could have had all of my data wiped out, I could potentially have seen malware/scripts injected into my websites to capture login credentials or other sensitive information. This attack served as a warning and though I have had to spend a number of hours rethinking the way my websites are set up and managed, at the end of the day I will have better control over the sites I manage, better practices in place for dealing with security, and with any luck, better personal habits for dealing with information security.

Last, but certainly not least, a big thank you to the folks at DreamHost for confirming my initial diagnosis, helping to find the  possible entry vectors, providing guidance on cleanup and purging, and just generally doing that great customer service thing that they do.

Back to Basics

Over the past year my personal life as undergone some fairly major changes. I started a new job a little over a year back and there were the obvious changes that go along with that. But more importantly my wife and I welcomed our first child into the world and that was a life changing moment. Now, most of you know that I don't talk about my personal life in the blog so suffice to say that we have thoroughly enjoyed our first year as parents. It is a wonderful experience and we eagerly await every new day to see what will happen next.

One of the things that changes when you have a new baby is the amount of time you can spend on yourself and your own hobbies and pursuits. I used to spend upwards of 4-6 hours every day outside of work on the computer blogging, coding, or otherwise toiling in one digital adventure or another. Now I find that the number ranges somewhere in the range of 0-2 hours per day. That is a pretty drastic reduction no matter how you slice it (about 80% for those of you scoring at home).

There are a number of projects that I have started and stopped over the past few years each of them trying to build a better mousetrap, or re-make something from scratch just to see if I could do it. With the limited time available to me now, I have become more focused on wanting to actually do more with the time I have -- this means not reinventing the wheel every chance I get.

My wife and I have both found that we have become far more effective with our time, getting more done with less time than we ever have before. In the past couple of months I have started to extend that to my digital life as well. Gone are the days when I focused on a writing a to-do list, a backup utility, a blogging engine, a photo manager or a disk-erasing tool. There are lots of great (free) tools out there which can handle those tasks very well, even if they don't satisfy all my neurotic desires (like how my historic completed work tasks should be handled, cataloged and stored for reporting purposes (you know, for when I will pull metrics on my completed work)).

I have also decided that diving in to learn a new, modern programming language is probably something that would realistically take more time than I'm willing to devote to the enterprise. Python, Ruby, Java, and the ASP.NET MVC framework are all on my list, but are undergoing changes and enhancements so frequently that I'm having trouble keeping up with what's out there, nevermind trying to actually learn the stuff. But I do want to become a productive programmer in some language outside the rather constrained, and somewhat self-imposed, .NET bubble in which I have spent the majority of my professional career. Ideally I would like to write in something that I can port between operating systems without too much headache. Being able to produce code that will run on anyone's machine is a great asset -- especially when you have Windows, Mac and Linux machines in your own house to start with.

So the question is what can I learn that will allow me to:

  1. write code for multiple platforms
  2. grow as a developer
  3. not have to keep up with constant enhancements

The answer I came to was 42 C. It seems to satisfy all of the criteria above for me in a way that other languages don't.

C is by nature intended to be a multi-platform system. If you're able to confine your applications to CGI or the command-line this is made even easier.

C also requires developers to know much more about how computers and compilers work than more contemporary languages like C#, Java or Python. Though it arguably makes programming more difficult, I think it will help me become a better programmer over time as I learn some of the trickier parts of getting a computer to do what I want it to do.

The current ANSI standard specification for C was introduced in 1999. This means that for the past 12 years, the standard for C programming has remained essentially unchanged. This makes C a good choice for someone who doesn't have a great deal of time to keep up with changes and enhancements in the specification.

For all these reasons, and my own simple curiosity I'm embarking on an adventure to learn and become proficient in C. I make no assertions that I'm trying to master the language as I can't see myself getting beyond the hobbyist or perhaps open-source contributor stages. I do have some ideas for the first couple of projects I would like to tackle once I get the basics out of the way. Hopefully I'll be able to release some source code back into the world over the next year or two -- after all, I'm in no hurry.