How I Lost (Almost) Everything, And Why You Won’t

I excitedly tore open the envelope which I expected to contain my first sizeable affiliate paycheck. I pulled out the contents, raised it in the air and sure enough… a check for over $10,000. The rush, the adrenaline… the pure excitement felt great.

The phone rang and I answered the call from my mother, “I can’t login to Chef Club… something isn’t working.” Hmmm… okay, showing some type of MySQL error – that’s odd. I instant message my host who says the server probably just needs a reboot. He reboots it… and what follows is an absolute nightmare.

“Your server was compromised… we’re trying to get everything back up and running.” An hour passes,… two hours… five hours….  “We’re doing our best.  We’re going to reload your backup files but it could take a few more hours.”

After 24 hours have passed, the tune changes a bit, “Your server has a bad hard drive and it simply died… it’s gone.” Fortunately, they had kept backups. “When we tried to partition the hard drive and reload your databases, the data was corrupted and automatically deleted.”

But they still had my backups… right? “The corrupted files were your backups.”

They were happy to inform me that they were able to restore the contents of my public_html folder to it’s previous state and only the databases were lost. ONY THE DATABASES? Since I run mainly wordpress and vbulletin sites, EVERYTHING is contained in those databases. All the content, posts, everything… And they were all gone… POOF…. Disappeared.

I changed from a shared server to a dedicated server in late November 2007, shortly after making the decision to resign from my advertising sales job and pursue internet marketing full time. Fortunately, they DID have the MySQL backups from when my sites were on a shared server. And, that makes explaining the loss pretty simple:

Everything I’ve done since I began doing Internet Marketing as a full time job is lost… and all because of a failed hard drive with corrupted backups. I was furious, upset, distraught, stressed and words couldn’t fully express my emotions. My vbulletin forums now appeared completely dead. Time, money, effort and luck went into improving their activity. I had created a dozen or so mini content sites that were now non-existent. And every single news update, blog post, everything content wise… all gone.

I took a couple days off simply because I could not dive back into rebuilding my network without a breath of fresh air. I was too frustrated to start plugging away. So, after a 2 day break, the first thing I did was to conduct some research to put my own “iron-fist” backup system into place.

And that is my message to you: before you do anything else, take a look at they system you use to backup your information. If other people are backing up your information, that’s great, but you need to diversify that backup because if something terrible happens… you need to be covered. And, nobody will put as much effort into making sure you are covered than YOU. Make your own backups.

I went to BestBuy and bought a 1TB external HD with the idea of doing nightly backups. After doing my first full backup, manually, it was a 1.5GB file that took 9 hours to transfer from my dedicated server to my harddrive. Obviously, nightly complete backups are impractical.

There is a backup method called incremental backups or differential backups that only transfer files which are new, updated, changed, etc… it basically transfers the difference between your last backup and your server’s current state. However, incremental backups with MySQL databases are difficult - since the sites are live, if changes are being made as incremental backups occur, it can break the database. And, we don’t need any more problems.

I decided to set up a simple system until I had a more complex system in place:

  • Weekly backups to my external harddrive
  • Monthly backups burned to DVD
  • Home computer backed up continuously to my external  harddrive
  • Home computer backed up to online storage at http://www.mozy.com
  • I’ll also put full backups to Mozy as well

With this system, the most I would ever lose is a week of backups and there is virtually no chance I could lose everything. If my server messed up, my server backups were corrupted and a fire at my house destroyed my computer, external hard drive and the backup DVDs… I would still have a maximum week old backup at Mozy.

I’m currently working with my hosting provider to do nightly MySQL backups that will get sent to my home computer via FTP and these backups would be incremental. They would be produced by creating MySQL dump files of the databases before backing them up incrementally, so they always include a full, working database that is unbroken and cannot impact the functionality of my websites.

Rsync and RDiff are two open source systems for doing these backups, but you have to technically advanced to set these up. If your hosting provider can backup the files directly to THEIR server, you could get them automatically every night using WGET.

The factor that my backup system currently doesn’t allow for is automation, which is a key to any good backup system. I’m currently working with my host to get this done, and chances are, you might need some help as well. Do some research and see what system might make most sense for you. But, instead of learning from your own mistakes, learn from MY mistakes:

Don’t wait to put a backup system in place until after you need it!

Do it now. The whole point of backups are to ensure safety of your data regardless of the unforeseen disasters that could happen. We know… an unforeseen disaster probably won’t happen to you. You can wait… NO YOU CAN’T.

I waited and I paid the price. Remember… you are running a business. Be accountable for your actions. Prepare yourself for the road ahead. Do your research. Put a system in place. Automate it. Check it to make sure it works. When and if you do have a data meltdown, you won’t feel like “the luckiest person on the planet”… you’ll be happy you’re so damn smart and prepared.

I’m currently trying to rebuild my network of sites and feel like I’m light years behind. I’m so unbelievably overwhelmed it’s ridiculous. Don’t let this happen to you. Protect yourself.

And, if you’re interested in the following topics, I would greatly appreciate your involvement in my forum communities that have experienced tough blows to their vitality:

 Here are some sites that I found useful when doing my backup research:

  • Online backups at Mozy
  • Rsync - open source software for incremental backups
  • Rdiff - open source software for differential backups
  • wGet- software for automating process of GETTING backups from your web server
  • Do It F&%@ing Now - why you need to get off your butt to not only put a backup system in place… but to keep working, working, working at this web thing.

Alright folks… time to stop feeling sorry for myself, get off my butt and D-I-F-N!