“What do you mean you have no backups?? My responsibility?! But I thought it’s your responsibility as my hosting provider to do my WordPress backups?! I don’t care about your server, I just want my business to be restored as fast as possible.”
—
Now that Labor day weekend is over, it’s time to discuss some serious matters. More specifically – website backups. For the past five years, I was operating a bunch of different sites and systems first at FastCompany and presently at HBO (not including numerous smaller cliens). My main responsibility is to make sure everything works all the time and provide a disaster recovery strategy for the mission critical data.
Unfortunately, many of the smaller companies doesn’t even consider potential data loss (in some extreme cases – entire online business data loss!). A lot of clients that I worked with in the past just assumed that the hosting company would take care of their Drupal, WordPress etc website backups.
Well, let me tell you a story. It was early morning and I was happily sleeping in my bed (I didn’t wake up at 5.30am at the time :)) when I received a call from one of my clients.
He had some emergency with the WordPress website and wanted my help to restore the operation as quickly as possible. He was losing money for every minute that the site was down (pretty common for any type of e-commerce site). I did some performance and security enhancements for him in the past, but he refused to host his site on my platform, saying that he had his own guys to take care of this. Totally fine by me, by the way.
I asked what sort of issue did he currently have, trying to judge the importance and prioritize accordingly (basically I wanted to know if I should get out of the bed NOW or sleep some more and take care of it later).
He was very emotional on the phone: “Sergey, my site doesn’t load and I’m losing money! Can you please help me, please! My guys were looking at it for the past 40 minutes and could find a problem.”
Well, the problem sounded serious enough so I actually had to get out of my bed and take a look. The site didn’t load indeed. As I started digging into the problem, I figured the IP address of the server changed for some reason and the DNS record was still pointing to the old one.
Easy peasy! Well, that’s what I thought initially… I tested the site on the new IP but it didn’t work either. What the hell?
Once I received control panel credentials from the client, I noticed there were no WordPress files or database uploaded to the server. It looked like brand new hosting account basically.
The client could not believe what I was saying. He started nervously calling to the hosting support line assuming there was some kind of mistake there. Maybe the IP address was wrong after all.
About an hour later he called me back (most hosting providers do not excel at support, by the way, pick wisely as that’s one of the most important components, not the price like most people think!). His voice sounded very weak, I could barely pick up the words.
“Sergey, those bastards lost all of my data… They had an incident with the server last night, some power surge in the data center or something and multiple hard drives burned… Our business WordPress website was on one of those drives and they don’t have a copy… They said backups were my responsibility, not theirs, so I couldn’t go after them… Everything is lost, everything! What do I do now, Sergey?! What do I do!” and he started sobbing.
HAPPENS ALL THE TIME YOU GUYS!
The whole business could be wiped out in a matter of seconds if you don’t have a proper recovery strategy in place and guess what, most businesses (at least in my experience) have no plans for that.
Yes, most good hosting providers have RAID systems in place. That means hard drive failures should not cause the data loss. In the real world, as you saw just a moments ago, even RAID systems do fail…
Remember, website backup (whether it’s WordPress, Drupal, custom CMS or custom code) is your responsibility! I understand, as the business owner, you have to do hundred other things as well. Most of them may seem much more important than implementing the proper backup strategy. Don’t overlook it, unless you like to dance on the razor’s edge!
What the hell is the proper backup strategy? Glad you asked. I was waiting for it.
When I take on new clients, I follow my simple 3 step website backup and recovery strategy:
Step 1: Enable backups
Seem simple right? Yet, 95% of website owners never do that. Remember, every hosting provider is different, some offer backup services for a few extra dollars a month. Just take it, much better than nothing!
For some popular CMS’s on the market like WordPress or Drupal, there are lots of different backup plugins. You can use those too, instead of (or in addition) to the one provided by the hosting company. Trust me, more backups never hurt (it can affect the performance of your site if you do it very frequently, use common sense and don’t become too paranoid).
One gotcha though – you need to take some time to understand and test how that particular backup solution works. Test it. See where it stores data, what and how exactly it makes a copy.
There could be some significant differences when it comes to database backup versus regular WordPress site files backup.
It’s a good idea to create some simple instructions (even for yourself), so when disaster strikes 6 months or 2 years down the road you can quickly restore the current backup srategy in your mind. Trust me, we humans tend to forget things and we do it very VERY quickly.
Step 2: Keep your website backups off-site (any remote location will do)
This point is very important. A lot of the backup solutions on the market store your backup on the same server as your current website.
Plugins are not always doing a better job. I’ve seen a lot of the WordPress plugins that just create a new subdirectory for the backups and store files inside of the project folder on the same server.
Do you know what is going to happen when the server your site is running on goes down (crashes)? Exactly! All your backups will be lost along with your precious website! And it’s not always disks failure, by the way. A lot of different things like network partitioning, power outage, cooling system failure in the server room, a security breach in the data center to name a few can cause your server to be unavailable temporarily or permanently.
You will be in MUCH BETTER SHAPE if you store your backups on the external machine or cloud service like S3. In case of S3 all files are automatically replicated to other data centers within the availability zone, so you don’t need to worry about that part.
You can even send a copy to the another region for very sensitive data. Obviously, it costs more money which could be substantial depending on the dataset size. My recommendation is to start with something simple and improve later on.
Step 3: Periodically test your website (WordPress, Drupal or whatever) and database backups and procedures
This step is very commonly overlooked, but definitely not less (if not more) important than the previous two!
Here is what usually happens – a client completes first 2 steps (enables backups and store those externally) and naively thinks that he is totally protected from all possible disasters.
First of all, let’s pause for a sec and give this guy a credit he deserves. Even following just two first steps of the disaster recovery strategy will put him into the top 2% of people who care about their personal and business data.
Unfortunately, it’s not enough… I can’t tell how many times I saw corrupted backups that cannot be restored! Yep, you read it correctly, you may have a backup file, but still, lose some important data.
I’m not going to go into nitty-gritty details of the proper backup files creation. The post is running too long already. Just trust me on this one, depending on your backup method your backup file may become damaged and impossible to restore.
This is especially important for database backups, as they tend to be quite large. Sometimes you can’t just use mysqldump command to create database backup, you have to do filesystem snapshots and also store changes (aka binlogs) in case you want to do the restore in time operations (someone dropped a table accidentally) or audit queries that are coming to the database.
Do not worry about the last paragraph as it usually applies to large businesses. The MOST IMPORTANT thing to remember is that you need to periodically test your backups and restore procedures.
Pay attention to the last word “procedures”. You may easily overlook it – don’t! Here is why. I’ve been consulting some pretty big clients and what I noticed in multiple cases (most of them actually) was the lack of the up-to-date restore procedures that caused unnecessary downtime for the business.
Basically, the backup files were periodically created and stored in the cloud, but no one really tested the complete restoration process. I had to chase down different people to get bits and pieces of information out of them (big systems get quite complex your know), get all the backup files from different backup locations and initiate the restore process in particular order (also important sometimes).
If you never practiced your restore process, I can guarantee that in case of emergency you will waste a lot more time than necessary. Plus you will be under a lot of stress as well, as people will be running around asking the same questions over and over again: “Are we back yet? How much longer?”. You may even notice some high-profile executives silently standing behind your back, adding yet another dimension of stress to your already pretty stressful situation…
To minimize the amount of stress and unnecessary downtime I highly recommend to maintain restore instructions. You also need to keep them up-to-date (ideally even automate as much of restoration process as possible) and regularly exercise full restore process.
—
Ok, guys, have to go now. The post is getting out of control…
What happened to the poor soul that called me in the morning and didn’t have any backups? Fortunately for him, I had a copy of the entire site on my workstation (remember, we did some performance optimizations for him in the past). The database was about 3 months old but better than nothing.
His business was saved by a miracle basically. Please, please, please – don’t end up in his shoes. It might be THE END for your whole online business. I saw it happen a few times and it wasn’t pretty…
Leave a Reply