SharePoint 2010: Test Your Recovery Plan Before You Need It

Having backups and having a working recovery plan are not the same thing.

That distinction became painfully clear for one customer recently.

The organization had a backup strategy in place for its SharePoint 2010 environment and, like most organizations with regular backups running successfully, there was a reasonable level of confidence that the environment could be recovered if something went wrong.

Then something went wrong.

When the time came to recover the environment, a mistake during the recovery process resulted in the production content database being overwritten. Instead of restoring the information they were trying to recover, the organization inadvertently lost approximately three months of operational data.

The backup process itself was not really the problem. The problem was that the recovery process had never been sufficiently tested.

A Successful Backup Doesn't Prove You Can Recover

It's easy to look at a successful backup job and assume that your SharePoint environment is protected.

The backup completed. The file exists. The logs say everything succeeded.

Unfortunately, none of those things prove that you can actually recover the environment when you need to.

Recovery introduces an entirely different set of questions.

• Do the people performing the recovery understand exactly what needs to be restored?
• Do they understand the relationship between the SharePoint farm, SQL Server, and the content databases involved?
• Can the recovery be performed without putting the existing production data at additional risk?
• Have your customizations been documented and are in a position to be re-deployed?
• Has anyone actually gone through the process before?
• And perhaps most importantly: has anyone verified that the restored data is usable?

Until those questions have been answered through an actual recovery test, you do not really know whether your recovery plan works... you only know that your backups are running.

Don't Learn Your Recovery Process During an Emergency

A production outage is a bad time to learn how your recovery procedure works.

When systems are unavailable and users are waiting, administrators are already operating under pressure. Decisions are made quickly, and mistakes that might be obvious during a planned exercise can become very easy to make.

That is exactly why the recovery process should be exercised before there is an emergency.

Create a test environment. (don't gloss over this one, there's a reasonable chance you're gonna need a spare farm in a hurry)

Restore databases.

Verify that the expected sites and content are available.

Make sure the people responsible for recovery understand the procedure and document the steps while there is no pressure to get production back online.

You may discover that your documentation is incomplete.

You may discover that your backups are not protecting everything you thought they were.

You may discover that your recovery point does not meet the expectations of the business.

Those are all good things to discover during a test & terrible things to discover during an outage.

Test Before It Matters

Backup and recovery planning is one of those things that is easy to postpone because, most of the time, everything is working.

But the first time your recovery process is exercised should not be the day the business depends on it.

1. Schedule the time.
2. Build a recovery environment.
3. Restore the data.
4. Document what happens.
5. Fix whatever does not work.

Then do it again.

The worst possible time to discover that your recovery plan doesn't work is when you actually need it.