The backup in the box
A backup nobody has restored is a bit like the cat in the box. Some thoughts on the code that only wakes up on bad days.
I’ve been calling backups which haven’t been tested “Schrodinger’s data”. “Is it backed up or not?” you ask. Well, it’s in both states until you actually try to restore something. And you’ll do that at the worst possible time of the year.
Backups are the example that springs to mind but there’s always a bit of code in the attic in your system. It’s code that only executes in the event of an issue. The restore code. The migration down step. The fail-over to a replica. Production code is poked and prodded all day long by actual humans, which means bugs come out quickly. Attic code sits untouched for months, accumulating quiet bugs. The first time it’s run for real, we’re all very sweaty.
The amazing thing is how often we find that no one thought of it. There was typically someone. They had a backup job. They might even have an alarm that went off if the backup job failed. Oh, the alarm showed up in an inbox that hasn’t been looked at since the person who set it up moved on. And the backup job had been nicely failing to that inbox for weeks.
(And don’t even think about the replica, eh? It’s a faithful copy of your stupidity as well as your data. Drop a table and it drops a table, too. Damn efficient, isn’t it?)
Maybe the problem isn’t so much “how often we miss it” as “this recovery stuff is nothing”. If you test a restore and it passes you don’t get anything back. There’s nothing to demo. It’s all theoretical until it’s what you care about.
How long will it take to do a full restore? Twenty minutes? Half a day? Most of us would have to guess wildly. And it’s better to find out over a coffee in the calm morning than to have an important person on the phone.
Attic code has a bit of melancholy to it. Code written with great care while imagining the worst. Waiting and faithfully perhaps buggy for the day you don’t want to have.