Yesterday, after 15+ years of reliable, 24x7, clean, data-integral service, Postgresql on one of our primary DB servers decided it was time to corrupt itself in a horrible, not-immediately obvious way. We had async replication running, so we quickly rolled over to the hot standby, only to find some time later that the corruption replicated, as well. This is far and away, the only example I've yet seen of PostgreSQL having a data corruption issue. And it's the worst incident of data corruption I've ever seen.
Thankfully, we had backups of all of our data from numerous points in time, so recovery is a few hours-long process of loading all our customer data from backups. They will be a bit upset to find out that their work for the last day or so is lost, but if we had not had these backups, it could easily have meant the death of our company.
We've always had excellent backups, always daily, sometimes hourly, for all the services we host. Multiple sets of backups, replicated online to multiple locations, just in case. And so, tonight, recovering from a long, very stressful day, watching the databases load in another window, I thank myself yet again for our determination to never be without recent, verified, known-useful backups.
Are you backing up your data?
[link][7 comments]
