So at 2:00 this morning the city of Boulder lost power. It was out for three hours, during which time the facility where I’m co-locating my web servers had a diesel generator running. Around 4:00 the generator sprung a fuel leak and shut down, so my UPS batteries kicked in. They lasted a little over half an hour, and then all my computers shut down.
When the power came back on, four of my systems rebooted themselves. The other two didn’t. This was, of course, a problem– those were the database servers. I woke up to clients calling and asking where their web sites were.
Although this is really just a case of bad luck, it goes to show how complicated it is to keep servers going all day, every day. Fortunately my clients are understanding; they could easily fly off the handle and scream at me about incompetence…