Well, it’s been one of those weeks.
On Monday, internet security researchers reported a flaw in the OpenSSL software that underlies almost every open-source security package in the world. Any web server or login account that relies on the software was vulnerable to an attack that would allow the bad guys to steal user credentials, decrypt financial data in browser sessions, impersonate secure sites, and so forth. It was generally agreed this was a Bad Thing– Bruce Schneier himself remarked, “On a scale of one to ten, this is an eleven”.
It was dubbed “Heartbleed”, and it’s such a big deal that it even has it’s own… logo?
So, starting on Tuesday, I went to work patching the software on all of my servers. I currently own 115 of them at my hosting facility, and I manage about 20 more for various clients. Although the patch itself wasn’t all that complicated, it required manually updating every server, rebooting it, and confirming that it came back online okay and all of the services were running normally. Yeesh.
Every night this week I went down to my basement office after dinner and camped out there until about 1:30 in the morning. After three days of that, I was pretty beat. Then, today, I updated the login keys my team and I use to access all of our servers. This was more a precautionary measure than a necessity, but we all agreed it was prudent. That took most of my afternoon. The next step– which will wait until next week– is to revoke and re-issue the security certificates we’re using on various web sites.
Of course I’m not alone: I suspect almost every system administrator in the world was putting in extra hours this week to mitigate Heartbleed. Since this was completely unexpected, all of the projects I’d planned to do were sidelined, and next week will be a game of catch-up. Hoo boy.
At the end of the day, though, I feel pretty good about updating over a hundred servers, communicating with my clients about it, answering their questions, and keeping everything running. This afternoon I commented to Laralee:
Her response (because she’s awesome) was “That doesn’t sound too hard.”