BoomFlag

I’ve owned BitRelay, my web-hosting company, for about twelve years now. For most of that time, I’ve sort of considered it a “side business”. My main source of income has always been web programming, not web hosting. Hosting is notoriously a low-margin business– in general, web providers are in a race to the bottom. My cheapest accounts run $20/month for simple static sites, and that same $20 would probably buy several years’ service at one of dozens of low-cost hosting providers out there.

In short, it hasn’t been about making money– it’s been about having access to dedicated servers and an environment that I know and control. Back in 2001 when I got started in the hosting business, there was no “cloud”, nor were there even virtual servers. $20/month was par for the course, and dedicated hardware was much more expensive than it is today. So it made a lot of sense for me to set up my own operation to support my main business of building web sites.

Fast forward to today, when I have four full racks with over a hundred servers, and probably well over $100,000 worth of hardware. I’m not quite sure how I ended up here, but this isn’t a garage operation any more. In addition to the growth I’ve seen on the technology side, I still feel like having a hosting company is a great thing. It gives me a lot of capability and flexibility as I work with clients who have very specific (and often quite complicated) needs. Perhaps the best thing about hosting is the servers just hum along, running 24/7 and earning money while I sleep.

That is, until something goes wrong. Then everyone is calling and emailing, asking why their web site is broken (or down entirely), or why they can’t get email, or what’s going on with the business-critical system we built for them. Those are tough conversations, and cause a lot of stress all around. Fortunately major incidents are very rare; I’ve had only a handful in twelve years.

One of them happened a couple of weeks ago, which unfortunately was when I was on Trek and completely out of communication with the technical world for four days. My team at Zing did an amazing job of working through the challenges, reassuring clients, and even considering alternative solutions for an apocalypse scenario. It didn’t come to that, but they were sweating. When I found out what had happened, I felt pretty bad about it. I hadn’t given them the tools or access they needed to properly troubleshoot, leaving them with some difficult options (one of which was to actually drive up to the middle of Wyoming to find me on the trail somewhere!).

In any case, I learned a hard lesson and realized that I’ve been treating all of this like a side business for over a decade, when in fact it needs to have much more of my attention. My clients absolutely depend on me. Sure, some of them have a little brochure web site that could be down for a day without a lot of impact… but others have mission-critical stuff that would easily cost them thousands or even tens of thousands of dollars if it was offline for a day. Ouch.

As such, I’ve been working for the past two weeks on planning all sorts of redundancies, failovers, and even alternate hosting solutions. I need to write documentation, set up access, and train my team so the next time I’m incommunicado (ahem, fall backpacking trip with Thom) they have what they need to battle the emergency. It’s been interesting to consider worst-case scenarios and figure out how to bring things back to life.

I cross my fingers that nothing goes so wrong that it requires these drastic measures, but I hope that within a month or so I’m in a much better position to offer these services to clients and feel confident they’ll be rock-solid. I guess we’ll see.

Kyra just finished her first high school softball game. She decided to try out for the Skyline team this year, and tonight they had their first game.

Unfortunately I didn’t get a good photo– this was shot through the fence, past the batter, with a nice view of the football team practicing on the field beyond the fence. Kyra played second base for a few innings; she didn’t get a chance to bat because they were pretty much three-up-three-down each inning. The other team whomped them 18-0.

Still, it’s pretty cool that she’s on the team and I’m excited for her and the other girls to get better through the season.

Well, tonight marks the end of an era. Our last rat, Squinkie, passed away.

She had a tumor that was almost half of her body weight, and it was infected pretty badly, but she didn’t really show signs of pain or sluggishness. So we decided to have her put down.

That’ll end the string of pets at our house. If the kids want to get another pet, I’ve told them they’re welcome to do so… when they’ve moved away. The rats were a lot of fun, and they really grew on me, but it’s time to move on.

Josh and I just went to Subway to grab a sandwich. The Sandwich Artist there was amazingly fast. She whipped out the bread and had everything ready lickety-split. Wow.

Of course, the guy handling the toppings was working in slow motion, so there was a big sandwich traffic jam on the counter. Go figure.

For some reason, the little smiling ghost logo used by the Symfony PHP framework team bugs the living heck out of me.

I did something bad and the entire site crashed… I don’t want a smiling Pac-Man ghost telling me about it. Grrr.

Yeah, another sweet application for our full-time on-site programming job:

This evening we headed over to the Boulder County Fair. Laralee pointed out that we’ve lived in Boulder County for fourteen years and we’ve never gone to the fair.

It was a gorgeous evening, and there were a lot of kiddie rides, goats and cows and chickens and bunnies, and plenty of vendors selling food. Of course, in the tradition of county fairs everywhere, all of the food was deep-fried. Mmm. We ended up getting snow cones from Alex, who was working his first shift at Boulder Snowie. Although we had to buy cones for the kids, I get free snow cones for life because I helped with the company web site. Nice!

The kids had coupons for free ice cream, so of course we had to take advantage of that. Here’s Zack playing Mr. Cool as he eats his.

We headed across the street to Red Robin for dinner– always good food. Then we came back to the fair to see Face in concert. They’re an acapella group, and a lot of fun.

Unfortunately the boys were getting restless: Zack doesn’t care much for music, and Alex had been working at the fair all day already. So after an hour we headed home. Good fun all around.

Ahh, High Adventure. It’s an annual tradition in the Boy Scouts: the “big trip” for the older Scouts (14 years or more). This year the trip was up in the Rawah Wildnerness in north central Colorado. The boys learned how to fly fish prior to the trip, in the hope of catching some dinner.

As it turned out, High Adventure kicked their butts. I came up on Day 3 to relieve one of the leaders who couldn’t stay for the full six days, and heard the story of what had happened the previous few days. The hike from the trailhead to camp was about six miles, and much more difficult than anyone had anticipated. Several of the boys really struggled with their packs. After arriving at camp, one of the boys and one of the leaders had pretty bad altitude sickness (camp was at 10,500 feet), and another boy refused to wear a hat and forgot his sunscreen, leading to a really bad sunburn and probably the beginning of heat exhaustion.

The second day was spent in camp, mostly lounging around because several of them felt pretty beat. A few headed up to a lake to do some fly fishing, but even that proved tough because the mile-long trail to the lake had a 1,000-foot elevation gain.

Then came the rain. And hail. It dumped all night, and a few of the boys ended up with wet sleeping bags (never a good time). On the dawn of Day 3, they had a camp meeting to decide what to do, and the majority decided it was time to head home.

So there I was, pack on my back and ready to hike up on Day 3, and they all came down the mountain to tell me we were leaving. Umm, what?

To his credit, Alex was one of three boys who wanted to stay. He had packed well, kept his stuff dry, and he’s actually in really good physical shape. But since most of the boys were done, the group decided to leave together. We drove through Fort Collins and enjoyed some awesome pizza at Beau Jo’s, then went over to the home of one of the leaders to watch a movie. It didn’t quite live up to the “High Adventure” name, but I think several of the boys learned really good lessons about packing light, drinking, using sunscreen, and getting in shape before a hiking trip.

I found the greatest tie in the world. A guy named Bill Porter made it by hand. It’s a white tie with LED’s inside of it, and a microcontroller that plays Tetris. As you wear the tie, Tetris blocks fall down the length of it and stack up at the bottom.

He put instructions online, and I’m sorely tempted to give this project a shot. It would be so awesome.

It took me a moment to get this joke.

An electron is driving down the highway, and a policeman pulls him over. The policeman says, “Sir, do you realize you were traveling 80 miles an hour?” The electron sighs. “Great, now I’m lost.”