Weekend afternoons/evenings at work, we're perpetually in a state of being short-handed. There are typically only two or three of us on duty until our Singapore office comes online, but they go the minimal-staffing route on weekends, too. So at best, there are only four or five of us on duty at night to handle 30,000 managed routers and switches.

Usually, this isn't a problem. Routers are, on the whole, pretty reliable beasts — certainly as compared to PCs. I have personally seen routers with "up" times in excess of five years — running for over five years without crashing, without failing.

[One story, almost certainly apocryphal, is told in networking circles of a router that had been walled up in a disused closet, and for years no one ever realized it was there until they got curious about where some particular ethernet cables were leading.]

When these usually-reliable devices fail, though, they can do so quite catastrophically. Such was the case this weekend.

Saturday night I spent the entire night on one conference call to deal with a switch that had failed. When that switch failed, we lost access to scads of other devices — how many, exactly, I never learned.

The call started early that morning before my shift; a card in the switch was replaced, which was when the real problems began. I joined twenty other people on the call that evening and spent the night trying to troubleshoot by phone. No one had access to the switch except a technician on-site with a terminal, so I would tell him to type in a command and he would read the output back to me. This went on until the wee hours of the morning, and eventually I passed off the call to one of my graveyard-shift colleagues.

Sunday when I started work, I learned to my dismay that the conference call was still going on, and I was again needed on the call. I spent the entire night on the call. When I left, we were trying to get the hardware to replace the entire switch, not just the one card.

For all I know, that call is still going on. I have never been so glad that Monday is one of my days off.

Posted by Russ at 06:59 PM, February 4, 2008 in Geekery