“This is the Call That Never Ends…”
Posted By Russ Emerson on February 4, 2008 at 6:59 pm
Weekend afternoons/evenings at work, we’re perpetually in a state of being short-handed. There are typically only two or three of us on duty until our Singapore office comes online, but they go the minimal-staffing route on weekends, too. So at best, there are only four or five of us on duty at night to handle 30,000 managed routers and switches.
Usually, this isn’t a problem. Routers are, on the whole, pretty reliable beasts — certainly as compared to PCs. I have personally seen routers with “up” times in excess of five years — running for over five years without crashing, without failing.
[One story, almost certainly apocryphal, is told in networking circles of a router that had been walled up in a disused closet, and for years no one ever realized it was there until they got curious about where some particular ethernet cables were leading.]
When these usually-reliable devices fail, though, they can do so quite catastrophically. Such was the case this weekend.
Saturday night I spent the entire night on one conference call to deal with a switch that had failed. When that switch failed, we lost access to scads of other devices — how many, exactly, I never learned.
The call started early that morning before my shift; a card in the switch was replaced, which was when the real problems began. I joined twenty other people on the call that evening and spent the night trying to troubleshoot by phone. No one had access to the switch except a technician on-site with a terminal, so I would tell him to type in a command and he would read the output back to me. This went on until the wee hours of the morning, and eventually I passed off the call to one of my graveyard-shift colleagues.
Sunday when I started work, I learned to my dismay that the conference call was still going on, and I was again needed on the call. I spent the entire night on the call. When I left, we were trying to get the hardware to replace the entire switch, not just the one card.
For all I know, that call is still going on. I have never been so glad that Monday is one of my days off.
That’s pretty amazing–at least it offers secure employment! :)
Speaking of which, any change on your “contract” status?