Ever have a train of thought that started somewhere, then progressed a bit, then spawned two or three other thoughts, which took on a life of their own, and all of them important, all of them relevant, and each and every one really important? Now what happens when that happens recursively and begins to stack up?
Think fractals.
Then make whatever you picture that to look like much bigger.
Cram it all into my head.
Add water. Bake at 350°.
So it all started when at about 11:300 on Friday, our team at work gets word that at 9pm the power to both of our floors is going to be cut, hardlines are going to be sawed through and rerouted, potentially an outage for 6 hours (or more).
We immediately go into reactive mode, trying to handle this on a couple fronts.
- See if we can postpone the shutdown
- Consider power alternatives, like a temporary generator
- Design a shutdown and powerup procedure
- Figure out who can and will do the actual work
- Figure out what machines we have where
There’s more, but I don’t recall everything.
We get a management guy to go talk to the building management to postpone the power-down, while the rest of the tem works on other tasks.
After determining that a generator would probably not have the capacity needed for powering out 50+ servers in our local server room (not even considering the A/C!) for more than two hours at a time – it needs to be manually refilled with gasoline – we accept the harsh reality that a shutdown has to happen.
One drafts an informative email to everyone in the company (including branch offices) informing them that the company is being forced to go offline for a while, while trying to push for postponement.
Another takes a quick inventory of what we have on in the back room – where we keep desktops (used as testing and dev servers sometimes).
I head into the server room and take a physical inventory of the machines there – and slap together a Visio diagram of servers, names and rack placements.
We discuss people’s availability, ad everyone pretty much volunteered to be where and when they were needed – we were also running a scheduled equipment move in our datacenter – unaffected by the powerdown, but has to happen this weekend, otherwise other planned items are at serious risk.
Finally, at about 3:30 or so, we get the wave-off – the shutdown has been postponed for a week.
This reveals a couple of things – we are not prepared for a power outage, and we need to design something to keep us up and running in the event of power failure – at least until things can have an orderly shutdown.
During all of this mild chaos, someone sends and email, reminding us of it being System Administrator Appreciation Day – something that completely missed us all. Funny that.
So there’s going to be a fair amount of work on this during the course of this week, and next weekend’s not going to be a picnic, that’s for sure.