mhoye The terrible thing about being this awesome is that it’s hard for people to fully appreciate it when I make it look so easy.
I spent part of last week getting my office out from under the catastrophic failure of our three ostensibly-redundant AC units in the server room. They’re water-cooled units that were all fed, very much to my surprise, from a single cold-water pipe that some baboon in the basement of our building broke. When I got to work Wednesday morning the server room had spiked to 43C, showing no sign of going anywhere but up, and about a quarter of my precious machines had already shut themselves down in self-defense.
We recovered from that mess with a little bit of luck, some nontrivial effort and a ton of advance planning done months ago; we did it a lot faster than anyone thought we could, and I’m proud of it. Still, it revealed a few gaps in our disaster-management and -recovery processes that need to be addressed. We were even lucky enough that the only immediate equipment failure was a single drive in an unused raid set, which was a tiny stroke of luck, and none of the machines that shut down to protect themselves were near the core of our dependency tree, which was huge. But the cold hard fact of the matter is that an orangutan turning the wrong valve seven floors away got quite close to paralyzing my company, and the orangutan found that valve before we did.
We’re going to solve the hell out of that problem, make no mistake; we are going to solve it so hard it’ll be walking funny for months. And ending the week with the personal thanks of my CEO felt pretty good. But this little vignette brings me, circuitously, to the thing I really want to talk about.
Every couple of months, somebody writes an article about how terrible their IT departments are for not letting them install whatever they want, how IT is just there to hold them back and not let them have any fun, and mostly that’s because IT is lazy and doesn’t feel like catching up, much less keeping up, with these shiny modern times. Even my friend David Eaves recently referred to a “distant IT overlord” in an otherwise well-aimed post recently, much to my chagrin.
First off, just to be clear let me tell you this: I would love to be a Distant IT Overlord. Love it! But as it stands users keep coming to my office with their needs and questions, and despite my best efforts (starting all my conversations with “no”, answering questions with “cut the baby in half” or “shoot the hostage”, you know, the usual) I still seem to end up actually trying to actually help them get their jobs done.
Embarrassing, I know! I keep pushing for those little changes in the decor that could make all the difference, like a moat around the office or having some random intern’s severed head hoisted on a pike by the server room door, but we never seem have the budget. Maybe next year.
These articles are invariably, and tellingly, written by people with no IT background at all. Which is OK, that’s most of us, but they’re also written by people with no idea what the implications of their demands might be, and no real interest in finding out. Some of that, as Eaves notes, is cultural and deeply ingrained, but I think he only gives the barest nod towards the reasons those attitudes exist and (surprisingly, from a veteran negotiation and policy wonk) to the idea that there’s an actual culture there, and hence an understanding to be had and if we’re patient and a bit lucky some common ground to be found.
I like to think that I understand a part of it, so I’d like to lay some of it out for you here. This isn’t to garner sympathy; I’m not going to claim that my job is harder than yours, and I know there’s a lot of people in IT who just have no business interacting with other humans. I’m even a little unhappy that this discussion always ends up with the Us V. Them tone they always seem to. But to some extent that is inevitable; in any company, the IT department is ultimately responsible for the well-functioning and continuity of the company itself, not the well-being or ease-of-use of the individual user or even the individual project, and that stance can be inherently problematic.
Which is not to say that everyone in IT is going to ascend bodily into heaven, sure, but we spend our days wrangling complicated, frequently opaque systems subject to an obscene variety of unsavory threats. The consequences of a misstep can be fierce and doing your job just right often means nobody ever notices, a fact that can warp your perspective a bit.
But it’s not that we don’t love you, users. Honestly, we do. We hate a lot of these things you complain about as much or more than you do, but the laws of physics and economics look very different in the server room than from your home office.
The canonical example is, why can’t I have more storage space? Why do I have to limit myself to five hundred megabytes on the network drives, why do I need to clean my mailbox out so often when I only have a few hundred megabytes of mail in it, how come you’re always asking us to delete our old mail when I can go down the block and buy a terabyte of drive space for a hundred bucks?
The short answer is that our equipment is really expensive. Our drive space, for example, costs somewhere between fifty to one hundred times what yours does, and it’s lot more complicated than just “buy a drive and plug it in”. And that fifty-to-one-hundred not an exaggeration; that one terabyte is four SAS drives, which cost about 15x what you think they should on their own. And they need drive bays on the NAS to live in, that may or may not be there. Will we be able to just spin that drive up or would we need to rebuild the array? Can we dynamically resize those partitions? Maybe not. And do you have several spare terabytes of extra space on the tape backup system, for redundant offsite copies? Do we even have enough slots in the tape robot to put more of those tapes into it? How many times can we add drives before we need to reconsider the amount of power coming into the server room? Buy enough of those, and do we need another UPS to power them all when the building power browns out? What do those cost?
That’s the process, and it sucks, but it’s also the only way any of this can be made to work reliably at all. Our gear doesn’t cost that much for no reason; it costs that much because having our whole company sitting on their thumbs with no email costs so much more. What does this change force us to also change, and what does that cost? Maybe a lot, and if we don’t have the budget for all of it we can’t do the first bit and we end up saying no. So we say we can’t give you an extra gigabyte of space on the mail server because it’s too expensive and you go back to your office thinking those guys in IT are a bunch of goddamn lazy cheapskates. And I understand, ’cause that’s sure what it looks like.
But that extra terabyte of drive space might not solve the real problem; the mail you want to save. Did you know that Exchange data stores can only be a certain size before performance starts to seriously degrade? Yeah, it surprised me too. You can set up multiple stores to keep those numbers down but it’s recommended you do that across different storage devices, so we need new servers for that, and then we need new OS licenses and CALs and if it seems like I’m saying it never ends that’s because it’s pretty much like that. These weird, seemingly arbitrary restrictions are all over the place (Microsoft, I’m looking at you) and working with and around them is definitely nontrivial.
And we’re on a budget, and the economy is in the toilet. And you want, but your department never seems willing to foot the bill, curiously. And it doesn’t help when we do things like check timestamps or file sizes just to see what our actual usage looks like, and find out that the reason you need more drive space is that you won’t delete your mail from 2002 (that you haven’t touched since 2004; yes, I can check that) or get rid of that pirated .ISO you’re saving or the porn or MP3s that shouldn’t be there anyway. Because (and this happens all the goddamn time) first of all, if you’d just done a little housecleaning you’d have plenty of room, and second now that I’ve seen that I need to talk to HR. And if it comes to that I guarantee that the story that gets around will be that you made a simple request to IT for a bit more drive space and the Distant IT Overlords had you escorted to the curb. That’s not what’s happened, sure, but in the absence of respect fear works just fine.
And, and and. And all of that is most basic example I could think of, and I’m still glossing over the uglier technical bits.
The most amazing thing about all this is that it’s not only possible for me in IT and you in userland to hold hands and find our way through this thicket together, but it’s possible to do it remarkably well; user expectations can be managed and met, networks can be disciplined and machines secured, management can be made content and all of that done while the organization grows and IT cooperates with users in nurturing it along. But it can’t happen case-by-case, and it can’t happen without resources that may be more scarce than you realize.
So here’s the thing: if you want more disk space or a new browser or anything from IT at all that isn’t a correctly-functioning version of the status quo, don’t ask me for it. Not because I don’t care, and don’t want people to have nice things, but because IT can’t do what you want for individuals users, for free. Go to your managers, and their managers, and my managers. If you have management onside then your problems are my problems, but more importantly I will have things like budgets and project milestones and all these things we need to make changes to a complex environment without boning the entire thing.
Sure, it will take time. But if your argument is compelling (modulo the resources it requires) you will either get what you want, or have a clear understanding of why it’s not going to happen. But you need to step back from the hardware far enough to understand that any complicated system is ultimately political; it’s about negotiating, weighing resources against priorities and carefully crafting plans towards possible futures.
You also need to step back from the hardware because it’s not your hardware, it’s mine, and you don’t know how it works and I’m possessive and don’t touch it. Which is to say that yes, in addition to being political it’s also personal, and you’re going to have to sell your idea.
Which is all to say, help me help you. I know, the Other Kind of IT Guy is out there, and he’s a bit of a dick, but he’s relatively rare and a management issue too. But if you sneer and think this simple and that I should just let you do whatever you want, that tells me enough about what you think of me and my job that you’ll have a hard time convincing me I should help you out.
And that guy we had to escort out of the building? Yeah, don’t be that guy. I’ve got my Distant IT Overlord boots right here.