NOTE: This is my first submission to kuro5hin.org, and honestly it's one of my
first attempts to write much of anything in a digestible form. This is intentionally
a little rambling and it wanders away from the central topic in quite a few spots. I
did this, I think, in an attempt to convey some of the confusion this weird but
"standard" human behavior inflicts on my mind. Sorry if I stray off the topic too far; hopefully we can get a decent discussion brewing.
I've intentionally left identifying details out of this discussion; I don't want to
lose my job by annoying the management staff any further. The only product I mention
by name here is Bugzilla, a bug/issue tracking system and kitchen sink, produced by
and for the Mozilla project. It is by far the best system I've ever used, by the way,
and if anyone needs or wants help setting it up or customizing it, give me a yell.
Although I'll not admit this at the office anymore, I'm still quite fond of the tool.
Now on to the ramblings.
I'm a system administrator who works in a small office owned by a gigantic corporation.
It's a weird environment; we have our own little network with our own servers & printers.
The office doesn't have more than forty people in it at this point.
The odd part is that we're owned by and answer to a giant telecommunications company.
The office was originally a venture capital startup, who got purchased by a
telecommunications company, who in turn got bought out by another bigger telecom.
When I started, we were a traditional dot.com fairy tale; a bunch of hackers with
a nifty idea, albeit an ultimately stupid one, with a couple system administrators
around to support the whole mess.
This of course changed as the big companies bought us out. It was great at first;
we had big budgets, lots of equipment, great working hours, etc. Oddly enough, I
didn't reap many of those benefits; I was expected to be there by 9:00am and not
to leave until 5:00pm, like a "normal" person. I never got any new hardware either.
All the boxes that currently run our internal stuff have been in service at this office
longer than I have.
It's strange to try to express the kind of environment this created. Even moreso
to try to explain how I came to be in the irritating position I find myself in now.
At first, I was an assistant of sorts. My role was to help the other admins.
When they quit (all of them, nearly simultaneously), the role of primary administrator
was dropped in my lap. It wasn't long before I was given the reigns and was afforded
quite a latitude in decision making when it came to the systems I managed.
A system wasn't working right? No problem. Build a new one to replace it, then gut
the old box to find out what went wrong. I stayed on top of security announcements
and patched systems the instant fixes were released. I eagerly helped other people
do the same to their Windows boxes while I maintained all the Solaris systems that
purred in the background to support them all.
The best part about that time, which seemed to end about nine months ago, was that
I had most everyone's respect. I was the guy everyone knew they could turn to when
they had a problem no one else could solve. If you had a weird idea and wasn't sure
how to implement it, or even if you should, I was the person you'd talk to. I'd
either shoot down the idea and offer a better alternative, or I'd help get it
Whether this respect was real or perceived, it was a good time for me. I had never
before worked with so much freedom. It paid off for the office in spades as I
worked feverishly to be rid of the legacy of the admins who'd quit (and has built
some *very* strange boxes) and streamlined the systems in the office.
I did all sorts of things to improve the end-user experience there. I merged all
the different file servers, both NT and Solaris-based, into one beefier system
with plenty of disk space, CPUs, and memory to handle all the work. I reorganized
the shares to make things more logical, polling the users to see what they wanted.
I moved all the printers onto that same file server as well (Samba-based, of course),
consolidating print services and making everything more reliable and stable. I
implemented a backup system (we weren't backing things up before this ... ugh) and
fully automated it (save the actual physical tape rotations, which had to be done
Within six months of being officially handed the internal production environment,
I had turned the whole thing into a shining example of open-source to the rescue.
It was free software done right. Windows and Unix boxes of all flavors could get
to all the same shares and printers on the network. DHCP worked uniformly across
all platforms, avoided IP conflicts, and dynamically updated DNS. Secure shell
(ssh) was deployed across the entire network. NT's presence in the production
environment was reduced to a single server. Services were streamlined, made
more efficient, and things were made insanely simple for end-users.
With all these improvements behind the scenes, one might have expected (or at least
hoped for) some recognition, or at least an acknowledgement. Of course, one would
only expect or hope for this if one was a naive systems administrator, which I
admit I was.
Things started to go downhill when my coworkers approached me with a problem.
They had (and still have, despite everyone's efforts to kill it) a closed-source,
Windows-only trouble ticketing system that was completely failing to do its job.
The two people in the organization who'd pushed so hard to put the product in place
had long since left the company, and this damned thing was (and still is) the one
tool I couldn't displace.
They described their needs to me. I dug around on the net for awhile, and found
several tools of interest. The one I settled on was Bugzilla, mostly because the
thing just plain worked. It was easy to customize, easy to use, and it pounded the
hell out of the closed-source product it was meant to replace.
A low-key announcement was made that people could begin testing Bugzilla and
evaluating its use. A month or so later, it was quite obvious to everyone
involved that Bugzilla was the clear winner. It did everything people wanted,
and it seemed as if the other tool's days were numbered. Many people, including
my manager, asked that I put together a presentation to show off Bugzilla to
the entire office. I did so, and produced a tidy 30 page document explaining
how to use the system's features. Everyone was impressed, and eager to get
started using the new system.
Then, suddenly, the bubble burst. For the first time in my nearly two years of
employment at this office, I discovered there were people who *didn't* value
my opinions in the slightest. They came in the form of a manager, and a director
who managed her. This pair, affectionately known as "the twins" by just about
everyone in the office, are among the most intensely disliked people in the office.
Of course, nobody's ever said that to their faces. Suddenly, their very own
request for a replacement ticketing system became a personal war against me.
I'll wait a bit for everyone to quit laughing. I know the above sounds utterly
paranoid and insane. But I'm convinced it's the truth.
As I began pushing the migration forward (there was content in the old system
that needed to be preserved and ideally inserted into Bugzilla), I suddenly
began feeling resistance. It ultimately reached a point where these women loudly
proclaimed (to everyone) that they refused to let me "ram Bugzilla down their
Realizing early on that I would never win this war (you *never* win against a
manager unless you're one yourself ... only then have you got any chance
whatsoever), I made sure my own manager knew about the brewing conflict. What
I got from him in response seemed, at the time, to be unmitigated support. He
was all for Bugzilla. He recognized the usefulness of a platform-independent,
web-accessible open-source system for trouble ticketing. He knew it was a better
tool than what was already in place.
It turned out that much of what he said and claimed was lip service. Or at least,
that he couldn't deliver on his promises.
What was originally a simple migration turned into a nightmare. Suddenly the
twins decided that since we were replacing the system anyway, we might as well
throw every conceivable feature into the new system. Suddenly they just couldn't
live without a "knowledge base." The system being replaced *had* such a tool.
It had been used a total of (count 'em) fourteen times. It had a meager four
entries in it because nobody who used or maintained the system cared enough
to add to it.
A brief polling of my fellow employees revealed that trend would continue;
nobody cared in the slightest about a "knowledge base" tool when we had an
entire web site, maintained by tech writers and administered by myself, that
sported a full-fledged search engine.
No matter, the feature was tacked on as a requirement. The migration turned into
the most pathetic display of bureacratic nonsense I have ever seen. The two
managers sternly refused to use Bugzilla. They ordered their staff to do the
same. They clinged to their old system like a child clutches his battered stuffed
animal when his parents try to take it away.
It was decided by the brilliant management staff that we needed to form a
"committee" to examine Bugzilla versus a number of other trouble ticketing
products on the market.
You can imagine that it deteriorated from there. That all began in January.
It's July now. What's the current state of the office?
The old ticketing system is still online and in use. Bugzilla sits mostly
idle, used only by a handful of people who weren't involved enough in the
politics to care about the outcome. It's almost fallen out of use; I'm the
only one left who uses it. Only a couple of people use the old ticketing
system either; ironically enough it turns out we don't particularly need
a ticketing system at all. One would be nice, but it's certainly not worth
the money or any extended effort.
The worst of it is that the respect I'd supposedly earned from everyone has
gone. Out of this mess came a new opinion that everyone seems to hold for me
Gone are the days when I was considered a good administrator, or even a
competent one. People don't usually ask for my help anymore, unless it's
urgent and nobody else is around. I still perform the usual duties, like patching systems and fixing problems, but it goes largely unnoticed (as, I suppose, most systems administration work does).
A few weeks ago, someone finally told me (he claimed he was being "honest"
but I can't help but wonder if he wasn't being the in-person equivalent of
a "troll") how he thinks people see me.
It turns out most folks in the office now think of me as little more than a
Linux bigot. Even though our entire production environment runs Solaris, I'm
a big Linux bigot.
There's a number of problems I have with this. It may have come about because
of an effort we started a few months ago to migrate a few workstations off
Windows and onto Linux.
There were seven boxes in total that we converted. They were unimportant, and
my mistake was assuming that because the end users were the least-competent
in the office, their complaints would be disregarded as lack of experience.
I couldn't have been more wrong. Their incessant whining gave the twins more
ammunition than they could have possibly hoped for, and they used it against
me whenever they had a chance. They never did it to my face, but they took
shots at me whenever I wasn't around to defend myself.
What frustrates me about this is how fiercely these people cling to their
outdated ideas and notions. This brings me to the point of my rant -- people
are unbelievable opposed to change.
Part of the way I earned respect from my coworkers, and, ironically, lost it
again a few months later, was how efficiently and effectively I worked. In
meetings when I made presentations, people noticed how unreasonably fast my
notebook was, compared to their own.
People noticed how stable my workstations were; we had little in common when
it came to whining about reboots. When they exchanged tales of their latest
reboots to fix a crash or unfreeze their system, the best I could come up
with was whining a little because I had to give up a 20+ day uptime on my workstation to boot
a new kernel.
I handled mail faster than anyone else in the office through an effective
combination of a fast mail reader (mutt), a good mail filter (procmail), and
by taking the time to learn the tools as much as I could.
I clearly demonstrated, by deeds instead of words, that Windows was an
inferior product in every conceivable way, and that Linux was better for
almost any application. I proved the point by running Windows in a window
(thanks to VMware) with minimal strain on my workstations.
I think I began to lose that respect again when people tried to emulate me.
And this is where my logic and reason fails me. People naturally got curious
about Linux, and started messing around with it. But people didn't go about
it the "right way," if there even is such a thing.
Nobody asked me much about how I did the things I did, or what I was running.
Instead, they just locked on the word "Linux" and ran with it. Nobody knew
that I was running Debian GNU/Linux, so everyone who wanted to be like me ran
out and bought (yes, purchased, even as I offered to make copies of the bootable Debian install CDs) Red Hat Linux off their local Best Buy's
shelves. Now I'm not suggesting that doing so is a bad thing, but in this case I think it would have made more sense to go with the same distribution the experienced guy was running.
Of course, when things didn't go so well, and Linux handed them their
asses because it could just smell their inexperience, everyone turned on me.
I suppose they did it because there wasn't anyone else to be angry at. This
isn't a culture that encourages taking responsibility for one's own decisions;
it's better to blame someone else.
Instead of reaching the appropriate conclusions ("wow, maybe he's actually
skilled with this stuff, and maybe he could help me figure this out"), they
decided that Linux sucks, and that I was a bigot for supporting this obviously
All the evidence they'd seen got tossed straight out the window. Even though
these idiots had seen, first hand, that what I was using was faster, more reliable,
and much more stable than what they were using. They conveniently ignored the
fact that they made fun of their own OS of choice (Windows).
Because they couldn't grok Linux on their very first try, they chose to
cling tenaciously to their precious (but broken) Windows and publicly shun
those who used anything different.
Nowadays, I just work behind the scenes. Of course when something breaks, I'm
the first person (and the second, and the third :) to hear about the failure.
Even when I'm sitting there working on the problem, and people can clearly
see that I'm working on it, they'll still ask the same stupid questions.
It's usually one of those obvious questions ... while I'm fidgeting with my
workstation's IP address for example, because the newest beta of DHCP took
a nosedive, someone will invariably ask "hey the network's down!"
Then they'll get angry at me because I point out that the network's really
not down, but their workstation just isn't getting an IP address. I usually
have it fixed before they're done venting their frustration at me.
What can anyone do in the face of this kind of insane behavior? It's precisely
this kind of behavior that drove me away from religion; people don't behave
If I were a psychologist, perhaps I'd at least know the terminology to describe
this unexplainable need to defend one's decisions to the death.
I know most people don't like change. They like things to remain static; they
grow accustomed to life going a certain way, and question whenever something
differs from the routine.
Even if you show a person thousands of reasons why their chosen methods or
tools don't work, or could work better, that person will still cling
desperately to them. It becomes a personal war. A holy war. And above all,
a pathetic, pointless war.
Change happens whether you like it or not. When you think you've got your
entire life under control, there's always something waiting in the wings to
pounce on you.
It has always disturbed me how a group of managers can get together, choose
to fight change as hard as they can, and actually succeed in holding it off
for any amount of time whatsoever. It's this kind of behavior that sees
Solaris 2.6 running in a mission-critical production environment, despite
two newer major releases of that OS having been available for years.
The people I work with are no exception. With all my fighting energies
depleted (I've frankly given up trying to help any of these people any more),
the twins silently began blowing away the Linux installations with fresh
installs of Windows 2000. They've made a critical mistake; they took their
perceived victory in the "everybody versus Bugzilla" battle to mean that
their choices were correct. That because I've given up trying to fight them,
everything I've ever said or advocated is wrong.
What they don't realize is that changes are coming anyway. The managers at my
office have gone to great lengths to make sure I understand I won't be making
those changes, but I rest easy in the knowledge that I don't really need to.
The changes come anyway. And this time, they'll be handed down to them by an
upper manager. It won't come from somebody they can silence or railroad out
of the way. When the changes come, I know I won't be recognized as someone
who saw them coming and adapted. Instead, I'll just be one of the crowd.
I'm not used to that; maybe I'll like it.