Thoughts of a SysAdmin

Why is it you can splurge on a hotel room for $260/night for a conference, yet you can’t replace the 9-year-old desktop-class machine that I do all my development on?  You know, the development of things like the password change page that the entire department and all external users rely on?  The evaluation of new paradigms like a new LDAP structure?  Like LDAP failover / mirroring?  You know – the important stuff that keeps the whole infrastructure running.

Cripes, I didn’t even spend that much on a hotel when I was a consultant traveling to New York City and staying in Manhattan!

You know, when you’re supposedly a senior systems administrator with 15 plus years of experience, you really ought to be able to use Google effectively.  When you come to me with a problem you’ve been working on for several hours, and I figure out it isn’t really a problem but rather expected (and correct) behavior in under 20 seconds with a very simple Google search pattern, you tend to undermine my faith in your ability to do anything else with any competence at all.

For those of you who haven’t read the previous “nightmare” entry at http://sysadmin.ncphotography.com/2010/09/07/a-nightmare-before-friday/, this is a continuation of that saga.  You may be aware that a prestigious cancer researcher was recently demoted and had her salary cut by almost 50% (http://www.databreaches.net/?p=14479, http://www.databreaches.net/?p=14547) for being negligent in securing her project’s data.  This situation is arguably worse – the idiot programmer in question actively handed out the hostname, username, and password to the entire world.  His argument will be that he locked down the database to UNC-only connections, but let’s be honest here – it’s trivially easy to walk into one of the UNC libraries, find an open port, jack in, and sniff for an available IP address.  Heck, all you really need to do is find a weak machine somewhere and exploit it, then you’ve got immediate access to this database through a bounce-box.  You could even just spoof the IP you’re coming from – it really is just that easy!

Well, in light of the mammography study issues, I reiterated my concern to my boss.  The response I got back was that he has raised the issue with the programmer, and been told that it’s not an issue.  And nobody’s concerned!  Not a one of them are worried at all about having hostnames, usernames, database names, and passwords out on the Internet available anonymously from SourceForge to the entire world!  I’ve raised this issue with the departmental security person (he’s at the “dean” level of things, so fairly high up) and with my boss, and nobody wants to do anything about it.  Even after the mammography thing, they don’t get it…

I am out of options.  I can do nothing further to warn them of this disaster waiting to happen.  I feel like I’m standing alone on the borders of the Roman Empire, shortsword in hand, watching the Goths mass just beyond bowshot, listening to the banquet the rest of the legion is partaking of and telling me to stop worrying, that no group of natives could possibly breach the fort’s defensive walls.  Hearing the manongels being built, but not able to see them, not able to make anyone believe.

Why are so many people in the computer industry (be it corporate or academic) so blindly incompetent?

Not all of them involve making mistakes.  Some of them, and my most recent one, involve realizing just how much of an ass you’ve been in previous jobs by coming face-to-face with the same type of behavior from someone else in your current job.  Yes, this just happened to me, and I almost immediately realized I’d done the same type of thing in previous jobs.  Fortunately, I don’t think I’ve done it too often, but often enough to be embarrassed about it.

This is definitely something I’ll be keeping in mind as time goes on…  with luck, my new business venture will work out and I’ll have the opportunity to do something about all these little lessons I’ve learned over the course of my career.

When one of your people leaves the company / institution for another job, and that person was doing programming work that a second person in your group had expressed interest in, it is generally a poor management decision to retain the first person as a consultant (at, of course, highly inflated consultant rates) instead of moving the second person into the slot (even temporarily) to fill the need.  When someone leaves a company, the separation should be complete, signing a consulting deal with the newly departed simply rewards those who leave by giving them (in effect) a doubled salary and punishes those who stay by not allowing them the career growth they have ben requesting for the past 8 months plus.

Very poor decision.

I’m not really sure how to put this, so I’ll just lay it out straight.  You know how most organizations that sell computer hardware or software also sell support?  Well, this is for a reason.  When you’re about to buy $350,000 worth of hardware from one company, please don’t start talking about cutting out the $90,000 in support costs, purchasing no support, to throw that money at additional hardware since you already have support for some other hardware platform from some other company.  Even if both hardware platforms do roughly the same thing.  Just…  don’t go there.  Please.  The people that have to make things go want to have the support in place.  Really.

… is actually Master of most.  Unfortunately.  I am a systems administrator.  Let’s look at what I’ve done today…

- Discussed the pending breakdown of compute environments in another organization, trying to figure out how to mitigate said breakdown

- Buttered up a couple of sales people, playing the make-nice-nice game

- Replaced the staple pack in the copier

- Tracked down a missing shipment – well, started trying to, anyway

- Planned out a few projects on the whiteboard, but haven’t gotten clearance to initiate them yet

- Sent miscellaneous emails

Notice, nowhere in here do you see anything that deals directly with hands-on-keyboard sysadmin things.  What I wouldn’t give to be allowed to be the sysadmin I was hired to be.

I’ve just discovered a problem.  Specifically, something that I feel is a very major breach of basic security practices, leading to a situation where significantly confidential systems information is available publically on the internet to anyone, without need for any identification.  A fairly major security vulnerability, potentially a violation of not just my workplace policies, but of certain statutes that carry significant consequences if violated.

My nightmare is thus – the individual responsible for this violation is the “fair-haired golden boy” of the group, and nigh untouchable.  I have raised several issues of similar, if much lesser, import previously with both my supervisor and my boss, and have been told in no uncertain terms to “just leave well enough alone” (not an exact quote, of course).

What would you do?

Be Smart, Don’t Be Stupid.

In other words, don’t screw around with production systems in the middle of the day with no notice.  If you need to reconfigure a cluster environment, do so, but make sure you send out notification.  Don’t just do it blindly in the middle of the day with no notice and then disappear when the people using the cluster appear to say “yeah, all my jobs got killed, and qlogin no longer works for me…”

When I’m on it, do not call me asking me about piddly little crap that doesn’t much matter in the long run just because a programmer has his knickers in a twist about his programs being too slow.  I. Am. On. Vacation.