The little things…

June 1st, 2012

It always seems that it’s all the little details that trip us up and cause the biggest problems. In life and in systems administration.

Well, it’s also those little things that have the biggest impact on others, and do the most towards solving problems and making life better. Yesterday, I was given “ownership” of our corporate backups. Ultimately, this just means that I take the blame for any major casters-up events, but it also means that I can make changes where I deem appropriate.

One of the things about the backup environment is that it send out a report every morning regarding the previous day’s backup runs, which has to be checked for errors. The report is generated by doing nothing more than running a series of commands against the backup database, so it reports all the jobs in chronological order based on start time. While this makes perfect sense, it is rather frustrating to have to page through a 2000+ line email to try and determine if there were errors. It’s far too easy to overlook the one character that has the numeric job status when you have 1,998 zeros and two sixes in that column.

Well, computers are great at repetitive tasks like “check each line of data for a non-zero value in this column”, so I decided to do some slicing and dicing. I brought out an old and trusted friend, awk, and told it to find me all the non-zero values and report them to me, then to find the machines those values are associated with and report on all jobs on those machines. Then I had it put all that information at the beginning of the nightly report email so I don’t have to scroll through the huge 2000+ line report to find the anomalies. I went ahead and let the report script put the whole big thing at the end, just like it wanted to, to avoid making it jealous of awk and getting all whiny and constipated later, but I really do like the ‘awk-ed’ aection better.

This morning, the other person whose job it is to go through this email and find the misbehaving systems came by and thanked me for making his job easier and his life better, since he now only has to spend about 30 seconds reading this email versus 5 to 10 minutes previously.

The Devil may be in the details, but often so is Salvation.

Reflections on a new job

May 25th, 2012

At the end of my third week in the new job, I’ve come to a couple realizations. Most recent is that I’m having to watch the clock closely so that I leave on time. I’m a contractor, so I’m hourly, not salaried. That means it’s all too easy for me to start putting in overtime at a job I enjoy, and this i just such a job. Problem is, the manager here isn’t allowed to give me overtime without the approval of his higher-ups, and unless something major goes bonkers that approval probably won’t be forthcoming. This sort of bugs me, because I’ve already found myself wanting to stay later and get something finished, but it’s ultimately not that big an issue, I just have to budget my time closely.

I’m also re-discovering why I went in to IT in general – and system administration in particular – in the first place. I’m having a large influence on how things are done already, making people’s lives easier and helping them sleep better at night. At least, I think I am, which is much the same thing as far as my enjoyment of the job is concerned. I’ve also already been given opportunities for input that my former boss would have never given me, and made decisions he would have never approved of simply because he didn’t understand the logic behind them (or was too caught up in having to appear in control of everything to actually let anyone else get any work done).

In future posts, I may detail some of the automation stuff I’m putting into place. It’s not really all that ground-breaking; honestly, it’s somewhat pedestrian and rote, simply because there are so many other places that have done similar things before. Still, it’s all my stuff, customized specifically for my new job, and I take a great deal of pride in the fact that not only did I put it into place, but that it’s been well-received by the admin team here and that it has (so far in it’s limited lifetime) worked quite well. Not flawlessly, but quite well.

I’ve still got a good bit of work to do to add functionality I want to add to what I’ve already put in place, but I’m going to launch by next big initiative next week, addressing a completely different topic than workflow automation. It’s something I see as sorely needed (again) for the future of the environment, but it’s significantly larger and more invasive than the simply installation automation I’ve done so far so I don’t know how well it’s going to fly. I have faith that the people here will see the need for it and the logic behind it, I just don’t know how easy it will be to actually make the necessary changes to put it into place. I’m hopeful.

Wow. That’s something I haven’t been able to say about my job for a long time. It feels really good to be able to say that.

Formalizing an Infrastructure

May 17th, 2012

When I started the new job, I knew I would have a lot to learn. I have learned a lot, and I still have a lot more yet to learn. One of the things I have learned is that there is very little in the way of formalized process-oriented infrastructure here. System builds are done effectively the way they were done at my last job (which I wanted to fix but wasn’t ever given time or permission to fix) – which is to say, by hand with an instruction sheet nearby.

Lots of things are hand-copied around, or schlepped around using a script that does the equivalent of an scp then an “ssh root@<server> mv”. Formalized? Heh. Process-oriented? Right. Auditable? I can hear the paramedics yelling for defibrillators now, and the auditors haven’t even gotten here yet.

One of the things I wasn’t sure of was how many waves the company wanted me to make. Should I sit down, keep my mouth shut, and do it their way? Should I offer small changes to establish myself, then sit down and shut up? Or should I go all-in and make as many waves as necessary to fix things up and get them running right? In other words, am I supposed to be a quietly competent consultant, or a bombastically loud deity of a consultant?

Well, neither – they’re both too extreme. But much to my surprise, the company wants me to make waves. Lots of them. They want me to discuss things first, but if I see something broken, they want to know of my desire to rip it apart and fix it, and so far they have let me rip away. There’s one bit I’d love to rip out completely and redo from scratch, but it’s far too ingrained in too many areas for that to be an easy extraction, so I’m letting it stay in and just going to open a longer-term discussion about it.

But oh what glorious fun it is to tear apart and rebuild an entire infrastructure… especially when I’m so good at it, and it’s ultimately so easy! Yes, I said easy – it’s complex, but the complexity is due to size of the task, not difficulty. Hot diggity, I’m back at work!

My game is back…

May 14th, 2012

… and I am on top of it once again.

I’ve been at the new job for a week now – today starts my second week. I’m really enjoying this new gig so far – I’ve already had a pretty big impact on how RHEL installs are done, and that project isn’t even as fully complete as I’d like yet. I’m already comfortable with the new way of doing things, and I think I’m about to get official blessings to roll out a new package build system, which in turn will get integrated with the Satellite for package deploys. That will take a few weeks to complete, and that’s just the beginning of what I want to do here.

Not only that, but my coworkers, and in particular the lead UNIX admin here, have already repeatedly demonstrated just how much better they are at both technical and interpersonal matters than my last boss. I made a typo yesterday that took me and the lead admin about 30 minutes to figure out, and he told me he would “correct the problems caused by PEBKAC”. Not only does he know what PEBKAC means without having to look at the (mostly erroneous) definition in Wikipedia or Urban Dictionary, he actually uses the term correctly! My last boss (supervisor and boss, really – and yes they were separate people) had no idea what the term even meant, then when I explained it to them, and explained how I used it, told me I was wrong. Umm, excuse me, I am not wrong about how I used it, you are wrong for not knowing popular well-known slang from the industry you’re in.

I really am in a way better place now than I was a month ago.

New beginnings

May 9th, 2012

Well folks, it’s been a bit since my last post. In the interim, I have left my last job (thankfully) and started a new one.

Why thankfully? More on that later.

The new position is really awesome. The guys I’m working with are on top of their game, and although I’ve already come across some things they do in certain ways that confuse me, sitting on my initial reaction has let me realize that the way they do things makes sense. It’s not the way I would have set things up, but it works – in some cases, better than the way I would have set things up – and they are really good. I see a couple ways I can have a big impact, and I’m really looking forward to it. I’m not at all interested in becoming the next heir-apparent, or even some sort of team lead – I’ve been down that path and I’m not sure I want to go down that path again – but I do look forward to being closely involved and working with people who I respect and who demonstrate that they know just as well as I do how things should work.

So why am I so thankful? Well, aside from what I’ve just mentioned above, my last manager was… significantly less than fully qualified on a technical basis. Let’s just say I consider him to be a perfect example of the Dilbert Principle and leave it at that. My new manager is proof that every rule – including the Dilbert and Peter Principles – has exceptions, and I couldn’t be happier about it.

Hopefully I will be able to update this blog regularly, although if you go on my previous performance, you probably shouldn’t hold your breath.

For now, goodnight!

Bit of a rant

January 25th, 2012

I’m going to rant a bit about a bad support experience I’m having with VMware. This is not the first time I’ve had this same problem with VMware Support, nor do I expect it to be the last. It seems to be a characteristic of VMware’s support organization that on the rare occasions they are actually attentive, they are unable to actually offer solutions for my problems.

So what’s the problem this time? Quite simply, VMware is not reading my tickets before their support factory spits out a canned response routine. The first thing they do is ignore the contact preference field that they asked me to fill out on their web form to submit a help request. I had to submit four requests today for separate snapshot-related problems, and on three of them, the assigned support rep called back. Unfortunately, my organization hasn’t seen fit to give me a direct, unshared, fully functional incoming phone line, so I can’t really use the phone contact method. (That is a separate issue that has been a major problem for me for 3 years, but one which I’ve given up hope of getting resolved.) I specified on all four tickets that my contact method preference is email. I really wish their support organization would pay attention to the fields I fill in that they ask me to fill in on their support page.

While I’m ranting, let’s back up just a bit. When I log in to their support portal, I’m at first shown that I have no support entitlements and no open tickets. I’m given the chance to look at the knowledgebase, but that’s about it. I have to log in a second time, to the same support portal, with the same username and password, before they deign to acknowledge that I do in fact have support entitlements and open support requests. It has been that way for 6+ months, and yes I notified them of it 6+ months ago when I first noticed the issue.

Well, on to the tickets I filed today. One of them, the one where the rep actually paid attention to the contact preference field, gave me a kbase reference which, much to my surprised and delight, actually resolved the issue. A second issue has also now been resolved after the technician tried (and failed) to reach me on the phone and finally decided to actually address the issue via email, as I had originally requested. A third issue is still waiting on a response from the technician after I had to correct his re-interpretation of the error message I had given him verbatim in the initial ticket. He did ask for some additional information, but he also asked a question of me which had already been answered by the information requested in the initial web form.

Why does VMware seem to employ support representatives who not only do not speak fluent collegiate-level English, but who insist in ignoring the support requests I file in order to find the closest easily solvable problem to my actual problem? I’m sorry, but that closest problem isn’t the same, and it’s solution does absolutely nothing to help with the issue I asked your organization for help with initially.

Before anyone goes and excoriates me for not understanding technical support, let me state that I was a technical support representative for Red Hat for two years. Based on that experience, I am recommending to VMware that they move even their basic support representatives back into the United States and/or initiate a global rolling-center model. While I was at Red Hat, we found that the outsourced model did not work – as it manifestly does not – and established (at the time) three major geographic centers, staffed by individuals who spoke the languages of the countries in their geographical regions fluently. I have no problem with VMware establishing a large presence in India or other overseas locations. I have a very large problem with VMware not providing me with competent fluent English-speaking support representatives who actually pay attention to the fields in the tickets that get filled in by the selections I make in their web form.


October 5th, 2011

… things wouldn’t go right if you paid them.

Off Topic: Brewing

April 7th, 2011

So I’ve been doing some thinking recently about a career switch – I’ve always thought about it in an eventual sense, this time I seem to be taking myself a bit more seriously.  It all started with my homebrewing kit, and the reviews of my beer I’ve gotten.  I’ve been tossing around ideas in my head for opening up a brewpub, or maybe a packaging brewery though I’m leaning toward the former.  I’m nowhere near any point where I need to look for capital, or even look for sites, but the more I think about it the more I think I could make it happen.

So at the risk of letting the cat out of the bag too soon, here are the ideas I’ve had so far – with suitable thanks to all the people I’ve bounced the off of for refinement and (when necessary) scrapping.

Idea 1: themed beers.  It started with a book I read…  then I divided the beer world arbitrarily into four categories – ale, lager, barleywine, and mead.  These categories correspond with my division of the ancient mythological world into four “western” pantheons – Greek, Egyptian, Roman, and Norse, respectively.  Each variety of ale would be named after a Greek mythological figure, lagers would be Egyptians, and so on.  The name of the beer type?  Mythobeer.  Or maybe Mythbeer.  Mithbeer?  All those spellings look a bit…  wonky.  That’s one area I’m still actively thinking about – well moreso than other “settled” areas, that is.

But a brewpub doesn’t just make beer, it also makes food.  And serves food.  So we start with the basics – lunch and dinner.  That’s much is a definite, this next bit is still under evaluation.  I could split the building into a “brewpub” and a “restaurant” area – much like Tyler’s Taproom (in Durham, NC) is split into a pub and a speakeasy.  The brewpub would have simple fare – burgers, dogs, sandwiches, along the same lines of Tyler’s or Ham’s (before it went out of business) or Carolina Brewery (yes, I’m using examples local to the RTP area of North Carolina because that’s the area I know).  The restaurant side would have a more upscale selection of fare, similar to Top of the Hill in Chapel Hill, maybe even something closer to a Ruth’s Chris or a Sullivan’s, but not a steakhouse.  Then I was told I should consider (very seriously) doing breakfast, as that is the cheapest meal to serve on a per-plate basis; so combine that with another idea of doing a coffee house, I turn the brewpub into a coffee house and the restaurant into a breakfast restaurant before 10:30 am – blend from breakfast to lunch between 10 and 11, and depending on interest level perhaps have “breakfast” available until 3 or 4 in the afternoon, if not all day.

The problem with all of this?  At least, the first major problem, anyway?  Capital – I don’t have any.  So, as I put more thought into this, and as I start planning things out, I need to also be thinking very hard about where to get capital from – traditional bank loans? SBA loans? Venture Capitalists?  All things I need to ponder…

Gratuitous insults and other tech annoyances

March 4th, 2011

On Wednesday of this week I received what was easily the most offensive email I’ve ever gotten in my career.  It was a one-sentence email written in response to my creation of a new virtual machine which was a direct result of a programmer’s request for said virtual machine.  It said,leaving out the salutation and signature areas:

Please get <redacted>’s permission before creating new VMs, volumes and services.

What that means is that I’m no longer allowed to do my job.  Or at least significant portions thereof.  I’ve been reduced to the equivalent of a level 1 helpdesk weenie.  I am being micromanaged.

Today, it got worse.  Yes, believe it or not, there was a way in which it got worse.  The gentleman that sent that message now claims he is merely asking me to follow pre-established policy regarding system maintenance, a policy which I have been following, but which (the last time we discussed it) specifically did not cover routine things like this.  Why?  Because this isn’t really a system maintenance – at least the simple addition of a Windows XP virtual machine isn’t – it’s sort of like adding a PDF file to a wiki page.  Something that’s done all the time as a routine, regular, understood part of the job.

And yet today, the person I’m supposed to get permission from couldn’t even explain, at a third-grade level, how a hardware-based firewall is supposed to work.  At all.  And this is the joker I’m required to get permission from in order to do my job?

I need a new job.

Bang head here

February 20th, 2011

So I’ve gotten pretty good with basic iPhone programming.  Or at least, good enough to pass as semi-qualified.  I’ve managed to get an app in the App Store and it’s selling about a copy a day so far – not a huge amount of income, but probably a lunch or two a month.  I’m fairly happy with that, since this is the first app I’ve done specifically for myself, from my own idea.  I’m working on another one now that I’m hoping will have a slightly broader appeal, but I’m still not expecting it to go gangbusters.  I am, however, earning a few more things about programming that I didn’t necessarily expect to – and thanks to Admiral Frijole (yes, that is a nickname), learning a lot more than I expected to about interface design.  See, he’s able to think like Apple (though he’ll tell you it’s thinking like a user of the iPhone, not like Apple), and has pointed me in a few different directions to make the interface not just better, but completely different from anything I would have thought of on my own.

The hardest part now is figuring out how to go from sketches / mockups to the code – I’ve got a few ideas, I just need to see which one(s) work.

In the meantime, I promised a practical demonstration of doing things in code as opposed to in Interface Builder (IB), so let’s look at how I built the initial tab environment for the app that’s selling now.  First, I start with the fact that I need a TabBarController in order to make the tabs at the bottom easily, so I make the header file for the AppDelegate look like this:

#import <UIKit/UIKit.h>

@interface Brewinator_Pint101AppDelegate : NSObject  {
     UIWindow *window;
     UITabBarController *tabBarController;

@property (nonatomic, retain) IBOutlet UIWindow *window;
@property (nonatomic, retain) IBOutlet UITabBarController *tabBarController;


Now that we have the basic TabBarController defined, we simply need to built it out. I’ll be straightforward and tell you now that I’ve left some of the actual code out of this sample code that have no bearing on the interface, so the code you see will get you a pretty interface but not much else. In the applicationDidFinishLaunchingWithOptions() function in the AppDelegate.m file, I added the following code:

    UINavigationController *localNavController;

    tabBarController = [[UITabBarController alloc] init];
    NSMutableArray *localControllers = [[NSMutableArray alloc] initWithCapacity:2];

    TypeViewController *typeViewController;
    typeViewController = [[TypeViewController alloc] initWithTabBar];
    localNavController = [[UINavigationController alloc] initWithRootViewController:typeViewController];
    [localControllers addObject:localNavController];
    [localNavController release];
    [typeViewController release];

    NameViewController *nameViewController;
    nameViewController = [[NameViewController alloc] initWithTabBar];
    localNavController = [[UINavigationController alloc] initWithRootViewController:nameViewController];
    [localControllers addObject:localNavController];
    [localNavController release];
    [nameViewController release];

    tabBarController.viewControllers = localControllers;

    [localControllers release];
    // Add the tab bar controller's view to the window and display.
    [self.window addSubview:tabBarController.view];
    [self.window makeKeyAndVisible];

A few things to point out here. The TypeViewController and NameViewController classes are defined elsewhere; the header files for those classes need to be #imported in the AppDelegate’s .h or .m file appropriately. The initWithTabBar() routine called on each of them also needs to be defined and implemented properly inside each ViewController’s .h and .m file respectively.  If you want more than two tabs, you will need to initialize the localControllers array with the appropriate capacity and then repeat each ViewController “stanza” once per desired tab.

Once those are taken care of, you should be able to build and run the code in the simulator, and it will give you an interface with two tabs at the bottom that you can switch between.  Until you define the interface in each ViewController’s loadView() function, they won’t really look like much, but that’s a post for another day.

For now, enjoy your tabbing display!