The little things…

It always seems that it’s all the little details that trip us up and cause the biggest problems. In life and in systems administration.

Well, it’s also those little things that have the biggest impact on others, and do the most towards solving problems and making life better. Yesterday, I was given “ownership” of our corporate backups. Ultimately, this just means that I take the blame for any major casters-up events, but it also means that I can make changes where I deem appropriate.

One of the things about the backup environment is that it send out a report every morning regarding the previous day’s backup runs, which has to be checked for errors. The report is generated by doing nothing more than running a series of commands against the backup database, so it reports all the jobs in chronological order based on start time. While this makes perfect sense, it is rather frustrating to have to page through a 2000+ line email to try and determine if there were errors. It’s far too easy to overlook the one character that has the numeric job status when you have 1,998 zeros and two sixes in that column.

Well, computers are great at repetitive tasks like “check each line of data for a non-zero value in this column”, so I decided to do some slicing and dicing. I brought out an old and trusted friend, awk, and told it to find me all the non-zero values and report them to me, then to find the machines those values are associated with and report on all jobs on those machines. Then I had it put all that information at the beginning of the nightly report email so I don’t have to scroll through the huge 2000+ line report to find the anomalies. I went ahead and let the report script put the whole big thing at the end, just like it wanted to, to avoid making it jealous of awk and getting all whiny and constipated later, but I really do like the ‘awk-ed’ aection better.

This morning, the other person whose job it is to go through this email and find the misbehaving systems came by and thanked me for making his job easier and his life better, since he now only has to spend about 30 seconds reading this email versus 5 to 10 minutes previously.

The Devil may be in the details, but often so is Salvation.

One Response to “The little things…”