Information Overload

Anyone who has been in IT for more than 2 days knows that systems monitoring is important. Heck, I’m willing to bet you know it’s important even if you’ve only been in the field for 2 hours. If a system goes down, you need find out about sooner than the customer, because when the customer finds out, you want to be able to say “Yes, we’re addressing the situation now.” Especially if the customer is your boss.

There is, however, a trap you can all too easily fall into – monitoring too much. Or at least, giving your sysadmins too much information from the monitoring system. Some of this is inevitable, especially when first setting up a new monitoring system. Some of it is politically driven, and you will get alerts for log messages that really don’t mean diddly in the grand scheme of things, because auditors want to see green lights turn yellow when someone mis-types a password. Much as it pains me to admit, these are some things that you just need to learn to live with.

What you should be careful to avoid at all costs is alerting people when things are working. Especially sysadmins. Quite often, the sysadmins don’t care – at all – if things are working as expected. A typical response: “So what? I’ve got these three fires to put out, five systems to build, four developer questions, and eight help desk escalations to deal with. Could you please go away and let me do my work?”

Okay, so obviously nobody will come up to you and congratulate you when your systems are working – your boss might show his/her appreciation somehow, but that will be at a group lunch or during a staff meeting or something where you’re already blocked off from doing useful work. So what’s the concern here? Email. Automated script email output. When you put a monitoring script in place, do not, for the love of the little people, let it send email to the entire world when it finishes running and all it’s checks have been passed. Have it sit there silently and quietly. Only when it notices a failure should it make it’s presence known. If it talks too much, your sysadmins will start to ignore it – just like the little boy who cried wolf. Then when there is a problem, that signal gets lost in the noise of the chattering script.

One Response to “Information Overload”