kentcowgill.org/blog

I'm struggling with some major frustrations at my job.

I recently learned of a workaround for a race condition that has been an annoyance for some of my coworkers for nearly a year. Pointing out the silliness to my manager was a bit fruitless - she quickly turned around and assigned the 'fix' to me.

So I took a look at the code.

It was a mess.

At least 27 different files call this code, so to properly refactor the code, all of those files would have to be changed and tested.

But there's no tests.

There can't be tests⁶. Why? Because these files are a mishmash of perl and HTML in a custom templating system which are sucked in and evaluated 6 ways to sunday on every web request. Files include other files, variables are set in weird objects, namespaces are a giant jumble, etc. so forth and so on.

And the people who originally wrote this code have left the company - my guess is in frustration. They were trying to rearchitect the code, but it was taking too long, and so were told to just hack it to make it work.

But surprise, there were performance problems, because the dynamic data originated from flat files on the filesystem which were parsed multiple times per page request. And this custom template system is running under mod_perl - why not make some persistent data?

So I'm writing an email to my boss, and struggling with whether to include the following:

“ I don't think further code reviews for $WEBSITE are worthwhile, since we lack manpower to make any changes. Further, since we don't have any tools to perform at least unit if not functional tests, it's impossible to refactor existing code without being certain nothing was broken in the followup.

Due to the lack of manpower and lack of morale in existing manpower, nearly every IT project will eventually fail abysmally and become completely unmaintainable and unscalable. $COMPANY will continue to struggle along and churn through developer after developer - assuming we are allowed to hire some more to replace the ones that have left in frustration. ”

But I don't think I can do it.

Would you?

⁶ Actually, I've been trying to find free time to finish the Test:: suite I've been developing, which is currently capable of splitting the perl from the HTML, evaling the code in its own namespace, running Perl::Critic and Devel::Cover on the code and HTML::Lint on the HTML, but I'm having trouble with the recursive templating creating new variables in the existing namespaces. Also since I lost half of my team members (including the team leader), it's a little tough to find that free time.

I keep getting reminded of the story about the broken window I read recently in a great book I've purchased called The Pragmatic Programmer.

Most recently was this set of conversations between a coworker and myself, and my manager and myself.

Coworker Y: "Kent, can you please run sudo command xyz on the production cluster?"

me: "Sure, but only if you promise to fix the race condition that causes condition X on the production cluster."

Coworker Y: "I would, but it's not on my platter."

me: "Whose platter is it on?"

Coworker Y: "Coworker Z."

me: "This has been a known issue for nearly a half a year, hasn't it?"

Coworker Y: "Actually, about 9 months."

... time passes ...

phb: "Kent, does sudo really need to be run for command xyz? Can't you just¹ do it with normal user privileges?"

me: "I assume so - they wouldn't ask me to do it otherwise, no? BTW, the cumulative time taken to run or have someone else run [sudo] command xyz every time this happens is probably much greater than just fixing the race condition that causes condition X, which as I understand has been a known issue for at least 9 months."

phb: "They tried to fix the race condition and failed. But I didn't know coworker Z had been running sudo command xyz all along, I thought he had another way to deal with it. Until he comes back from vacation, can you take care of these requests?"

me: "I'd much rather the race condition get fixed."

phb: "I agree, however coworker Z is out of the office this week and coworker Y is swamped -- we will have to wait."

me: "Coworker Z hasn't been out of the office for 9 months, though."

phb: "Coworker Z and coworker W tried a couple of times and could not get it fixed."

me: "Did they exhaust the entire technical resources of our company?"

phb: "No one really had lots of time on their hands. If coworker Z tells me again they can not fix it, I will hand it over to someone else for sure."

I really wish conversations like this didn't happen.

I really wish that more people at my job cared more about what they did.

Why did coworker Z and coworker W give up?

Why didn't they ask anyone for advice?

Sure it doesn't take much time to run sudo command xyz, but how much time is lost from breaking your concentration, switching contexts into "firefighting" mode, running the command, and trying to pick back up wherever you left off? And then what if you're so distracted by this craziness that you have to write a blog post about? How much time gets lost then?

I really wish people would care more.

¹ Andy Lester pointed me² to a great rant of his about this very phrase "can't you just...".

² And a second article linked from Andy on complexity management (I think he's on to something here)