kentcowgill.org/blog

I've been tasked with setting up new QA and staging environments at work.

With a twist.

We have several teams working on several sections of our main website. One team migrates code from development to production on a regular basis. I believe I have mentioned this before. In case I haven't, they push files several times an hour. I honestly think they use the production cluster as their QA environment, but that's the subject of another rant.

Also, we are making fairly large changes to our backend. The front-end changes for the backend changes are to initially be carried out on this new QA server.

Do you see the problem?

While developer A makes changes on QA, they don't get reflected on dev, staging, or production.

Developer B makes changes on dev, and they go straight to production. QA and staging fall out of sync.

Hence the suboptimal code migration path:

Life would be much better if we just started using version control. :(

Another mini work-related rant. The names, projects, and task names have been changed to protect the innocent.

phb: "Please work on project X, which incorporates tasks Y and Z."

me: "Hey, how about we split those up into two separate projects? Z is pretty straightforward, Y will take some doing. Y and Z aren't really related other than the fact that they show up on the same web page."

phb: "No, as long as task Y doesn't take more than a few days, please do them together."

me: "So, what datasource do you want for those Y items?"

phb: "Oh, just pick the most recent Y items per category."

a few days pass, where I do the lions share of tasks Y and Z

phb: "Oh, by the way, some categories don't get new Y items very often - we want to have the one that gets displayed change more often. Can you build in some capability to do that?"

me: "Hrm, nothing in the database lets me select different Y's based on date updated or anything. I'll have to change the database, and the module that interacts with the database, as well as the internal application system that produces all these things..."

phb: "So, we've thought about it, and let's cancel task Y out of this project."

me: "You're kidding, right? I've already done Y and Z, and let me tell you - I put in a lot of effort into figuring out Y, but making it so that Y and Z show up when a user logs in, and have the old look when a user isn't logged in, that was tough. Not to mention that not even part of Z can be used (again which I've already done), since if you remove Y, that part of Z is now worthless, too."

It started out simply enough: implement a new design which pulls in new datapoints. In figuring out where the datapoints come from, the initial guess was wrong, which was to use the most recent data items for each category. It seemed likely enough, but once the business owner was asked about the source of the data, it turns out that guess was wrong - he wanted to be able to dynamically choose which datapoint was displayed per category.

Normally, this isn't a problem - in fact, that's what web developers do all the time. However, this project was for modification of a legacy application, which in a rather complex way, displays one of these datapoints dynamically - but without being from a specific category. In order to accomodate the new request, a lot of the backend of this system would have to be changed - the database, the perl modules, the content management system, the XML files.

And of course task Z is under intense deadline pressure.

Let this be a lesson to make sure all the requirements have been at least thought about before beginning a task.

Or perhaps, if a developer notes that tasks should be separated, he probably knows what he's talking about.

HELLO?

SPRING?

WINTER CALLED, IT WANTS ITS CRAPPY WEATHER BACK.

THANKS,

-KENT

So I went to the grocery store last night to pick up a few things for dinner. I breezed through the produce department and picked out a few things to make my impending salad complete - romaine lettuce, spinach, tomatoes, and carrots. No need for a little hand basket, since I have so few items, and surely I'll be able to get out and back home quickly. I speed over to the self service checkout lines, but notice that one is out of service, and there's a line for the rest. Or several lines, even though there's a little sign suggesting that people should line up in a single line. But the self service checkout line is a different story altogether. So I decide to line up in the express lane - no more than 15 items(!) ... since there's only one woman in that line.

So I line up behind her. She visibly counts her items, I suppose to make sure she doesn't break the "15 items" rule. Good, glad she's a little conscious of ethical actions. But, her groceries are all spread out on the conveyor belt, taking up much more room than they needed to. Fine. A little rude, but I can forgive many things, including slight unintentional rudeness. The person in front of her is taking a while to pay, so I have to stand there, holding all my vegetables, no doubt speeding their eventual spoilage due to holding everything in hand.

Then it's her turn. But she produces a packet of papers or something. Interesting. Especially with that gold ankle bracelet she's wearing. But her groceries are mainly stuff for kids, or at least seemingly appropriate for a household with lots of kids in it - 3 gallons of milk, several jugs of fruit juice, eggs, etc.

Then I notice the cashier rings up three items, writes something on one of the slips of paper from the packet, punches in a long series of numbers into the register, opens the drawer, and puts the slip of paper in the drawer. Cashier rings up three more items, writes something on another slip of paper, punches in a long series of numbers into the register, and puts the slip in the drawer. I longingly look to the self service lanes, and notice the person I would've been behind in the primary line is using the self service machine. I would've been next, assuming no one decided to form their own line and cut me off.

The cashier rings up three more items, writes something on the next slip of paper, punches a long series of numbers into the register, and puts the slip in the drawer.

By this time, I've already gotten out my money, and my Jewel card, just to go as fast as absolutely possible. Did I mention I'm trying to get home to eat?

The cashier rings up three more items, writes something on the next slip of paper, punches a long series of numbers into the register, and puts the slip in the drawer.

I again longingly look to the self service lanes, and notice the person I would've been behind in the primary line is done. I would've been ringing up my own items, assuming no one decided to form their own line and cut me off.

The cashier rings up three more items, writes something on the next slip of paper, punches a long series of numbers into the register, and puts the slip in the drawer.

Finally, she's done - 15 items! The cashier asks her if the groceries behind the grocery divider are hers (no, they're mine, that's what the grocery divider is there for!), of course she says no. The cashier performs a little more bookkeeping caused by the slips of paper she "paid" with, and then the woman in front of me (and I assume her husband) finish bagging their groceries, and finally it's my turn.

I'm out in record time! Jewel card, four items, cash at the ready, bing bang boom - I'm done! I rush out to my car, and notice that the car I parked next to was in fact owned by the woman (and what I still assume to be her husband) - and they're finishing loading their groceries into their SUV. Er. Their SUV?

I get a little closer. The woman puts her cart in the cart corral as I pass by, close enough to see what kind of SUV it is. It's a Lincoln Navigator.

You know, a Lincoln Navigator. Hailed thusly: "Navigator set the bar as the premier luxury SUV". MSRP apparently starts at $46,575.

I'm absolutely floored.

Can anyone very kindly explain to me what method of payment this woman and her husband were using to buy their groceries, three items at a time, no other money exchanging hands, no paper, no plastic?

Can anyone explain to me what lottery these people have won to pay for their groceries? Is it the grocery lotto? "Congratulations, winners - you get free groceries, but you can only pay for them three at a time with these special coupons!"

I have a sneaking suspicion that in fact these slips of paper that required the purchase of three groceries at a time that requires a lot of extra time and bookkeeping on the part of the cashier and that restricts the groceries to basic necessities etc. are actually either welfare, LINK, or some other government subsidized financial assistance.

A show of hands please. What kind of government subsidized financial assistance gives people groceries for free, and yet lets them have enough money to drive around in a Lincoln Navigator?

I just noticed during that last post that I was really getting irritated by the little squeaking noise my MacBook Pro makes every time I hit the spacebar.

I seem to recall being annoyed by this when I first got it back in September or August - I don't quite remember which, but I have a distinct lack of being annoyed by it up until very recently.

Was I not typing that much?

Was I just tuning it out?

Seems that now that I'm again aware of it, I am now painfully aware of it - it grates on my very last nerve.

It also seems I'm not alone. Teddy Hwang writes about it. So does someone at nnforums. And there are other random smatterings of people complaining about it 'round the net.

Fortunately, my work office is about 4 blocks from an Apple store, so getting it fixed shouldn't be too terribly painful. But annoying.

Here's a handy tip for writing code.

If you find yourself doing the following:

That's a pretty good hint that your object is written poorly.

Instead, your object should be hash based, and return a given hash value for a given hash key. Like so:

You'd think this is fairly common sense, but I just ran across some code that not only loops through a list every time it needs a particular $thing from an $obj, but it used the exact same code copied and pasted 10 some-odd times throughout a single file (and in many other places where the module is used too).

I invite you to peruse at your leisure a great book I've been reading - Refactoring: Improving the Design of Existing Code written by the same guy who wrote UML Distilled, Martin Fowler.

PLEASE FIX YOUR MACHINE SO THAT IT DOESN'T SPIT THE QUARTER IT'S GRACIOUS ENOUGH TO RETURN FROM THE DOLLAR I FEED IT IN SUCH A MANNER AS TO CAUSE THE QUARTER TO BOUNCE AND SPIN UNDER THE SODA MACHINE, RENDERING SAID QUARTER UNRECOVERABLE.

THANKS, BYE

I'm struggling with some major frustrations at my job.

I recently learned of a workaround for a race condition that has been an annoyance for some of my coworkers for nearly a year. Pointing out the silliness to my manager was a bit fruitless - she quickly turned around and assigned the 'fix' to me.

So I took a look at the code.

It was a mess.

At least 27 different files call this code, so to properly refactor the code, all of those files would have to be changed and tested.

But there's no tests.

There can't be tests⁶. Why? Because these files are a mishmash of perl and HTML in a custom templating system which are sucked in and evaluated 6 ways to sunday on every web request. Files include other files, variables are set in weird objects, namespaces are a giant jumble, etc. so forth and so on.

And the people who originally wrote this code have left the company - my guess is in frustration. They were trying to rearchitect the code, but it was taking too long, and so were told to just hack it to make it work.

But surprise, there were performance problems, because the dynamic data originated from flat files on the filesystem which were parsed multiple times per page request. And this custom template system is running under mod_perl - why not make some persistent data?

So I'm writing an email to my boss, and struggling with whether to include the following:

“ I don't think further code reviews for $WEBSITE are worthwhile, since we lack manpower to make any changes. Further, since we don't have any tools to perform at least unit if not functional tests, it's impossible to refactor existing code without being certain nothing was broken in the followup.

Due to the lack of manpower and lack of morale in existing manpower, nearly every IT project will eventually fail abysmally and become completely unmaintainable and unscalable. $COMPANY will continue to struggle along and churn through developer after developer - assuming we are allowed to hire some more to replace the ones that have left in frustration. ”

But I don't think I can do it.

Would you?

⁶ Actually, I've been trying to find free time to finish the Test:: suite I've been developing, which is currently capable of splitting the perl from the HTML, evaling the code in its own namespace, running Perl::Critic and Devel::Cover on the code and HTML::Lint on the HTML, but I'm having trouble with the recursive templating creating new variables in the existing namespaces. Also since I lost half of my team members (including the team leader), it's a little tough to find that free time.

I don't mean to make this the "Andy Lester" week, but the company I work for is in technical debt -- there's no better way to put it.

For a little introduction on the concept of technical debt, there's a nice wiki page. But what brings it home for me are the slides from Andy's infamous technical debt, which are available on his website.

So what am I griping about?

Version control.

Love it.

Hate it.

Gotta have it.

Unfortunately, it's a big undertaking.

What we currently have in place is a script that does:

It even has enough brains to accept multiple files as arguments, even if they're in separate directories, and create the backup in a given files' subdirectory. While this (kind of) works, it's a pain for "rolling back", which I just had to do due to technical problems at our other office out of state. Especially painful when there had been changes to files that haven't made it to production, but the developer (me, in this case) is blissfully unaware of said changes, so it's not clear which version to restore.

Ugh!

Similar frustration for trying to re-restore the changes for eventual re-promotion.

But moving to a real^™ version control system is a chore. Why? Because there are 5-6 years worth of these archive files that would be nice to have as version history. Altogether, there are about 99,300 files total. And the perl program I wrote to import them into our version control system³ takes an average of about 3 seconds per file⁴. My math tells me that's about 3.5 days.

³ Perforce seems to be what the company has elected to use, even though the few times I've tried it, I've hated it. Nowhere as intuitive for me to use as more reasonable (and by the way, free) version control systems such as subversion.

⁴ Pitifully slow. But since the Perforce server is a bit far topologically, speed issues are to be somewhat expected.

I keep getting reminded of the story about the broken window I read recently in a great book I've purchased called The Pragmatic Programmer.

Most recently was this set of conversations between a coworker and myself, and my manager and myself.

Coworker Y: "Kent, can you please run sudo command xyz on the production cluster?"

me: "Sure, but only if you promise to fix the race condition that causes condition X on the production cluster."

Coworker Y: "I would, but it's not on my platter."

me: "Whose platter is it on?"

Coworker Y: "Coworker Z."

me: "This has been a known issue for nearly a half a year, hasn't it?"

Coworker Y: "Actually, about 9 months."

... time passes ...

phb: "Kent, does sudo really need to be run for command xyz? Can't you just¹ do it with normal user privileges?"

me: "I assume so - they wouldn't ask me to do it otherwise, no? BTW, the cumulative time taken to run or have someone else run [sudo] command xyz every time this happens is probably much greater than just fixing the race condition that causes condition X, which as I understand has been a known issue for at least 9 months."

phb: "They tried to fix the race condition and failed. But I didn't know coworker Z had been running sudo command xyz all along, I thought he had another way to deal with it. Until he comes back from vacation, can you take care of these requests?"

me: "I'd much rather the race condition get fixed."

phb: "I agree, however coworker Z is out of the office this week and coworker Y is swamped -- we will have to wait."

me: "Coworker Z hasn't been out of the office for 9 months, though."

phb: "Coworker Z and coworker W tried a couple of times and could not get it fixed."

me: "Did they exhaust the entire technical resources of our company?"

phb: "No one really had lots of time on their hands. If coworker Z tells me again they can not fix it, I will hand it over to someone else for sure."

I really wish conversations like this didn't happen.

I really wish that more people at my job cared more about what they did.

Why did coworker Z and coworker W give up?

Why didn't they ask anyone for advice?

Sure it doesn't take much time to run sudo command xyz, but how much time is lost from breaking your concentration, switching contexts into "firefighting" mode, running the command, and trying to pick back up wherever you left off? And then what if you're so distracted by this craziness that you have to write a blog post about? How much time gets lost then?

I really wish people would care more.

¹ Andy Lester pointed me² to a great rant of his about this very phrase "can't you just...".

² And a second article linked from Andy on complexity management (I think he's on to something here)

Snippet from a real conversation with names and features renamed to protect the guilty, on implementing a test feature for a piece of software I've been tasked to write:

me: "Here, look at feature A, which is a test of X!"

phb: "Oh, that's wrong, it's supposed to be B."

me: "... But that's not spelled out in the requirements."

phb: "Oh, it's there."

me: "No, I just read them again, B is not mentioned. X is mentioned, Y is mentioned, B is clearly not mentioned. B is implied indirectly, but it is not mentioned. I implemented A to test X, but nowhere do the requirements say that A should be B. In fact, A is not mentioned, either."

phb: "But it's in there"

me: "No, X is mentioned, and to test X most easily, A should happen, but A is not mentioned, and B is clearly not mentioned. Regardless, if you wanted B a particular way, perhaps you should've, I don't know. mentioned it?"

Had I only had a copy of the requirements handy, I would've happily requested said PHB to point me to the specific section.