kentcowgill.org/blog

I don't mean to make this the "Andy Lester" week, but the company I work for is in technical debt -- there's no better way to put it.

For a little introduction on the concept of technical debt, there's a nice wiki page. But what brings it home for me are the slides from Andy's infamous technical debt, which are available on his website.

So what am I griping about?

Version control.

Love it.

Hate it.

Gotta have it.

Unfortunately, it's a big undertaking.

What we currently have in place is a script that does:

  cp $current_file ARCHIVE/$filebase-`date`.$fileextension 
  rsync $current_file $production_cluster

It even has enough brains to accept multiple files as arguments, even if they're in separate directories, and create the backup in a given files' subdirectory. While this (kind of) works, it's a pain for "rolling back", which I just had to do due to technical problems at our other office out of state. Especially painful when there had been changes to files that haven't made it to production, but the developer (me, in this case) is blissfully unaware of said changes, so it's not clear which version to restore.

Ugh!

Similar frustration for trying to re-restore the changes for eventual re-promotion.

But moving to a real^™ version control system is a chore. Why? Because there are 5-6 years worth of these archive files that would be nice to have as version history. Altogether, there are about 99,300 files total. And the perl program I wrote to import them into our version control system³ takes an average of about 3 seconds per file⁴. My math tells me that's about 3.5 days.

³ Perforce seems to be what the company has elected to use, even though the few times I've tried it, I've hated it. Nowhere as intuitive for me to use as more reasonable (and by the way, free) version control systems such as subversion.

⁴ Pitifully slow. But since the Perforce server is a bit far topologically, speed issues are to be somewhat expected.

I ran across some guys' blog recently where he decided that for every 100 posts to his blog, he should figure out some of his blogging statistics. Or something like that.

One of which was to see what time of day he generally posted. He explained some procedure for doing a database query from the shell, piping the output through some command line utilities, and then running some commands or somesuch to feed into some graphing program (gnuplot) to produce a chart.

Seemed like a terribly manual process.

Inspired by that, I created the following:

Which is pretty much what I stated, but it updates automatically. As in every time it gets displayed. What you see above is current, even now. However long ago this was actually written.

For the morbidly curious, the SQL to create the data looks like this:

    select count(*) as posts,
      substring(blog_date,12,2) as hour
      from blog
      group by hour;

I keep getting reminded of the story about the broken window I read recently in a great book I've purchased called The Pragmatic Programmer.

Most recently was this set of conversations between a coworker and myself, and my manager and myself.

Coworker Y: "Kent, can you please run sudo command xyz on the production cluster?"

me: "Sure, but only if you promise to fix the race condition that causes condition X on the production cluster."

Coworker Y: "I would, but it's not on my platter."

me: "Whose platter is it on?"

Coworker Y: "Coworker Z."

me: "This has been a known issue for nearly a half a year, hasn't it?"

Coworker Y: "Actually, about 9 months."

... time passes ...

phb: "Kent, does sudo really need to be run for command xyz? Can't you just¹ do it with normal user privileges?"

me: "I assume so - they wouldn't ask me to do it otherwise, no? BTW, the cumulative time taken to run or have someone else run [sudo] command xyz every time this happens is probably much greater than just fixing the race condition that causes condition X, which as I understand has been a known issue for at least 9 months."

phb: "They tried to fix the race condition and failed. But I didn't know coworker Z had been running sudo command xyz all along, I thought he had another way to deal with it. Until he comes back from vacation, can you take care of these requests?"

me: "I'd much rather the race condition get fixed."

phb: "I agree, however coworker Z is out of the office this week and coworker Y is swamped -- we will have to wait."

me: "Coworker Z hasn't been out of the office for 9 months, though."

phb: "Coworker Z and coworker W tried a couple of times and could not get it fixed."

me: "Did they exhaust the entire technical resources of our company?"

phb: "No one really had lots of time on their hands. If coworker Z tells me again they can not fix it, I will hand it over to someone else for sure."

I really wish conversations like this didn't happen.

I really wish that more people at my job cared more about what they did.

Why did coworker Z and coworker W give up?

Why didn't they ask anyone for advice?

Sure it doesn't take much time to run sudo command xyz, but how much time is lost from breaking your concentration, switching contexts into "firefighting" mode, running the command, and trying to pick back up wherever you left off? And then what if you're so distracted by this craziness that you have to write a blog post about? How much time gets lost then?

I really wish people would care more.

¹ Andy Lester pointed me² to a great rant of his about this very phrase "can't you just...".

² And a second article linked from Andy on complexity management (I think he's on to something here)

Please forgive me - I'm working on updating this. "This", of course, can mean different things. But what I mean is that I'm working on finishing my Grand Unification Project. What's that? Well, I'm finally bringing everything together. And finally making my blog public. It's done in Catalyst, everything else is plain CGI.

All this means is that you might find some things that don't work from time to time. If you encounter this, please excuse me. I'm only human. I can't know about every single bug. Yet.

So far, most of my blog has been updated. I even have a new version of my photos up and running. Even my resume has been somewhat updated... :)

So I'm in love with catalyst all over again.

I got tired of trying to figure out all these fancy-schmancy Object Relational Mapping database classes and trying to massage them to be able to understand my simplified database - which is a single table which is joined to itself for attaching a comment to a particular entry. I really didn't see any need for a separate table, since the structure of a comment doesn't really differ from the structure of a root post. Other than the fact that a root post doesn't have a parent.

Thinking that restructuring the database was just too much work, I just ripped out the ORM and created a model class that simply ran queries and returned the results.

Also, I added support for running my blog as a CGI and found that it was a really simple conversion, and only needed to update a few methods that I wasn't properly using before.

Snippet from a real conversation with names and features renamed to protect the guilty, on implementing a test feature for a piece of software I've been tasked to write:

me: "Here, look at feature A, which is a test of X!"

phb: "Oh, that's wrong, it's supposed to be B."

me: "... But that's not spelled out in the requirements."

phb: "Oh, it's there."

me: "No, I just read them again, B is not mentioned. X is mentioned, Y is mentioned, B is clearly not mentioned. B is implied indirectly, but it is not mentioned. I implemented A to test X, but nowhere do the requirements say that A should be B. In fact, A is not mentioned, either."

phb: "But it's in there"

me: "No, X is mentioned, and to test X most easily, A should happen, but A is not mentioned, and B is clearly not mentioned. Regardless, if you wanted B a particular way, perhaps you should've, I don't know. mentioned it?"

Had I only had a copy of the requirements handy, I would've happily requested said PHB to point me to the specific section.

I hope you're not looking at this using Internet Explorer. If you are, I'm surprised you're able to read this. "This", of course, referring to my new home page that I've been working on lately.

I've run afoul of some of the more heinous differences between Internet Explorer and the rest of the more standard-compliant web browsers. Specifically, as it relates to Tableless web design using solely CSS for element positioning. It looks so easy at CSS Zen Garden!

Please pardon my dust as I learn to work around these issues. Assuming you can even read this. And I just checked. It's quite likely you can't :(.

Sweet.

It only took re-writing the thing from the ground up, switching out completely the modules I was using - but I finally managed to get that silly picture stripper/thumbnailer working again.

Were it not for Imager by Tony Cook or Email::MIME::Attachment::Stripper by Casey West (with my friend Yaakov recommending the former and perlmonks.org for the alternative to MIME::Parser) - I'd still be spinning my wheels.

During the debugging, I'd get MIME::Parser to appear to work, but then Image::Magick would stop working. Or MIME::Parser would fail me in strange and non-reproducibly ways but Image::Magick would faithfully render an incomplete image file.

But no more.

I've resolved to figure out why my photo uploader keeps cutting off my pictures I email from my phone.

It's driving me crazy - it used to work like clockwork. Snap a little photo with the phone, punch a few buttons, click click bang, and whaddaya know, my photoblog gets automatically updated. With a thumbnail and a picture that aren't broken.

I'm working on it as I write this, waiting for slow email to arrive at my server.

We'll see...

So, I'm a moron.

Seems I'm not making proper use of the Model:: modules. I had originally found an old out of date tutorial making use of the Class::DBI helper scripts, so that's what I used in making this blog. Problem is, the tutorial was woefully incomplete, AND CDBI seems to work - at the surface - without any additional configuration.

Except when you have a stupid datamodel that requires you join a table to itself and get a count of child rows for each parent row.

These days, using Class::DBI seems frowned upon by the catalyst community, so I think instead of trying to fix what's broken, I'll just use a different model to access my data - HOPEfully one that isn't woefully broken, lets me write some of my own nasty SQL and actually reference the correct results to stuff into TemplateToolkit.

Too bad I'm busy working on my wishlist, otherwise I'd tackle this issue straight away.