kentcowgill.org/blog

Hooray!

Shortly after complaining about how long my main page was taking to load, I started thinking up ways to make it go faster. As far as I could see, one option would be to try to parse the perl with perl - but just borrow all the ideas and regexes from the perl.vim vim syntax file. Easy enough - just figure out what the metacharacters in vim map to in perl, etc etc and so forth.

About a third of the way through my translation, I realized it was going to be harder than I was expecting/hoping, because there are some conventions in vim's syntax files that makes that sort of thing somewhat easy.

Plus, I wasn't even sure how good the translations were, or whether or not they'd even do anything useful:

my( $I, $i )
  = ( qr/[A-Za-z_]/, qr/[0-9A-Za-z_]/ );

my $packageRef = qr/(?:$I$i*)?(::|')$i/;

my $varPlain
  = qr/\\?(?:[@%\$]|\$#)\$*(?:
        $I$i)?(?:(?:::|')$I$i*)*\b/x;

my $functionName
  = qr/\\?&\$*(?:$I$i*)?(?:
        (?:::|')$I$i*)*\b/x;

Yuck.

I toyed with the idea of doing something with PPI, but quickly dismissed that.

I then thought I might've remembered reading something about creating some kind of "vim server" with Text::VimColor - so a quick read of that yielded nothing. But I noticed the link in SEE ALSO to Apache::VimColor, which thank goodness mentions Cache::Cache (and sibling Cache::FileCache).

What was I thinking?

So the solution was exceedingly simple:

use Cache::FileCache;

# to create a key for the cache
use Digest::MD5;

sub vimformat {
  my( $self, $text ) = @_;
  my $key = Digest::MD5::md5_hex( $text );
  my $cache = new Cache::FileCache;
  my $return = $cache->get( $key );
  if( ! defined $return ){
    # do hugely time-expensive formatting, set
    # the value of $return to desired text
    # set the cache key at the end
    $cache->set( $key, $return );
  }
  return $return;

And just like that, the page is back to being under 2 tenths of a second to create. Even with (as of this post) more than 10 hunks of syntax colored code.

Yay!

So, I released another module to CPAN today.

It's called Perl::Critic::Nits, which at this point only has one policy module, one that discourages accessing what appears to be private member data of a class. Feel free to have a look. Though if you're reading this post significantly after the date I'm posting it, that version may no longer be the most recent release.

It pretty much looks for anything that accesses a hashref subscript or an arrayref subscript. And in case you can't guess - that can be a problem if you're not dealing with an object. Like if, for instance, you have a regular old hash reference.

Drat.

Elliot Shank reminded me that the code being examined isn't compiled, so it's a little tough to figure out whether or not something isa blessed object, or just a plain hashref. D'oh.

At least it's easy enough to disable a policy. :/

However, it will definitely catch attempts to access an object's member data directly, so I don't feel so bad about that. So long as the pseudo convention of assigning $_[0] to a variable called at least either $self, $package, or $class.

Please enjoy, and let me know if you run into any bugs.

So I'm really disappointed.

That syntax coloring I'm doing for the various bits of code?

Slow.

Like, really really slow.

I know I've posted about it before, but since then, I've got 8 separate syntax colored boxes on my primary page. This takes anywhere from 1.8 to 2.4 seconds to generate.

This is not an acceptable solution.

The alternative is that at this point, I will have to come up with at least 5 posts in a row that don't have any syntax colored blocks at all before the page generation times comes down to a more acceptable level. Until then, I apologize for the poor viewing speed.

So, I'm dutifully going through some 129 modules in our codebase, creating documentation and tests for everything I find.

I mean absolutely everything.

But I wonder if I'm going too far?

For instance, I happened upon a module called Boolean.pm. Seems like it ought to be fairly low-hanging fruit, so I open it up to find that the whole of the module looks like this:

package Boolean;
use strict;
use base q(Exporter);
@Boolean::EXPORT = qw(TRUE FALSE);

use constant TRUE  => 1;
use constant FALSE => 0;

1;
__END__

No joke.

Fine, the tests will be easy enough. I won't repeat them here for brevity, but suffice it to say that the only thing that threw me for a loop is that use_ok didn't seem to import the exported constants into the namespace of my test file - after testing the module, I just use Boolean; and then I can test the constants. Could I have just called import as the module inherited from Exporter? Probably, but I didn't.

The next module I come across is another top level namespace module named Database.pm. Whose primary purpose is to figure out which database server to have the rest of the code connect to, depending on a few external conditions. Part of the module declares a number of constants - the database connection strings - for internal (to the module) use only, and doesn't use base 'Exporter' and subsequently export those constants. However, it makes sense to me that I ought to test those constants as part of my test suite.

So, fresh with my knowledge that constants aren't imported into the test file's namespace via use_ok, I add a use Database; and expect my tests to pass. Oops, not so fast - because I'm testing the constants as barewords, and they're not exported via @EXPORT, they're not recognized as constants and my test file (with use strict at the top) fails miserably complaining about my usage of barewords.

So, what to do?

I decided I would force the Database module to be @ISA = 'Exporter' and add the constants to @EXPORT myself, on behalf of the module, in the test file. So now my test file looks a bit like this:

use strict;
use warnings;

use Test::More tests => 2;

BEGIN {
  use Exporter;
  @Database::ISA = ( 'Exporter' );
  @Database::EXPORT = qw/PROD DEV/;
}

use_ok( 'Database' );

use Database;

is( PROD, 'production:dsn:etc',
    'PROD points to correct db' );
is( DEV, 'development:dsn:etc',
    'DEV points to correct db' );

... but I have to wonder if this is worth all the bother. I suppose for now I'll leave it.

&qidYou may have guessed by now, but I've taken up a particularly onerous gauntlet at my job. I'm afraid to touch any modules. But holy cow, do they need touching.

What do we learn from Refactoring: Improving the Design of Existing Code ? Well, we learn that without tests, you can't refactor. Well, you can - it's really not wise to do so, however. It will be difficult to know if you've broken something or slightly altered a particular functionality.

Of course, tests also help you prove what you've got is a set of modules that performs to their specifications - or in the absense of a specification - proving that the code does what it looks like it's supposed to do based on certain circumstance.

Given that our codebase is huge, undocumented, and more important untested, I've come up with a few snippets of shell code to really move me pretty far forward with my task.

First, to create a directory hierarchy to house all my new tests:

for i in $(find . -name "*.pm")
  do touch $(echo $i | \
           sed -e 's/\./\/path\/to\/dir/' | \
           sed -e 's/\.pm$/.t/')
done

And to give the files some default content (it's ugly but it sorta works, YMMV):

for i in $(find . -size 0)
  do echo '#!/usr/bin/perl' >> $i
    echo "" >> $i
    echo "use strict;" >> $i
    echo "use warnings;" >> $i
    echo "" >> $i
    echo "Test::More 'no_plan';" >> $i
    echo "" >> $i
    echo "use_ok( '`sed -e 's/\//::/' | \
      sed -e 's/\.t//'`' );" >> $i
  done

These snippets, plus my previously mentioned vim tip are really getting me pretty fair along in creating a comprehensive test suite and a good set of documentation for all of our legacy code.

Oh, and as a side benefit, I'm getting to know the modules pretty well while I'm at it :)

I just read an interesting (but apparently now controversial) idea in Micheal Schwern's use.perl journal about trying to put face to names on CPAN.

I think it's a great idea.

BTW, as I'm testing this out, I'll be adding my gravatar to all of my posts. And if you, dear reader, feel obligated to leave a comment, and have a gravatar setup for your email address, give it a try. Please note that I do take some care to obfuscate email addresses from address farmers, so you should feel a little better about that.

Unfortunately, I haven't waved my hands appropriately or chanted the magic incantation, so my gravatar is just showing up as a default logo :(

You'll notice as soon as it shows up, I'm sure :)

Update: So I guess I figured it out after all :)

I wrote a few days ago about writing my own Test::Harness parser since I had been unable to install the module that does that, and does it well.

On my machine, Test::TAP::HTMLMatrix is beautifully installed - however where this module could do me the most good - as in having tests run all the time for me - I have historically had issues getting modules installed. At the very least, I need to identify all the various dependencies of every module and their dependencies in order for the SAs to search for .rpm files for ease in making the modules easily installable on every other production machine in the cluster. Which makes a certain amount of sense. As long as you don't remember that this test reporting module has no business on production anyhow.

But a brief perusal of the source for HTMLMatrix, and I'm starting to get the impression that having the SAs install it, plus all it's dependencies, plus all their dependencies, might be more hassle and take longer than I'd like.

Since I got my html harness pretty much working before trying to install HTML Matrix again, I might as well start using that at work to keep at-a-glace test results at a handy web page. Without all those dependencies.

So I did. And I copied all the tests over to our dev server. And I tweaked the hell out of them to make them actually run. More on that later.

The above picture is just a snippet of it to be able to fit into my blog, but you can see a fuller version here.

The code to create this is pretty simple. Here are the important bits of it:

#!/usr/bin/perl

use strict;
use warnings;

use Test::Harness::Straps;

my $strap = Test::Harness::Straps->new();

my $graph_width    = 600;
my $filename_width = 150;
my $percent_width  =  75;

print report_head();

my $cnt = 1;
my $tot = 0;
my $ok  = 0;

for my $file( @ARGV ){
  next unless -f $file;
  $cnt++;
  my $style = $cnt % 2 ? 'blue' : 'tan';
  my $result = $strap->analyze_file( $file );
  my( $this_tot, $this_ok )
    = ( $result->seen, $result->ok );
  $tot += $this_tot;
  $ok += $this_ok;
  my $graph
    = return_graph( $this_tot, $this_ok );
  my $percent
    = int( $this_ok / $this_tot * 100 );
  my $status = $percent < 100
      ? '<span style="color:red;">NOK</span>'
      : 'OK';
  my $perstyle = get_percent_style( $percent );
  printf <<END_REPORT, $style, $file,
    $file, $status, $graph, $perstyle, $percent;
  <tr class="row">
    <td class="%s">
      <a href="%s">%s</a>
    </td>
    <td class="status">%s</td>
    <td>
      <table class="graph">
        <tr>
%s
       </tr>
     </table>
   </td>
   <td class="%s ctr">%d\%</td>
 </tr>
END_REPORT
}

print report_end();

sub return_graph {
  my( $cnt, $run ) = @_;
  my $width = int( $graph_width / $cnt );
  my $leftover = $cnt - $run;
  my $bad_cell
    = qq{<td class="nok" style="width: }
    . qq{${width}px"> </td>};
  my $good_cell
    = qq{<td class="ok" style="width: }
    . qq{${width}px"> </td>};
  my $cells = $good_cell x $run;
  $cells .= $bad_cell x $leftover;
  return $cells;
}

sub get_percent_style {
  my $pct = shift;
  return
         $pct == 100    ? 'hundred'
       : $pct >= 95     ? 'ninetyfive'
       : $pct >= 90     ? 'ninety'
       : $pct >= 80     ? 'eighty'
       : $pct >= 70     ? 'seventy'
       : $pct >= 60     ? 'sixty'
       : $pct >= 50     ? 'fifty'
       : $pct >= 40     ? 'forty'
       : $pct >= 30     ? 'thirty'
       : $pct >= 20     ? 'twenty'
       :                  'ten';
}

The missing report_header() and report_end() routines add the head, title, stylesheet, body tags, etc. to turn the meat of the output into a web page.

Btw, if you have 1) unresponsive SAs or 2) too much bureaucracy to get modules installed, and 3) have an improper Scalar::Util installation, and 4) need to use Test::MockObject (or some other module that 'optionally' uses Scalar::Util::weaken - and some Enterprise Linux Distributions ship a broken version of Scalar::Util) you can use this sneaky workaround:

BEGIN {
  # We're not using feature XYZ in
  # Obj::Using::Weaken, so don't
  # care about our improperly
  # installed Scalar::Util
  require Scalar::Util;  # load it into memory
  # we know this causes warnings
  no warnings 'redefine';
  # load our own versions that do nothing.
  *Scalar::Util::weaken = sub { return @_ };
  *Scalar::Util::export_fail = sub { return };
}

Wikipedia vandals are awesome. I took a peek at the page for the iPod Touch (recently released by Apple Computer) and just happened to see that a vandal had added a new image in place of the fancy product shot that used to be there.

A few minutes later, it was gone.

However, I couldn't resist making a copy of the picture, posted here for your amusement. Be sure to check out a screen shot of the vandalized wikipedia page.

For more on wikipedia vandalism, check out their page on wikipedia vandalism, which hasn't (at the time of writing) been updated yet to include the iPod Touch vandal.

There are some more funny screenshots of wikipedia vandalism.

Perhaps my favorite is Stephen Colbert's repeated urges to continue vandalism.

I'm going through a lot of legacy code lately, and got tired of manually opening up the first legacy file I documented (to make sure I was following the same standard I had implemented - it's not quite in my long term memory just yet), scrolling through the tedious documentation, copying (or retyping(!)) the text into the file I was working on, etc.

So, I decided to write a quick little mapping for that menial task.

First step is to create a template for the POD I want to insert:

__END__

=NAME

=SYNOPSIS

=DESCRIPTION

...

And then this helpful mapping in my ~/.vimrc

" Automatically add a POD template
" to the end of a file
noremap ,ap maG:r ~/podtemplate<cr>'a

Mnemonic: Add Pod.

... which is at least smart enough to set a mark at your current position, move to the end of the file, insert the contents of ~/podtemplate, and return to your saved position. Caveats: It's NOT smart enough to know if you've already set an 'a' mark, and it's NOT smart enough to NOT insert the template if there's already POD in the file.

Ok, to be fair - it's really not a problem I have with CPAN.

A month or so ago, I won a copy of Perl Testing: A Developer's Notebook as a 'door prize' at a Chicago Perl Mongers meeting. I devoured it within a few days, reading it during my daily commute.

Of special interest was the section on creating a test harness (or custom status report) for testing output by making use of Test::Harness::Straps . This was very cool because I had tried to get Test::TAP::HTMLMatrix to install on my Mac earlier this year only to be met with failing tests, or failing tests of dependencies. I don't recall, but I was really bummed. Figuring out the issue was beyond me at the time - I looked into it, but didn't have a ton of time, and I couldn't quickly see the issue.

Anyhow, today I had a bit a inspiration to make use of the example code from the book and see if I couldn't use it to produce some fancy HTML output of my own to give an at-a-glance status of tests I had run, similar in nature to the cool Pugs Smoke Reports . So, I whipped open my copy of Perl Testing , typed in the example code, ran it against some tests I had recently written at work, fixed the bits that were broken (by consulting the documentation for Test::Harness::Results )... run, view, tweak, repeat ad nauseum.

I finally got something fairly close to approximating those cool Pugs reports. But there were some big differences - which I couldn't figure out how to implement quickly (after all - the rest of the development of this parser was really quick and easy) - and I was running short on supplemental time. I actually thought it might be beneficial to take another look at HTMLMatrix and see if I could tweak that a bit.

I took a look in my .cpan cache directory, but it wasn't still there. So I fire up my CPAN shell and try to install it (knowing full well it will fail, but my fingers are trained to type install MODULE).

And before I know it, it's successfully installed.