kentcowgill.org/blog

So I'm working on a new presentation for my local Perl Mongers group, and (indirectly) thanks to Ricardo Signes, I've got a cool way to get properly syntax colored code into my slides.

Ricardo has been working on an easy way to get syntax colored code into Keynote presentations. I wondered why he was bothering to convert the syntax colored code to RTF - then I realized why - I think because TextEdit.app on Mac OSX is a cocoa application, the font coloring is preserved through copying and pasting the code into Keynote, another cocoa application.

A little later on, I was creating some other presentation, and was copying some code out of my blog and happened to notice that the code I copied and pasted from Safari, my web browser, also retained its coloring information.

Problem solved, right? Anything I want to put into a Keynote slide, I should blog about first. Right?

Wrong.

That's too much blogging.

Instead, I wrote a teeny tiny little CGI to post up the syntax colored source of anything sitting around on my server, using the same Text::VimColor module on the backend.

This is that CGI:

#!/usr/bin/perl -T

use strict;
use warnings;
use Text::VimColor;
use CGI qw/:standard/;
use CGI::Carp qw/fatalsToBrowser/;

$ENV{'PATH'} = '/bin';

my $file = param('f') || 'photos/photoblog';
my $lang = param('l') || 'perl';

die "Bad file" if $file =~ /^[^a-z]/i;
die "Bad file" if $file =~ /[^a-z\/._-]/i;
die "Bad file" unless -f $file;

die "Bad type" unless $lang =~ /(?:perl|php|xml)/;

open my $in, '<', $file;
my $text = do { local $/; <$in> };

my $vim = Text::VimColor->new(
  string      => $text,
  filetype    => $lang,
  vim_options
    => [qw(-RXZ -i NONE -u NONE -N -T xterm)],
  vim_command => '/usr/local/bin/vim',
)->html;

... and then a little HTML to display it in the right font, font size, and using my standard code stylesheet - which is left as an exercise for the reader.

Wow.

Never stop looking for ways to improve existing code.

I had noticed that after I added tagging capability to my photo gallery that:

1. Page loads seemed sluggish.
2. Tagging 300 or so photos produced a large list of tags.
3. Drawing the tags seemed the slowest, as I could watch them appear in the tag cloud.

I actually started going through about 150-200 of my photos and re-tagged them to remove silly and/or superfluous tags.

But then on Sunday I was browsing my del.icio.us network and saw an article that caught my eye about optimizing MySQL. Since MySQL is the database that powers much of what you see here, I figured I'd take a look to see if I could help shave off any execution time for my database access, in case it mentioned anything that I didn't already know.

One of the items near the end mentioned a tool called mytop. I downloaded and installed it and quickly learned two things.

1. My database is pretty low traffic.
2. Unless I'm hitting my photo gallery.

I used the qps(Queries per Second) mode and saw that when I hit the first page of the main gallery (which only shows 10 thumbnails with their captions in the href title by default), I had an astouding 500+ queries run in a single second! Meaning that every time through a page load, I was querying my database for captions for all (currently) 500+ images.

That's insane.

I already had some code in place that conditionally added the pictures I've looped through to my Template Toolkit $vars variable, but I was performing the caption lookup outside that condition - in essence running it for each and every picture, even though I'm only displaying 10.

I moved the caption query inside the condition, and noticed an immediate improvement - page generation time went from a half a second to a tenth of a second. Really, a fairly simple fix for an 80% speed increase.

Going through some old code reviews, I found the following in some of my notes.

The idea was that there's a hunk of code that needs to produce a sorted and unique list of items. That description immediately brings to mind a few data types - an array and a hash - and a sort of some kind or another.

The code I encountered looked like this:

  my @pre_sorted = $s->arrayOfArrays;
  my $unsorted = \@pre_sorted;
  my @key_list = ();
  my @sorted = ();
  my %hash;
  # make the AoA into a hash, keyed on [1], the name
  foreach(@$unsorted){
    $hash{$_->[1]} = $_->[0];
  }
  # copy the keys into a sorted array
  @key_list = sort( keys(%hash));
  # build the sorted AoA
  foreach(@key_list){
    my @temp_array = ( $hash{$_} , $_ );
    push @sorted, \@temp_array;
  }

Don't get me wrong - that does do what it's supposed to do. But does it do it efficiently? Not really - that can all be replaced by the following:

  my %unique_items = ();
  my @sorted = sort { $a->[1] cmp $b->[1] }
               grep { ! $unique_items{$_->[1]}++ }
               $s->arrayOfArrays;

Which has the following benefits:

1. Much more succint.
2. Many fewer temporary variables.
3. More perlish.
4. Quite a bit faster.

How much faster? It depends a little on the size of the data passed to it. I found that most of the time, the data coming to it was very very small, but occasionally would have much larger data sets to sort. I recall having some spare time and more curiousity then is likely good for me, so I set up some benchmarks:

First, results from running the routines
500000 times on a tiny data set:
                Rate    Original New version
Original     98232/s          --        -71%
New version 342466/s        249%          --

Next, results from running the routines
200000 times on a small data set:
                Rate    Original New version
Original     61920/s          --        -69%
New version 200000/s        223%          --

Now running them 10000 times on a
larger data set:
              Rate    Original New version
Original    4032/s          --        -49%
New version 7937/s         97%          --