Elf Sternberg

A co-worker of mine is very fond of Test Driven Development, which isn’t really my thing, although I’m learning. I have a lot of respect for his code, which works. When I mentioned that, when working alone, I really enjoyed Literate Programming the look on his face couldn’t have been more clear. He said he hated Literate Programming, that there was no point to writing that much commentary.

So I’ve been doing TDD with one project, and LP with another project, and I’ve come to a rather amusing conclusion: Test Driven Development and Literate Programming are the same discipline.

There are a million analogies to programming: debugging a blank page.  Architecture.  Sculpture.  If these are strong analogies for programming, then how about this one: TDD and LP are the injection molding of programming.  Both attempt to describe the outside of the program, the shape of it, the outlines inside of which it should live, and the border over which it shall not cross.  Once you’ve written that outline, you then write a program that conforms to it.  LP does it piece by piece.  TDD does it whole-hog, checks for leaks, and goes and revises either the mold or the program and tries again.

But both are essentially the discipline of describing the program in excruciating detail, then writing the program to that description.   TDD is automated, true, and can tell you when you’ve lied to yourself; LP, in contrast, assumes that since your duty is to keep text and code in sync not only don’t you lie to yourself, you’re unwilling to lie to yourself.

There are many disciplines to programming, with “throw it at the wall and see what sticks” continuing to be the most favored among many developers.  (Believe me, I’ve worked at places where that was the official development policy).   But when you talk about constraining the specification such that you know for sure “the program is supposed to do this“, then LP and TDD are, at base, doing the same thing: describing the specification in such detail that the program ends up molded to the spec.

Ah, the bleeding edge.  It’s a war out there!

This morning, Facebook released fbwatir. I’ve just spent the past few hours knocking it around, and have come to the conclusion that it’s pretty mega-borked but it can be saved.

In fact, I now have it working with Cucumber and Firewatir. There are several major flaws in FBWatir, the biggest of which is that it assumes its own responsibility for the browser object. This is broken, and causes Cucumber to spawn an innumerable number of Firefox windows. Commenting out the browser invocation in FBWatir and putting into cucumber’s own env.rb file is much better.

I also discovered that there’s a bug in Firewatir 1.6.2 that assumes that “window zero” is always broken,  but Javascript indexes the windows starting with zero, so window zero is still a valid window ID.  Annoying as hell, but easily monkeypatched away.

I now have Cucumber working with FacebookConnect…

Given how little I know about Ruby, and given how my Ruby expert says I’m doing “inappropriate things” with the Ruby scope, that’s a freaking miracle.  We’re working together to make it work better.

One of the nifty things that Django provides is the
{% url backreference %} syntax, which allows
you to name the targets in your list of URL objects and then refer to
them by an explicit name. You can sometimes use the function name
instead, and Django has a way of turning the function name back into
the URL. It works fine as long as the signature of the backreference
and the signature of the function match.

It’s very nice for RESTful interfaces. But what about AJAXy webpages?
AJAX is all about D(X)HTML and the rendering and animation of user
interfaces in a broser, and there’s a lot of Javascript that comes
along with all that HTML and CSS. And embedded AJAX often comes with
its own set of URL calls. I mean, seriously, what if you want to write
something like this:

<example.js>=
$.getJSON(
"{% url results %}", {},
   function(resp) {
   $("#resultswrapper").html(resp.results);
});

Here, I want to replace the string “url results” with the url
that returns the results and shoves their visuals into the
“resultswrapper” HTML object, whatever it is.

You could do this in Django, making this a templatized object
and spewing it out every time. But often enough this URL
never changes during the lifetime of the program. This is
effectively javascript with a macro embedded in it that you want
substituted once, preferably at start-up. Well, I haven’t done the
start-up for you. But here’s a nifty little chunk of code that’ll do
URL reverse lookups in any static files in
your STATIC_DOC_ROOT directory (by the way, that
STATIC_DOC_ROOT setting is pretty useful for development, if you’re
serving static media out of your Django server, as devs frequently
do):

<addition to settings.py>=
import os
DIRNAME = os.path.normpath(os.path.dirname(__file__))
STATIC_DOC_ROOT = os.path.normpath(os.path.join(DIRNAME, 'static')) + '/'

And here’s the routine. Note that I’ve made it a Django command:
put it wherever you want and use it wisely:

<rendertemplates.py>=
from django.core.management.base import NoArgsCommand
from django.core.management.base import CommandError

from settings import STATIC_DOC_ROOT
from django.template import Template
from django.core.urlresolvers import reverse

import os
import os.path
import re

re_walker = re.compile(r'\.tmpl$')

class Command(NoArgsCommand):
    help = ("Run a series of templates through the Template handler to produce "
            "(semi) static files.  This is mostly useful for javascript "
            "handlers with Ajax calls in them that only need the urls "
            "defined when the application is first built or starts running.  It "
            "allows developers to use the {% url somethingorother %} syntax inside "
            "Ajax handlers without burdening the application at runtime.")

    def handle_noargs(self, **options):
        paths = [os.path.join(STATIC_DOC_ROOT, path[0], filename)
                 for path in os.walk(STATIC_DOC_ROOT)
                 for filename in path[2]
                 if re_walker.search(filename)]

        for fn in paths:
            fn_out = re_walker.sub('', fn)
            open(fn_out, 'w+').write(Template(open(fn, 'r').read()).render({}))

It’s not perfect (hey, I wrote it in about 20 minutes, mostly to
fix this problem in a quick and dirty fashion. Given that I didn’t
know how the innards of the Template class worked, and have never used
the new os.walk() function, this was pretty good.

As always, I’ve provided the source code for this.

I’m prepping for an interview this afternoon at what is primarily a Perl shop, and so I’m mainlining the O’Reilly Perl in a Nutshell book as a way of reminding myself about the little details of the language. I can write Perl just fine– I just made some major hacks to dailystrips for example, and I write various renaming and text processing scripts all the time in Perl, because it is the language for that kind of thing.

But it’s little corners like, exactly what does bless do, that I’m reminding myself of. I know it’s the OO instance creation operator, and I remember the instantiation procedure but what exactly does it do?

So I go read the source code for Perl, in C, because that’s where the magic is kept, and I discover to my not-so-surprise that bless is a complete and utter hack. It puts a flag in the dereferencer to point to a different function, one that seems added after-the-fact, that instead of handling a procedure one way, instead just handles it another way with the recently dereferenced scalar as the first argument. That’s all it does. OO is so “bolted onto the side” of Perl that it’s amazing how important it seems to have become to the language.

But there’s so much missing from Perl; the whole metaprogramming capabilities of modern languages like Python, Ruby, and Javascript is just gone– done instead with code generation and eval, good grief– and yet the capacity for obfuscation is so terribly great. In many ways, Perl feels more like Bash scripting with a much bigger library of niceties and bolt-ons, which may explain why I use it that way.

I’ve been helping a close friend with her website, which is written in rails. She admits that’s a mistake, now, because it shares database tables with a PHP application, and the communication between the two has always less than stellar. I’ve been looking into her current major problem, but while I’ve been at it, I’ve been looking for easy solutions to put her back on an all-PHP track. (This seems to be the right thing to do; PHP is the only language she knows well, and the PHP component is the only thing that works reliably.)

I looked at Symfony as an easy PHP-based route off the rails.  It’s not bad, but getting it to run on my box was a nightmare. Undocumented are the number of things with which PHP must be configured to get Symfony running. For the record, to get the sandbox application to run, you need not just the stock PHP, but PHP with PDO, SimpleXML, and SQLLite. None of those are stock on Gentoo (and apparently not on Ubuntu, either).  Once I had rebuilt PHP (3 times!), it was all working.

But now it’s up and I can play with it. Oh, and I got Drupal running, but I had to use my production box to make it so. Annoying, that.

I have C++ on my resume because I wrote it for several clients while I was working at Spry and CompuServe, but it’s neither MicroSoft C++ nor is it particularly up-to-date.  One of the things I did not recall was that Raucous, the RADIUS-to-Oracle interface I wrote many moons ago, was written in C++.  For some reason, I was convinced that it was written in C, but no, I found an original copy of it on one of my ancient work-from-home folders and, sure enough, it was written in C++.  Not just that, but good C++.

Raucous wasn’t a particularly complicated piece of software: it listened to a shared memory message queue, passed that message to Oracle, took the response from Oracle, and dropped the response into a another shared memory queue.  It was a primitive two-way consumer-producer channel between the RADIUS server and the Oracle DB, but it was absolutely rock solid.

There were five objects total; not a huge ecosystem.  Thread, Log, Radius, Oracle, and Service.   Service, Thread, and Log were all generic server objects designed to handle any kind of internet transaction, and are pretty much an early thread-based Reactor pattern implementation.  Radius was a kind-of Service that listned to the queue, and each Radius thread had a shared handle to the Oracle object (which was written in Pro*C, an SQL-embedded C) and each query was wrapped in thread management so as to not confuse Oracle.  It was definitely a “as long as Oracle doesn’t choke, everything will work fine” kind of program, and it ran beautifully.

The thing of it was, it’s written with Literate Programming.  There’s an incredibly verbose section where I go on and on about how the shared memory deque works, mostly because it was my first foray into Solaris shared memory and I wanted to make sure I understood it before doing any more.  It’s an interesting read that way.   Even better, every major interface got its own LP document, as if I were writing independent libraries and only the int main(int argc, char *argv[]) mattered after that.

I haven’t written much C++ since then.   It was informally verboten at the last job after we had one programmer whose C++ was so good, so beautiful, so perfect and pedantic that nobody else could read it.  But if this is what I’m capable of, I should definitely get my noweb install working properly and start writing in it again.

FireWatir is my preferred testing suite for developing web applications.   Watir is a domain-language that describes things you do with a web browser: navigate to this page, find this input field, type in this text, find this button and click on it.  Combinded with a unit testing framework like Ruby’s Unit::Test, and you can write scripts that perform the basic functions of any application website and report back on what works and what fails.

The basis of Test-Driven Design is simple, but getting it right seems a little weird.  You really do have to write the tests first.   So I’m going to start with the two absolutely simplest scripts you’ve ever seen.  I haven’t even run the application builder script (rails <project> or django-admin.py startproject <project>) and already I know two things that it has to do.  But before that, let me show you the boilerplate I use for my test-driven work.  Below is a script that I put into a directory in my application project called setup.rb:

END {$ff.close if $ff} # close ff at completion of the tests

require 'rubygems'
require 'firewatir'
require 'test/unit'
require 'test/unit/ui/console/testrunner'
require 'net/http'
require 'json'
require 'optparse'

$options = {:address => 'http://localhost:8000/'}
OptionParser.new do |opts|
  opts.banner = "Usage: example.rb [options]"

  opts.on("-a", "--address", "Base address to test") do |v|
    $options[:address] = v
  end
end.parse!

This script is meant to be called by all testing scripts in the system.  It does two things: defines all of the libraries that the tests will need, defines a default address where we expect to find the development server, and accepts an alternative address as needed.

FireWatir invokes and drives Firefox.  The END clause at the top of this script banishes the window in which the tests were run when FireWatir has run all of its scripts.

Now, here’s the most basic test you’ll ever run.  I keep this in a file called, surprise, basic_test.rb.  This test has an important side-effect, which I’ll discuss in a moment.

require 'setup'
class TC_Basic < Test::Unit::TestCase
  include FireWatir

  def test_01_server_presence
    $ff = FireWatir::Firefox.start($options[:address])
    sleep 2
    # Do this rather than an assert; the tests cannot run if the server
    # is not present.
    if ($ff.title =~ /Page Load Error/) then
      puts "Could not contact server"
      exit
    end
  end

  def test_02_homepage_presence
    $ff.goto($options[:address])
    assert($ff.title =~ /SecretProject/)
  end
end

This file imports the setup script, which in turn does all of the argument parsing and setting up of libraries and stuff.  So that’s all taken care of.   It also defines a single test case, and two tests within that test case.  The first test is not really a test at all; it does the invocation of Firefox, sending it the address of the server.  If it cannot contact the server, it does not “fail” as a test should, instead it halts the tests right there.

The second test is much more fundamental, and is an accurate test.  It sets one variable (the home page URL), and asserts one true thing about the operation (the page’s title is set correctly)

One thing to note about these tests: I haven’t actually written the code these tests will test yet. That’s important.  My job, now, is to make these tests pass.  Once I’ve done that, I will go on to other things: logging in and registration, and then user interaction.

Each test will define one thing the user can actually do: list the user’s documents, pick a document, edit some detail of the document, save the document, delete the document, etc. etc.  That’s what TDD is for.  TDD is part of your documentation: it tells future testers what you expected your program to do well.

I have added Restricting a function’s scope to a specific object to the Javascript Framework Phrasebook.

I’m very fond of gaffitter, a smart little console program that scans a list of files and/or directories and fumbles through them until it comes up with a subset of that list that will fit in a given space.  It’s perfect for taking large directories of stuff and segmenting them into archivable collections.

I recently ran out of disk space on my desktop and realized I had hundreds, nay thousands, of music directories that I needed to put somewhere else.  I suppose I could have bought more hard drive space, but more than that I wanted a lot of it just to be put away.   I ran gaffitter on the collection with a limit of 4.2 GB, the reliable size of a data DVD, and it said I had about 35 collections worth.  Great, I thought, but how to organize the output of gaffitter into subdirectories that I could then burn onto DVDs?

For that, I wrote mass_gaffiter.py.  It’s a very simple little script that uses gaffitter’s regular output as its input, and then turns around and spits out another script (a shell script this time) that, for each collection gaffitter has identified, creates a subdirectory and moves everything in that collection into the subdirectory.  When it’s done, your cluttered directory is organized into a collection of subdirs named “gaf_disk_01″, “gaf_disk_02″, etc., all ready for growisofs or whatever other DVD burning software you like.

I’m trying to get into the habit of sharing the little utilities in life that I can’t work without.  I think of them as little throwaways, but some of them I’ve kept for years, so I figure someone else might have good use of them. Here’s mass_gaffiter.py:

#!/usr/bin/env python

import sys
import re

re_m = re.compile(r'^\[(\d+)\] Sum')

f = open(sys.argv[1], "r")
accum = []
for l in f:
    g = re_m.match(l)
    if not g:
        accum.append(l[:-1])
        continue

    print 'mkdir gaf_disk_%03d' % int(g.group(1))
    print 'mv %s gaf_disk_%03d' % (
        ' '.join(['"' + i + '"' for i in accum if i]),
        int(g.group(1)))
    accum = []

Integrated Development Environments are all the rage these days, with Eclipse being the sort-of break-out winner.  I don’t use an IDE, exactly, I use that ancient rustbucket of a text editor, Emacs, and while Emacs isn’t exactly an IDE, this rule applies: don’t live with a broken IDE.

For the past six months, I’ve been working with Emacs 21 on my desktop at work.  I do development in a number of different languages that require syntax highlighting and proper indentation: C, Python, HTML, CSS, and Javascript.  The Javascript highlighting engine did not work correctly: the indentation interpereter was almost alway wrong, leaving me with poorly-indented code that was hard to read.  I ended up wasting, I’m sure, hours hand-indenting the code so that I could see what I was doing.

I finally got tired of this rigamorale and asked  if I could please update the Emacs install by hand.  We’re discouraged from putting into the base OS anything that’s not approved by Red Hat, but he said sure, if I needed it and could build it by hand, if they ever had to restore my OS I obviously could do it again.  So I quickly built Emacs from scratch and installed it in /usr/local, which I then immediately backed up to the virtualization drive to make sure it would be there if I needed it in the future.

It has made all the difference in the world.  Emacs 22’s javascript-mode just works, and it does indentation at four spaces per closure properly.  Not only was I able to actually see the syntactical error I was making in a javascript configuration pass in Ext-JS, but I could see how the entire dialog manager could be abstracted out further into a metahandler and reduced my code duplication by half. Now that’s programmer efficiency.

Just a simple reminder: if your IDE doesn’t work exactly the way you want it to, it is slowing you down.  It’s worse than having no IDE and no smart editor, because a broken IDE  distracts your from the task at hand, drains time and will you have allocated to the programming task at hand, and introduces errors into your attention. Managing your attention span is the number one skill for the Internet age, and a tool that comes between you and that successful management is a disaster.