Elf Sternberg

I suppose I should make clear that what makes me most annoyed about Winterson’s The Stone Gods (see previous post) is that the Used Furniture problem is more pernicious than just a good writer “slumming” around inside the SF universe, borrowing from the warehouse, or failing to think clearly either the justifications for her setting or the consequences of it.

It’s that, for the past twenty years– longer, if we add in all the forward-looking material from Asimov through the early Cyberpunks (and even the proto-Biopunks like Rucker and Hansen)– there has been a serious, ongoing conversation within the SF writing and reading community about the consequences of our current setting.  We look at Real Dolls and bad phone voicejail trees, at insurance companies giving breaks to people who install GPS trackers on their cars, at the breaking point of Moore’s law and the attempts to keep going, and extrapolate out from there.  This conversation extends from Charlie Stross to Adam Warren (author of Hypervelocity, the most underappreciated posthuman novel of 2007, and inventor of the term “tachycognitive”), from Greg Egan to Masamune Shirow, and Elezier Yudkowski to, er, me.   We’ve been at this for most of our adult lives, thinking about what it means to share our cognosphere with thinking creatures of another substrate, and the moral and human consequences of creating those creatures out of whole silicon and steel.

Winterson apparently is unaware of that conversation.  She treats SF as if it were a Western In Space, a place where she can dodge the “real world” and write whatever she wants, without justification, explanation or extrapolation.  She didn’t do her research.  She didn’t reach out to anyone who reads SF and say, “What’s the state of the art in thinking about these topics?” or “Here’s where I’ve gone with my manuscript, what do you think?”  None of her pre-readers, if she had any, were SF fans.  Nobody pointed her at Accelerando, or Diaspora, or Ghost In The Shell, or even The Journal Entries.  Those are the state-of-the-art for the conversation about human/machine interaction.  Winterson was unaware, and chose to remain so.   I guess I expected better.

Like many designers, I’ve been enjoying Clients from Hell, a collection of various incidents freelance designers and developers have had working with customers over the years.  Oh, boy, do I know some of these.  But there’s one that stuck in my craw: “Slow It Down.”  The basic complaint is common to Ajax-based applications: if you’re close to the server when using Ajax for state management (send commands back to the server to change or save some detail), the transition from “working” to “done” can be so short that some users won’t realize that the action they committed has been saved.

I saw this on a project for an intranet deployment once.  The user was presented with a list of alert states, and if the alerts were being dealt with he could click on a checkbox to “silent” the alert and it would stop harassing his pager.  Since it was an intranet, the LAN was very fast and the server lightly loaded, so the “working” spinner would blink by so fast some people complained that they didn’t know if the action had been saved.  Where, they asked, was the submit button?

In “Slow it Down,” the developer says that he added a 1.5 second delay “on the server.”  He gives this as an example of a dumb client, but in this case, the client is right and the developer is wrong, and the developer is solving the problem the wrong way.  Using sleep() on the server blocks the thread, may slow down other transactions, and most importantly blocks the client from performing other actions.   The correct solution is to do it on the client, where you don’t use up server resources and the user can have many such transactions going on at the same time.

Here’s the whole solution, using jQuery. I create a function that clears the spinner out. Then I add an event handler that handles my data transaction and registers an on-end event handler. That event handler checks to see if the transaction took longer than the “display spinner” timeout. If it did, it clears the spinner.  If it did not, it creates a new setTimeout event to clear the spinner after the specified amount of time has passed.

(function() {
    var timeout = 500;

    function swapSpinner() {
        $('#spinner_container').html('');
    }

    $('#submit_button').click(function() {
            $('#spinner_container').html(
                '<img src="/assets/images/spinner.gif" alt="" />');
            var form = $('remote_form');
            var data = $(form).serialize();
            var action = $(form).attr('action');
            var started = new Date();

            $.post(action, data,
                   function(data, status) {
                       var when = new Date();
                       var inter = when.getTime() - started.getTime();
                       console.log(status, inter);
                       if (inter < timeout) {
                           setTimeout(swapSpinner, timeout - inter);
                       } else {
                           swapSpinner();
                       }
                   },
                   'text');
        });
        return false;
})();

This is the correct way to do this: a server thread is freed the moment the transaction is done, but the cognitive message the server is doing something has been successfully delivered to the user, and the (admittedly minimal, but as Google can tell you gazillions of minimal costs add up) CPU load has been distributed to the client. Because javascript is single-threaded and event modeled, many such events can be going on at once, so in the example I described above, the customer could click several boxes in a row and the system would do the right thing.

The developer might say that this is a case of educating the user.  I disagree: this is a case of unpleasant surprise.  There’s a rule in web design: exceed your client’s expectations gently.  If you’re going to make transactions instantaneous, you’re taking away the cognitive step submit the transaction.  You must replace that with some alternative acknowledgment.  A delay does that.  A delay done right does that and creates new opportunities for further exploration of the technique.

Heathen Philosopher: One who has left his fly open.  (1811 Dictionary of the Vulgar Tongue).

For multi-select boxes where you have a list of things you want to filter on, and you want the queryset returned to be an OR’d set.

    qset = Q(mode__exact=modes[0])
    if len(modes) > 1:
        for mode in modes[1:]:
           qset = qset | Q(mode__exact=mode)
    r = Responses.objects.filter(qset)

Now, I need to figure out how to do this as a list comprehension!

For extra coolness, you can see what SQL you’re generating:

    print r.query.as_sql()

I can’t tell if I’m suffering from the geek equivalent of writer’s block, analysis paralysis, overload, or some combination of all of the above.

I’ve been working on a generalized platform for narrator, the story engine for the Journal Entries and everything else. As I’m working my way through the process, I’m thinking in typical RDBMS fashion, “Okay, the system has Authors and an Author has Serials and a Serial may have more Serials which eventually leads to having Stories.” (To make the whole “Serials” thing clearer: A universe has novels; novels have chapters. “Universe” and “Novel” are synonyms for Serial, and “Chapter” is a synonym, in this system, for “story”– an atomic block of text readable in one sitting. Theoretically.) I have a tree-based heirarchy for Serials working, although I’m damnably unhappy with where in the system the knowledge for “An author may have zero or more serials” is stored.

The actual architecture is sorta backwards: Each story has a foreign key to a serial, and a serial may have a foreign key to a parent, and ultimately all serials have a foreign key to a user, and so forth; the usual one-to-many relationships built within SQL. The nice thing about Django is that it notices these things and builds the backing querysets automagically, so you can traverse either way along this structure without having to write your own code.

And I pretty much have all that put together. I have some performance worries, and have seriously contemplated what this would work like under Mongo, but so far, it seems to be fine.

Time to upload my stories to the system, right? Four “Universes,” one merely labeled “Other Stories.” And I’m thinking, well, I could do it by hand– yeah, all 400 stories or so– or I could fake it just spew the data from the old database, through a transform, to the new one– fast, but misses the point– or, the choice I made, I could write a simple web-based API.

“Simple.”

It turns out that there are three mysteries to web-based APIs. First, there is RESTful verb architecture, meaning: do you encode the verb in the URL, or do you go with the HTTP verbs of PUT and DELETE along with the more traditional GET and POST?

RESTful verb architecture rigidifies the interaction with your system in strange ways. Consider my description above as a web page; a web designer would build a limited view of the author’s profile data, plus a tree-based view of the author’s serials and the stories they contain. Now, if your URL for an author is, say, “/author/<author_id>/”, it might make sense in an API to provide a complete view of the author profile and a complete list of his works, or it might make sense to have two two calls– one for profile, one for his works– but there’s no real way to structure the system in such a way that the URL refers some of the author’s information, and a tree of serials and stories. Those are four different relationships, referring to three different objects, one of which recurses.

This interacts with the other complete problem: how do you return data to the client? As… what? JSON? XML? YAML? A blob of text? Remember that one of the nifty miracles of narrator was its ubiquity: you could download the stories highly styled, or simplified for text-to-speech (I seem to have a lot of blind readers), or as text, or as a PDB Docbook format appropriate for ebook readers. Once the system has decided what action is going to be taken on the resource, that resources has to be rendered and returned to the client.

All of this, when dealing with a framework like Django or Rails, also seems to create the headache that they expect the REST commands to correspond to row actions on a table, which is not what’s going on here. An author isn’t just a row entry in the User table; it’s also a resource with other resources. Serializing it isn’t a simple matter of taking the row, the column names, and adding them together into a dictionary.

All of this comes with one surprisingly enormous pressure, from the Django community: DRY. Don’t Repeat Yourself. If you do, yur doin’ it wrong, as you Intarwebfolk like to say. (I am aware of all Internet traditions.) So when I start to write a generic method for handling the XML, YAML, and JSON versions, and have a “special” version for handling HTML, I worry that I’m repeating myself and the anxiety is crippling.

I should just say “Fuck it,” neh? Just write a stupid intake for “add a serial to a serial,” and another for “add a story to a serial,” and live with it. It might even become canon, and if I ever publish this thing, I’ll have people cursing me for using POX. Do everything in JSON (it’s what I know best) and worry about extensions later.

Oh, yeah, there was a third complication: Authentication.  That mystery is less difficult, but still annoying.

I got an email today from one of the super secret stealth start-ups, the one I really wanted, and they told me that they had chosen to go with another candidate. Bummer.

At one recent interview, an engineer asked me to “give him the sum of the two highest digits in an integer array. Any language.” I thought for a moment, then wrote:

array.sort()
return array[-1] + array[-2]

He said that was fine, but since it would be O(n*log_n), could I do it in O(n). He wanted a linear search instead.

After answering the question to his satisfaction, we got into a bit of a religious discussion about the merits of concentrating on performance during initial composition, versus concentrating on correctness. I’m much more a “get it working, then get it right, then profile the working program for bottlenecks and work them out.” It is possible that you can create systemically slow solutions that’ll require a re-write, but that’s actually far less likely than having I/O or CPU bottlenecks in small functions of code that can be fixed with optimization.

He later told me that, although I didn’t get the job, he voted to hire me. He liked my answers and appreciated that I was willing to stand my ground and discuss the relative merits of different approaches.

There’s one piece of advice SEO mavens always give out that always sets my hair on fire.  It’s this: “Always design your web pages with CSS.  No Tables!”

This advice is pure, unmitigated bullshit.

I just designed a website for a client.  I used CSS for a lot of things.  Surprisingly enough, I used tables for– gasp– tabular data.  A table of classroom assignments.  A table of type/key/value tuples for features of the software the students are writing for each assignment.   Tabular data.  That’s what tables are for.

SEO experts who terrify newcomers into thinking they’re doing something wrong when they use a table– a perfectly useful tool intended for the liquid display of tabular information, the outer perimeter of which can be affixed to whatever grid makes you happy– do nobody any favors.  Be specific.  Say “Don’t use tables for layout.  Learn about semantic design.  Use tables for what they’re good for.”

Well, that took a little bit longer than I anticipated. But, I’ve learned an important lesson and successfully brought a project in on time and budget, so I can’t complain.

When the customer says, “We tried this with the last contract but he was more of a designer than a developer. He almost got it, but couldn’t quite make it work,” don’t assume he still has that source code. Ask. That was my big mistake. It took me awhile to dig myself out of that hole, and yet when I was done I had a much better understanding of how Wordpress works than I would have otherwise.

Still, the site is looking fine now. The specific request was to make the “Categories” menu show “a list of pages associated with the current category,” which involved two magic tricks: one, in Wordpress, categories belong to posts, not pages. The two are separate database items. And two, once “pages have categories,” figuring out from the current page what category the visitor is in, and highlighting that category in the category list. This is a different task from doing the same thing through the current post.

I’ve learned about customized Category_Walkers (apparently, this is a Dark Art, as I’ve been able to find exactly one example other than my own that used it) and how to integrate category lookups with the Wordpress metadata engine.

FireWatir is my preferred testing suite for developing web applications.   Watir is a domain-language that describes things you do with a web browser: navigate to this page, find this input field, type in this text, find this button and click on it.  Combinded with a unit testing framework like Ruby’s Unit::Test, and you can write scripts that perform the basic functions of any application website and report back on what works and what fails.

The basis of Test-Driven Design is simple, but getting it right seems a little weird.  You really do have to write the tests first.   So I’m going to start with the two absolutely simplest scripts you’ve ever seen.  I haven’t even run the application builder script (rails <project> or django-admin.py startproject <project>) and already I know two things that it has to do.  But before that, let me show you the boilerplate I use for my test-driven work.  Below is a script that I put into a directory in my application project called setup.rb:

END {$ff.close if $ff} # close ff at completion of the tests

require 'rubygems'
require 'firewatir'
require 'test/unit'
require 'test/unit/ui/console/testrunner'
require 'net/http'
require 'json'
require 'optparse'

$options = {:address => 'http://localhost:8000/'}
OptionParser.new do |opts|
  opts.banner = "Usage: example.rb [options]"

  opts.on("-a", "--address", "Base address to test") do |v|
    $options[:address] = v
  end
end.parse!

This script is meant to be called by all testing scripts in the system.  It does two things: defines all of the libraries that the tests will need, defines a default address where we expect to find the development server, and accepts an alternative address as needed.

FireWatir invokes and drives Firefox.  The END clause at the top of this script banishes the window in which the tests were run when FireWatir has run all of its scripts.

Now, here’s the most basic test you’ll ever run.  I keep this in a file called, surprise, basic_test.rb.  This test has an important side-effect, which I’ll discuss in a moment.

require 'setup'
class TC_Basic < Test::Unit::TestCase
  include FireWatir

  def test_01_server_presence
    $ff = FireWatir::Firefox.start($options[:address])
    sleep 2
    # Do this rather than an assert; the tests cannot run if the server
    # is not present.
    if ($ff.title =~ /Page Load Error/) then
      puts "Could not contact server"
      exit
    end
  end

  def test_02_homepage_presence
    $ff.goto($options[:address])
    assert($ff.title =~ /SecretProject/)
  end
end

This file imports the setup script, which in turn does all of the argument parsing and setting up of libraries and stuff.  So that’s all taken care of.   It also defines a single test case, and two tests within that test case.  The first test is not really a test at all; it does the invocation of Firefox, sending it the address of the server.  If it cannot contact the server, it does not “fail” as a test should, instead it halts the tests right there.

The second test is much more fundamental, and is an accurate test.  It sets one variable (the home page URL), and asserts one true thing about the operation (the page’s title is set correctly)

One thing to note about these tests: I haven’t actually written the code these tests will test yet. That’s important.  My job, now, is to make these tests pass.  Once I’ve done that, I will go on to other things: logging in and registration, and then user interaction.

Each test will define one thing the user can actually do: list the user’s documents, pick a document, edit some detail of the document, save the document, delete the document, etc. etc.  That’s what TDD is for.  TDD is part of your documentation: it tells future testers what you expected your program to do well.

Storyboarding Princess Jera

I’ve recently begun experimenting with storyboarding my longer works the way Elmore Leonard does. He uses 3×5 cards, but I figure that with my crabbed handwriting and tragically bad organization, 1×1.5 post-it notes is going to work just fine.

The story to my left is the rubble of Princess Jera, a novel told in twenty alternating chapters of about 2500 words each, so about 50,000 words total, not novel length but good fun and practice.  The story is about half gonzo; Jera’s magic powers come from arousal and pubescence (classic themes beaten to zero Kelvin death by the X-Men and the Twilight series, but still pliable in the marketplace), the Traditionalists are terrified of a Mage upon the throne and the Mages are terrified that she’ll be like the last royal, discovered-at-puberty Mage, who damn near wrecked the kingdom with his self-indulgence and self-destructive tendencies.   That he wasn’t very bright compared to Jera doesn’t quite enter into their calculations.

With help from a friend in her father’s court, Jera finds documents from that era, four centuries prior, and embarks on a journey of self-discovery (ahem), discovers that there’s a word for her and the two mages assigned to her training have been mostly training her how to tamp down and ignore her powers.

Even worse, Jera discovers that an incident from four centuries ago that both her nation and the kingdom to the north portray as a great betrayal by her ancestors was something far, far more complicated… and she may be able to undo much of the damage.  There are those in the court who like things exactly the way they are.

The story ends with a violent confrontation in an ancient castle, a ruin where once her predecessor’s secret mage council practiced forbidden necromancy, and a discovery that will turn the entire region upside down with its implications.

The storyboard is an attempt to isolate important scenes and ideas.  Not all of those above will make it into the story.  And as I said, this is half of them; the other half is told in diaries, journals, and letters from her predecessor’s era, a cache of documents that stun Jera with their wanton depravity, so unlike the elegant, controlled concupiscience of her father’s Imperial harem.   And then among the documents Jera stumbles across the most dangerous find of all– the grimoire of another student of her predecessor’s mentor, a woman whose training made her cruel and dangerous.  Those post-it notes, with descriptions of various orgy scenes and other laciviousness haven’t  been put down yet.

It’s an attempt.  We’ll see how it goes.  I really should transfer these to a much larger canvas, like something from a very large sketchpad or something, but the notebook is a compromise between space and portability.