Elf Sternberg

I wish I’d known this a long time ago.  Django’s request object includes a dictionary of key/value pairs passed into the request via POST or GET methods.  That dictionary, however, works in a counter-intuitive fashion.  If a URL reads http://foo.com?a=boo, then the expected content of request.GET['a'] would be 'boo', right?  And most of us who’ve used other URL parsers in the past know that http://foo.com?a=boo&a=hoo know that the expected content of request.GET['a'] would be ['boo', 'hoo'].

Except it isn’t.  It’s just 'hoo'.  Digging into the source code, I learn that in Django’s MultiValueDict, __getitem__(self, key) has been redefined to return the last item of the list.  I have no idea why.  Maybe they wanted to ensure that a scalar was always returned.  The way to get the whole list (necessary when doing an ‘in’ request) is to call request.GET.getlist('a').

Lesson learned, an hour wasted.

Recently, I had the pleasure of attending another of those Seattle Django meet-ups.  This one was a potpourri event, just people talking about what they knew and how they knew it.   I revealed that I’d written my first aggregator, and that seemed to be an impressive statement.  Apparently Django Aggregators (database conditionals that perform sub-selected summarizing or filtering events) is something of a black art, much like Wordpress Treewalkers were a black art I figured out in just a few hours.

Aggregators consist of two parts: The Definition and the Implementation.  Unfortunately, Django’s idea is that these are two different objects, bound together not by inheritance but by aggregation (both the definition and the implementation are assembled in a generic context, one providing access to the ORM and the other to the SQL).   The definition is used by the ORM to track the existence and name of the aggregate, and is then used to invoke the implemenation, which in turn creates the raw SQL that will be added to the SQL string ultimately sent to the server, and ultimately parsed by the ORM.

I needed to use aggregation because I wanted to say, “For any two points’ latitude and longitude, give me the great circle distance between them,” and then say, “For a point (X, Y) on a map, give me every other place in the database within n miles great circle distance.”

The latter was not possible with Django’s Queryset.extra() feature.    You can add a WHERE clause, but not a HAVING clause, and this definitely requires a HAVING clause when running on MySQL.  Using an Aggregator with a limit forces the ORM to realize it needs a HAVING clause.  Besides, it was a good excuse to learn the basics of Aggregation. Ultimately, I was able to do what the task required: find the distance between any two US Zip Code regions without making third-party requests.

I make absolutely no promises that this code is useful to anyone else.  The Aggregator is definitely not pretty: it’s virtually a raw SQL injector.  But it was fun.  Enjoy: Django-ZipDistance.

Following up on the The Onion’s announcement that they’re using Django comes this priceless discussion of the technical challenges of doing so with several members of The Onion’s technical team. They were using Drupal before.

Among the things I discovered:

• Grappelli, a customizable theme for the Django admin
• uWSGI, a high-performance WSGI container separated from Apache,
• HaProxy, a viable open-source TCP/IP Load Balancer.

I’m constantly reminded, when I work on IndieFlix, “You’re not YouTube. Don’t code like you are ever going to be YouTube.” And they’re right. If I ever reach that level of technical difficulty, I’ll deal with it then. But we very well could have traffic issues similar to The Onion’s, and that’s not a bad target to aim for.

Also not to be missed in this conversation: The Onion cut 66% of their bandwidth by upstream caching 404s

Java is Pass-By-Value, Dammit!

Quite possibly the most important article I’ve ever read, because it finally, finally explains to me what Java’s object-passing model is really all about. I’ve never understood it, and now I do: it’s exactly backwards from pass-by-reference, so it’s exactly backwards from the languages with which I grew up. The Object (and derivative) variables are pointers, not references, and calling them references only confuses people who grew up writing C (as I did).

Even more oddly, it explains to me what I’ve never quite understood about Python’s object model, because Python’s object model is exactly the same.   Reproducing the code in the article above in Python creates the same result:

class Foo(object):
    def __init__(self, x): self._result = x
    def _get_foo(self): return self._result
    def _set_foo(self, x): self._result = x
    result = property(_get_foo, _set_foo)

def twid(y):
    y.result = 7

def twid2(y):
    y = Foo(7)

r = Foo(4)
print r.result  # Should be 4
twid(r)
print r.result  # Should be 7

r = Foo(5)
print r.result  # Should be 5
twid2(r)
print r.result  # Still 5

This demonstrates that Python’s code remains pass-by-value, with pythonic “references” in fact being pointers-to-objects. In the case of twid2, we change what the pointer y, which exists in the frame of the call twid2, points to and create a new object that is thrown away at the end of the call.  The object to which y pointed when called is left unmolested.

This is important because it changes (it might even disrupt) the way I think about Python. For a long time, I’ve been using python calls out of habit, just knowing that sometimes objects are changed and sometimes they aren’t.  Now that the difference has been made clear to me, in language that I’ve understood since university, either I’m going to be struggling for a while incorporating this new understanding, or I’m going to be much more productive.

Do you have that one thing that you have to constantly look up?

In python, to replace elements of a string, there are two operators.  One is a strictly linear search, the other uses regular expressions.  The regexp call to replace part of a string with another string is sub, and the string call to replace a part of a string with another string is replace.

I get these backward all the flamin’ time, and it drives me insane.  I use both of these every day, why the Hell can’t I remember which is which?

Today’s little snippet: Filtering a loosely coupled many-to-many relationship.  As revealed earlier,  I don’t really “get” the difficulty with many-to-many relationships.  I don’t even get the difficulty with signals; if you define the many-to-many manually, handling signals on it is trivial compared to trying to do it manually in one of the referring classes.

Today, I was working on an issues tracker.  There are two classes at work here, the Profile and the Issue.  One profile may be interested in many Issues, and obviously one Issue may be of interest to many Profiles.

This calls for a ProfileIssue table that stands independent (in my development paradigm) of both Profiles and Issues.   As I was working on a dashboard, I realized that one of the things I wanted was not just a list of the issues the profile was following, but also a list of the issues that the profile was responsible for creating.  As it turned out, adding that query to the ProfileIssueManager is trivial, but requires a little knowledge:

class ProfileIssueManager(models.Manager):
    def from_me(self, *args, **kwargs):
        return self.filter(issue__creator__id = self.core_filters['profile__id'])

The secret here in knowing about the core_filters attribute in the RelatedManager.   It contains the remote relationship key that you can use;  calling from_me from profiles works, but calling it from anywhere else doesn’t.  The IssueRelatedManager won’t have a profile_id and this will blow up.  That’s okay; using it that way is an error, and this is a strong example of Crash Early, Crash Often.

I can here some of you cry, “Now why, why would you need such a thing?” Well, the answer is pretty simple: templates. Having one of these allows me to write:

<p>Issues tracked: {{ profile.issues.count }}</p>
<p>Issues created: {{ profile.issues.from_me.count }}</p>

And everything will work correctly.

The correct call for posting to a user’s facebook wall with Python and pyfacebook, after you’ve established both user authentication via FacebookConnect and gotten stream_publish permission, is:

request.facebook.stream.publish(
    message = render_to_string(template_path, fb_context),
    action_links = simplejson.dumps(
        [{'text': "Check Us Out!", 'href': "http://someurl.com"}]),
    target_id = 'nf')

See that ‘nf’ down there in target_id?  It’s not on any of the Facebook documentation pages, but that is the correct string to post to your user’s facebook Newsfeed. (For that matter, the fact that you have to run the action_links through simplejson, and that they have to match the PublishUserAction API action_links spec, is also not documented; the documentation says it just needs an array of arrays.)  I have no idea how to post to some other user’s newfeed, but at least I’m one step closer.

Oh, another important tip: in order to make my news “stories” consistent, I’m using a template to post them to Facebook.  The template must not have newlines within, or they will show up on Facebook and it’ll look all ugly.  Every paragraph should be one long line of text without line breaks.

I was reading through the Wordpress source code, trying to figure out a problem for a contractor, when I saw the function compact().  When I saw it I boggled, read the description, and shook my head.

Compact() takes a list of variable names as strings, and returns a hash of those variable names and their values.   So if you have something like:

$title = "My blog";
$link = "foo";
$h = compact('title', 'link');

You get back a hash of array(’title’ => ‘My blog’, ‘link’ => ‘foo’).

That, to my thinking, is completely messed up.  You’re giving this function, which has its own scope, explicit permission to twiddle with variables in the current scope and create a new variable.  It’s one of those things that convinces me that PHP is an unholy mess of silliness.

And then my brain reminded me that, hey, you can do the exact same thing in python.  So, I have:

import inspect
def compact(*args):
    return dict([(i, inspect.currentframe().f_back.f_locals.get(i, None))
                 for i in args])

def foo():
    a = "blargh"
    b = "bleah"
    c = ['1', '2', '3']
    return compact('a', 'b', 'c')

print foo()

And sure enough, this spits out: {’a': ‘blargh’, ‘c’: ['1', '2', '3'], ‘b’: ‘bleah’}

I hang my head in shame for giving Django developers one more thing around which they can develop bad habits.

Last year I briefly flirted with Ruby.  I once infamously described Python as “Perl for grown-ups,” and if that’s true, then Ruby is the hot 20-something firecracker chick of your programming mid-life crisis.  You’ve got nothing in common but damn the interfacing is great.  Maturity is getting over it and coming back to the stable reliability and maintainability of Python once more

But if Python is the language of the mature developer, than Django is that stage of midlife where life becomes way too complicated.  (Studies indicate that 44 is “the worst year of life” for most people: kids, family, career, finances, professional, social and romantic obligations all pile on hard around this time, leaving most people too little time to think or plan for happiness.)

I’ve been spending the day trying to port narrator, the program suite that runs my story site, from Ruby to Python.  Ought to be easy, right?  I decided to make it harder by porting it to Django instead.  I have the data models, and they’re great for Python.  Easy to port.  But I had to make it harder on myself, by trying to create containerized subsystems for my stuff: stories belong to authors, but authors might have series, series might have novels as well as stories, but there might be standalone novels, and you might have series so short and standalone that they go in your short story collection with an ident or something.  This immediately suggested to me a database trick I know called modified preorder tree traversal, a technique in which allows you to store hierarchical information in a meaningful manner.  There’s even a pre-build MPTT script for Django unmysteriously named django-mptt.

And that’s where I wandered off into the weeds.  I tried to understand how the mptt worked and how to incorporate it into the models I was building, and eventually my head started to explode.  It involved something called “generics,” which are a contributed library for Django’s ORM that uses metaprogramming to create one or more model classes with a many-to-many relationships to many objects in many different models.  It’s very cool.  It’s very esoteric.  It’s very hard to understand.  The layers between implementation and concrete realization are many and intertwined.

One of the differences between Rails and Django is that Django is “just” a bunch of Python libraries loosely assembled.  But Python encourages reading the source; that’s the idea, it’s supposed to be a readable, self-documenting language.  And sure enough, it is, if the problem domain is small enough.  It’s when you start to layer domain on top of domain, solution on top of solution, that the system become unwieldy.  The headaround for some Django applications is beyond the average programmer (and believe me, I am an average programmer).  Assaf Arkin encapsulated this idea perfectly in a recent post about object-oriented programming, quoting Travis Jensen:

“My point is that the architectural complexity of these applications inhibit a person’s understanding of the code. … Without actually doing anything, applications are becoming too complex to understand, build, and maintain.” Every layer makes perfect sense in isolation. The cracks start showing when you pile them up into a mega-architecture.

This seems to be the problem even with small Django applications. By presenting the four components of Django– the ORM, the Templating language, the Route/View library, and the Administrative envelope– as four separate components– The Django Project has greatly increased the cognitive load on the programmer.  Django becomes harder to learn than Rails because of the extra mental effort needed to grasp all the intricacies.

There is a perfect irony here: Django is loosely coupled enough that you can do whatever you want with it and it will probably work.  But understanding what crosses that loose coupling is difficult, and when you’ve got all the layers, plus contribs and plugins going, you need  to pull out pencil and paper for even the smallest of efforts.  Rails, in contrast, is a hodge-podge of different technologies all thrown together into a pot, but because the average web developer is actively discouraged from being curious about more than what he needs to get the job done, Rails is easier to grasp from the very start.

It’s probably worth it.  The payoff is a Deep Understanding of the joys of Python metaprogramming.  Done right, I’ll probably have an even better grasp of the abstractive power of it all.  But it’s a long, slow slog.

A couple of weeks ago I bought the book “The Definitive Guide to Django,” and I’ve come to realize, to my frustration, that the book is already outdated.  My big headache this week was dealing with the administration interface, which the Django people swear is one of the coolest features of the entire application server.  The problem is simple: between Django 0.96, which is when the book was written, and Django 1.0, which is what I’m running, the interface was completely changed.

In 0.96, the way you defined a database table as being “administratable” was to add to the Python definition of the table a subclass entry class Admin: pass.  The Administration app would automagically pick out those tables that were administratable and they would appear in the admin interface.

In 1.0, it’s completely different.  Instead, you must create in your application, next to your models, a file admin.py which contains registry lines for each model and, optionally, an administration interface class that describes how the model should be administered.  It’s all covered in the Django tutorial which doesn’t really help you if all you have is the book.

This makes obvious sense. Administrative details are independent of model details, and although the argument could be made (and was made) that they’re implementation details of the model, making it a separate decorating class also makes just as much sense. Yes, it means that the details of a class (the administrative class of a model) are in two different places, but it also means that administrative features of your application can be restricted in deployment just by deleting the admin.py file.

You did keep a copy of admin.py in your source control, right?