http://beta.nfb.ca/ - Canada's National Film Board is undergoing a major change to what it offers and how it offers it, courtesy of some Python technology (Django).

Full-length films available for your viewing pleasure. I particularly enjoy watching those from the 60's and 70's - it truly feels like walking back in time.

Stumbling through Python-related posts on Planet Python led me to a curious Python 3.0 discovery.

First I noted phililon wrote Chameleon: byte compiler for ZPT and Genshi ("Fricken fast", apparently), which led me to Google Spitfire (supposed to be even faster - it's certainly very fast on Python 2.5), which led me to Google's implementation of the Genshi bigtable.py benchmark.

As I'm currently beta testing / porting my code base to be compatible with Python 3.0, and never having seen a performance test I didn't want to try or mangle the interpretation of, this naturally led me to check out my choice of web framework QP's default template package QPY on Python 3.0.

Using the Google adaptation of the Genshi bigtable.py benchmark, I added an entry for QP; the benchmark results for packages which are included in or run on both Python 2.5 and Python 3.0 are interesting:

% python2.5 test/bigtable.py
ElementTree                       743.76 ms
StringIO                          137.80 ms
list concat                        22.25 ms
QPY Template                      125.84 ms

% rmtempfiles

% python3.0 test/bigtable.py
ElementTree                       412.08 ms
StringIO                          136.97 ms
list concat                        23.19 ms
QPY Template                       36.09 ms

Sadly there isn't much to compare QPY to on py3k at present. Hopefully other web frameworks, including the big ones, are busy at work producing code compatible with Python 2.x and 3.0. My purpose for doing the comparison was to see if Python 3.0 had a material effect on other templating packages, not so much to compare functionality of say Django templates to QPY as their intended audience and feature set are so very different.

Incidentally the py3k improvement for QPY represents a 71.3% speed-up for QPY on Python 3.0rc1 over what was already amply quick on Python 2.5.1. Whatever the difference is, its not unique to QPY: ElementTree also benefits greatly, turning in a 44.6% better performance.

These timings are done on an old desktop Unix workstation; more modern machines will turn in much better numbers but relatively speaking I believe these improvements will generally hold up.

On David Binger's Macbook Pro:

# py3k  22 ms
# 2.5.1 55 ms
from timeit import Timer
print(Timer('qpy_template()', 'from qpy.example.example2 import
qpy_template').timeit(1000))

Using timeit to get time for qpy_template() call:

25   ms  Python2.5.1 with quote.so
50   ms  Python2.5.1 without quote.so
11.7 ms  Python3.0 with quote.so
44   ms  Python 3.0 without quote.so

Compiled speedup is 25/11.7 = 2.1
Uncompiled speedup is 50/44 = 1.1

David also notes:

When using profile to test template execution, the it looks like we get a speedup from 254 ms to 199 ms per qpy_template() call between Python 2.5.1 and Python 3.0. Nothing in the profile jumps out at me as being responsible for the change. The C code does, though, do some extra work for Python 2 to force the default encoding to be utf8. Maybe that adds up to a significant amount of time.

QPY? The QPY template model is going to look upside down if your experience or preference is with any of the many code-in-content template packages out there. Here's qpy-enabled Python source compared to a couple of other templating approaches:

table = [dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7,h=8,i=9,j=10)
      for x in range(1000)]

# QPY
def qpy_template:xml():
    '<table>'
    for row in table:
        '<tr>'
        for col in row.values():
            '<td>%s</td>' % col
        '</tr>'
    '</table>'

# Genshi
def test_genshi():
    stream =  MarkupTemplate("""
    <table xmlns:py="http://genshi.edgewall.org/">
    <tr py:for="row in table">
    <td py:for="c in row.values()" py:content="c"/>
    </tr>
    </table>""").generate(table=table)
    stream.render('html', strip_whitespace=False)

# Django
def test_django():
    context = DjangoContext({'table': table})
    DjangoTemplate("""
    <table>
    {% for row in table %}
    <tr>{% for col in row.values %}{{ col|escape }}{% endfor %}</tr>
    {% endfor %}
    </table>""").render(context)

QPY makes writing quote-safe 'templates' as easy as writing Python code itself.

One of these days I'm sure to need a code-in-content solution and for raw performance alone the Cheetah inspired Google Spitfire is sure to be on my list of packages to look at. But I hope that day doesn't come soon as I've grown rather fond of the content-in-code approach of QPY which allows me to write Python code that looks like Python code.

A version of QP supporting Python 3.0 (and prior Python versions of course) will be released when Python 3.0 soon hits the streets. While doing some testing I created a lighttpd configuration example illustrating setting up a virtual host with SCGI (QP provides http and scgi servers natively) and SSL support:

$ cat server.key server.crt > server.pem

server.modules = ("mod_access",
                  "mod_scgi",
                  "mod_accesslog")

$HTTP["host"] == "www.example.com" {
    $SERVER["socket"] == "0:443" {
        ssl.engine = "enable"
        ssl.pemfile = "/my/ssl/server.pem"
        ssl.ca-file = "/my/ssl/server.crt"
    }
    scgi.server = ( "" =>
        ( "localhost" =>
            ( "host" => "127.0.0.1",
              "port" => 9000,
              "check-local" => "disable",
              # default disable-time is 60 seconds
              "disable-time" => 5 # seconds
            )
        )
    )
}

==================================================
QP web server direct    2.38 [ms] (mean)
lighttpd 1.4.20
lighttpd proxy to QP    2.41 [ms] (mean)
lighttpd SCGI           2.31 [ms] (mean)

APACHE 1.3.41
APACHE 1.3 proxy to QP  3.39 [ms] (mean)
APACHE 1.3 mod_scgi     2.30 [ms] (mean)
==================================================

The attached image is a screen shot of a web browser running on a Nokia N800, a Linux-based internet tablet. Weighing only ounces, the wireless device is a great platform for Python developers as the language has more or less become the default dynamic language for the device and, it would seem, for Nokia. GUI developers commonly employ pygtk/glade for Maemo applications; I'm not aware of much web development being done on the N series tablets as yet.

Pictured is the output of a highly Pythonic web framework and object database combination, QP and Durus - the app is merely a template provided by running mkqpapp.py.

I wanted to see what QP and this little tablet, underpowered by laptop or server standards, could do - here is a benchmark between a fairly fast Unix desktop, across a fairly slow wireless link, generating 10 concurrent request streams to the device

frog# /home/mw% siege -b -c10 -t10s http://n800:8000/
Transactions:                164 hits
Availability:             100.00 %
Elapsed time:               9.85 secs
Data transferred:           0.06 MB
Response time:              0.58 secs
Transaction rate:          16.65 trans/sec
Throughput:                 0.01 MB/sec
Concurrency:                9.71
Successful transactions:     164
Failed transactions:           0
Longest transaction:        3.66
Shortest transaction:       0.19

Not bad, considering its a full stack web framework and object database running on a little machine weighing only ounces that also is running what amounts to be a Gnome environment, browser, mail and other apps, all powered by a lowly TI 320MHz Armel architecture CPU.

Out of curiosity I added a hit counter to exercise the object database and the transaction rate was a respectable 12.36/second.

Worlds smallest portable web application demo machine!

Once I figure out how to make Debian packages for the armel architecture I'll post a deb link for a one click install of Durus, QP, QPY and Dulcinea.

Incidentally, while sqlite is in common use on Nokia tablets, there's clearly no reason why Durus could not be used. N-series Python developers might find that to be an ideal persistence pairing to go along with their web or GTK apps.

According to Simon Willison, Django now has auto-escaping of variable contents within templates. Good job.

I wouldn't even consider using a framework that didn't provide some sort of sane system for auto-quoting that which has not already been explicitly declared as safe. Both Quixote and QP have had this feature for many years.

Quixote's PTL template approach gained this capability circa 2002; QP's QPY, which is a very close cousin to PTL but decoupled from the web framework itself, has always had this ability.

In October 2003 one of Quixote's original authors, Neil Schemenauer, put forward a proposal that ultimately led to the implementation of a sane auto-quoting behaviour in PTL templates. (See also his PyCon 2003 talk)

In the fall of 2005 QP - a slightly more opinionated version of Quixote - was born of the same development shop, along with QPY, an unbundling of the PTL concept with string-like classes that make it easy to work with, virtually interchangeably, quote-aware and plain str / unicode objects.

QPY's smart string-like classes could be used by any templating system or web framework that hasn't solved this problem already.

In the summer I was writing a series of articles - a web application diary of sorts - illustrating web development with Python and the QP web framework. After several thousand words and seven installments, the series paused back in July.

As I get my obligation stack beaten back to merely mountainous I plan on continuing with the series and indeed have plans to re-factor the whole lot and make it available in a subversion repository, step by step. The latter idea is not mine; I ran across another such article series in the Pythonosphere which had used subversion to good purpose.

To help encourage me to write more, I jumped ahead and put a slightly enhanced version of the Journal application on-line as of the last couple of days; all the old Pyblosxom hosted articles have been imported.

As contemplated back in June, I did decide to embrace breakage - all the old content has shiny new URL's, although a simple url_mapping dictionary and a 301 Moved Permanently redirect have more or less looked after avoiding 404 errors all over the place. Pretty easy to do, this is part of my SiteDirectory class:

def _q_traverse(self, path):
    """
    First check to see if request is hunting for an old URL,
    redirect if so.
    """
    uri = get_request().get_path_query()
    dest = get_publisher().get_root()['url_mapping'].get(uri, None)
    if dest:
        return redirect(dest, permanent=True)
    else:
        return JournalDirectory._q_traverse(self, path)

Incidentally my url_mapping has almost 1700 different combinations pointing to 478 content items and various query strings and other site features. Most of the values in the mapping were created during the import of the old data; apparently I was less than consistent in the way I pushed out Pyblosxom over the years I used it. Now at least there is one URI for each resource and no more.

Back soon with part eight of the series.

Python object databases need some love too

Flávio Coelho recently performed an examination of various Python database API and ORM interfaces to MySQL, Postgres, and SQLite, and included a benchmark for cPickle.

Here's an addition to Flávio's Fastest Python Database Interface article and script to include Durus: performance.py.

I also pointed out on Flávio's blog that his cPickle benchmark needed to include pickling 100,000 "Person" classes, in addition to 100,000 simple tuples - this to show the overhead of class instantiation and serialization / deserialization which all of the ORM's and object databases share in some form or another. An example of both can be found in performance.py.

User interface handling in QP

In part six of Python Web Application Diary we looked at data persistence using Durus and wrote Sancho unit tests and a skeleton Pyblosxom migration tool. Today we'll extend the basic UI created in part five for our Entry object, and we'll create a base UI for the Journal object too. At that point we'll be able to wire our new JournalDirectory into SiteDirectory (also referenced in part five) - a QP application's "master controller" - which will allow us to start publishing previously created journal entries to the web.

class JournalDirectory(Directory):
    """
    This class provides the functionality to display recent entries,
    to navigate to a specific entry, and to present RSS and Atom feeds.

    Our application urls for this UI component will be:

    ../             "index" or recent entries
    ../1234/        calls the EntryUI "index" method
    ../new          create a new Entry
    ../index.rss
    ../index.xml
    """

    def get_exports(self):
        yield ('', 'index', None, None)
        yield ('index.rss', 'rss', 'RSS', 'RSS feed for this journal')
        yield ('index.xml', 'atom', 'Atom', 'Atom feed for this journal')

    def __init__(self, journal):
        require(journal, Journal)
        self.journal = journal

    def index [html] (self):
        title = "%s's journal" % self.journal.get_name()
        header(title)
        '<p>This is the journal of '
        self.journal.get_name()
        '</p>'
        for e in self.journal.get_recent_entries(count=10):
            '<p>%s %s %s</p>' % (e.key, e.get_text().get_format(),
                                 href(str(e.key),e.get_title()))
        footer(title)

    def new [html] (self):
        ' not implemented '

    atom = rss = new

<a href="./1234">Some title</a>

http://mikewatkins.ca/blog

http://mikewatkins.ca/blog/1234
http://mikewatkins.ca/blog/3456

def _q_lookup(self, component):
    try:
        key = int(component)
    except ValueError:
        return not_found('%r is not a valid journal entry identifier.' % component)
    try:
        entry = self.journal.get_entry(key)
    except KeyError:
        return not_found('%r was not found.' % key)
    return EntryUI(entry)

Object              UI
--------------------------------------------
Entry               EntryUI
Journal             JournalDirectory
JournalDatabase     JournalDatabaseDirectory

class SiteDirectory(Directory):

    def get_exports(self):
        yield ('', 'index', 'Home', 'The Home page of this site.')
        yield ('blog', 'blog', 'Weblog', None)

    def index [html] (self):
        title = get_site().get_name()
        header(title)
        '<p><strong>It worked!</strong></p>'
        '<p>The <strong>%s</strong> application lives at <br /><tt>' % title
        get_site().get_package_directory()
        '</tt></p>'
        footer()

    mw_journal = get_publisher().get_root()['mw_journal']
    blog = JournalDirectory(mw_journal)

class SiteDirectory(JournalDirectory):

    def __init__(self):
        # I've implemented the JournalDatabase object and migrated some
        # data into a Journal.
        mw_journal = get_publisher().get_root()['journals'].get('mw')
        JournalDirectory.__init__(self, mw_journal)

In part six of Python Web Application Diary it was noted that soon we'll have a functioning, if basic, journal application. Since I plan to follow the release early release often plan for this particular application -- otherwise a desire for perfectionism will never see the project complete -- now is good time to think about areas of breakage that may occur.

The number one and two issues that come to mind are:

1. URL breakage: Will the > 470 journal entries moving over from Pyblosxom to the new application lose their existing URLs? I really dislike the Pyblosxom URL scheme, and will probably vote to go with breaking it. I'm a bad person. It won't happen again. Promise.
2. Feed breakage: New URL's and possibly new entry id's tend to do nasty things to feeds, or rather, to the consumers of feeds. I'm tempted to discontinue RSS support while I'm going through this process but need to review my logs to see what the impact of that decision would be.

In fact feed breakage is my larger concern - the last thing I want to do is subject the official or unofficial Python community feed aggregators and their readers to dozens of old and now mostly irrelevant posts.

Ironically that same issue is why I've never asked to be included in either feed, at least not until I moved the data off to another platform.

For now, until I flip the switch, I've disabled the entry / item content in my existing Atom and RSS feeds by creating empty Pyblosxom flavour files for story.xml and story.rss -- this will result in no blog entries being published via either feed. I've also added to the description element in both feeds a notice:

<description>$blog_description
:: Feed off-line while upgrades being done
</description>

Coming up in part seven of the series, accomplishing enough to flip the switch.

In part five of this series we dove deep into QP and looked at the fundamentals of any QP application - SitePublisher and SiteDirectory - as well as explored the use of QPY templating. We also built a rudimentary UI for our Entry object.

In this installment of our web application diary we'll work more with the Durus object database by injecting some data into it; exploring the interactive interpreter (one of the cool features of Durus to be sure) and starting the basis for a conversion script to take weblog data in PyBlosxom format and insert it into our blog application database.

pythoni

% qp -i blog
Profile, connection, publisher, root, sessions, site, users
->>

->> pu

->> from durus.persistent_dict import PersistentDict
->> mydict = PersistentDict()
->> root['test'] = mydict
->> connection.commit()
->>
%

% qp -i blog
Profile, connection, publisher, root, sessions, site, test, users
->>

->> test
<PersistentDict 17020>
->> test.items()
[]

->> test[1] = 'My first persistent data'
->> connection.commit()
->>

% qp -i blog
Profile, connection, journals, publisher, root, sessions, site, test, users
->> test.items()
[(1, 'My first persistent data')]
->>

from dulcinea.base import DulcineaPersistent
from dulcinea.sort import attr_sort
from qp.lib.keep import Keep, Keyed, Stamped
from qp.lib.spec import add_getters_and_setters, boolean, both, datetime_with_tz
from qp.lib.spec import init, pattern, string, spec
from qp.pub.user import User


class Entry(DulcineaPersistent, Keyed, Stamped):
    """
    An entry in a journal.
    """
    title_is = spec(
        (string, None),
        "A string briefly describing the Entry")
    text_is = spec(
        (string, None),
        "The entry conten")
    published_is = spec(
        boolean,
        "Boolean indicating if Entry can be published")
    author_is = spec(
        User,
        "User responsible for creating entry")
    created_is = datetime_with_tz

    def __init__(self, author):
        Keyed.__init__(self)
        Stamped.__init__(self)
        init(self, author=author, created=self.stamp, published=False)

add_getters_and_setters(Entry)

from parlez.journal import Journal

class JournalTest(UTest):
    # we'll write this first, and then write Journal

    def _pre(self):
        # set up a journal which we'll use for most tests.
        self.j = Journal('science', User('einstein'))
        # it is automatically taken down following each individual test

    def init_test(self):
        # we want Journal to have a URL name and an owner, so force it
        Journal('musings', User('joe'))

    def create_entry_test(self):
        assert isinstance(self.j.create_entry(), Entry)

    def add_test(self):
        e = self.j.create_entry()
        self.j.add(e)
        assert e in self.j.get_all_entries()
        assert e == self.j.get_entry(1)

    def only_published_test(self):
        # nothing in
        assert self.j.get_all_entries() == []
        e = self.j.create_entry()
        self.j.add(e)
        e_published = self.j.create_entry()
        e_published.set_published(True)
        self.j.add(e_published)
        assert e not in self.j.get_entries()
        assert e_published in self.j.get_entries()
        # publish e now
        e.set_published(True)
        assert e in self.j.get_entries()
        # both should be in reverse sorted result, e last
        assert [e_published, e] == self.j.get_recent_entries()

if __name__ == '__main__':
    EntryTest()
    JournalTest()

../entries/categoryname/file1.txt
../entries/categoryname/someotherfile.rst
../entries/python/2007-06-08-08-44.rst

Some article title
#author Mike Watkins
The article content.

.h2 A subtitle

More content. Etc.

from qp.lib.site import Site
from parlez.journal import Entry, Journal

def bloxsom_to_mapping(entrypath):
    # here you'll deal with the specifics - see a future article
    data = {}
    # ...
    return data

def add_journal_entries(data, journal):
    for path, entry_data in data:
        # path I might store, or some component of it, in the Entry
        # object to facilitate mapping old to new URLs in the future.
        # for now, just ignoring it
        created, format, title, content = entry_data
        entry = journal.create_entry()
        entry.set_format(format)
        entry.set_title(title)
        entry.set_text(content)
        # normally we don't bypass getters/setters
        entry.created = created
        entry.stamp = created
        journal.add_entry(entry)

if __name__ == '__main__':
    BLOXSOM_ENTRY_PATH = '/home/mw/bloxsom/entries'
    APP_NAME = 'blog'
    JOURNAL_NAME = 'mw'
    USER_ID = 'mw'

    # the Site object gives us the ability to access
    # configuration information and live objects
    site = Site(APP_NAME)
    pub = site.get_publisher()
    root = pub.get_root()
    users = root['users']
    # make sure I exist in Users
    if USER_ID not in users:
       user = pub.create_user(USER_ID)
       users.add(user)
    if 'journal' not in root:
        journal = Journal(JOURNAL_NAME, user)
        root['journal'] = journal
    # move bloxsom data into Entry/Journal
    add_journal_entries(bloxsom_to_mapping(BLOXSOM_ENTRY_PATH),
                        journal)
    # made it here, commit everything to the database
    pub.get_connection().commit()
    # that's it!