Overview of Firmant

Firmant is a static web framework written in Python. It is static, meaning no code is run to generate the response to a client's request; instead, all code is run at compile time (the time at which the site is "compiled" to a static website). It is a web framework as it facilitates the construction of websites/applications. It is written in Python because I ♥ Python (yes, Firmant supports unicode).

I've explained my rationale for creating such a project on the front page of the Firmant documentation.

Bootstrapping the Tutorial

The code referenced in the tutorial is available in usable form from my public git server.

To follow the tutorial you will need to install Firmant and its dependencies. I used Fedora 12 (but also use Firmant from Fedora 13). The dependencies include:

• decorator
• Docutils
• jinja2
• lxml
• PySettings
• pytz

Other dependencies may be necessary (e.g. if you wish to run the doctests or build the documentation).

Creating the new project

I create a new project and include the complete license text of the BSD license. Whenever creating a new repository to be exposed to the public, my first commit always expresses my desired license or copyright so that others know that they may use the code I publish. You can see this in commit debbde7a815b1a89861748ada7ff435e4270b594.

The next step I do whenever building a Firmant-based application is to add a suitable Makefile and empty settings file. For those following along, this happens in commit 963010723e0e3373948ce434be34a3546c84ae86.

The key thing to note is the way in which the firmant script interacts with the settings and the environment. In the Makefile, my rules have the structure:

env FIRMANT_OUTPUT_DIR=preview \
    FIRMANT_PERMALINK_ROOT=file://`readlink -f preview` \
    firmant settings

Environment variables of the form FIRMANT_x=string will be added to the settings object as x = string. I do this so that I may have one config for the site, and override the output directories and permalink roots. The output directory is a local filesystem directory in which all output files will reside. The permalink root is the base url where the files will be published. In the above example, they correspond to a preview directory and the URL to its absolute path on the filesystem.

I also take the opportunity to create a blank module for the application (seen in commit eb6a01c0467dfb4b9f63caa43a17f266b70d9029).

Creating Objects and Parsers

All objects must inherit from firmant.parsers.ParsedObject which provides a constructor to allow keywords that correspond to the object's slots. Object instances must provide an _attributes property that is a dictionary that will be used as the attributes for url mapping in Firmant. This causes objects to automatically have a permalink property set to the URL derived from the object's attributes (don't worry too much about this for now, I'll elaborate on URL routing later, for now, we just need to determine the permanent URL for our object).

I've decided to use the reStructuredText syntax for our wiki object. Firmant provides special support for this using firmant.parsers.RstParsedObject. The RstParsedObject accepts an additional _pub attribute which corresponds to the docutils publisher object. The user is expected to declare _pubparts as a list of two-tuples. The first value of each tuple is the object's attribute. The second value of each tuple is the part provided by the docutils HTML writer.

Bringing this all together gives us (found in cfbf2707165c78a108ff9dd029c597f34997c2cf):

class WikiObject(parsers.RstParsedObject):
    __slots__ = ['path']

    _pubparts = [('content', 'fragment')
                ,('title', 'title')
                ]

    def __repr__(self):
        return 'WikiObject(%s)' % getattr(self, 'path', None)

    @property
    def _attributes(self):
        return {'path': self.path}

The representation is just for debugging purposes and not necessary. Notice how we have declared the explicit attributes (the path of the wiki object, e.g., CreatingAFirmantWiki), and we have implicitly declared content and title as being derived from the written docutils code. Our wiki objects are unique to the path at which they reside.

The parser for creating a WikiObject is not much longer (still in the same commit):

class WikiParser(parsers.RstParser):
    type = 'wiki'
    paths = '.*\.rst'
    cls = WikiObject

    @decorators.in_environment('settings')
    def root(self, environment):
        settings = environment['settings']
        return os.path.join(settings.CONTENT_ROOT, settings.WIKI_SUBDIR)

    def rstparse(self, environment, objects, path, pieces):
        attrs = {}
        attrs['path'] = unicode(path[:-4])
        attrs['_pub'] = pieces['pub']
        objects[self.type].append(self.cls(**attrs))

Here I have used the special firmant.parsers.RstParser. Starting at the top of the class declaration:

1. type is declared to be wiki. This should be a human-readable string and will be displayed to the user when parsing.

2. paths is a regular expression that declares which objects will be parsed The path of every file under the directory returned by root (relative to root) will be tested against this regex, and only those that match will be parsed.

3. cls is defined to be WikiObject. This is just a good practice as it makes it easy to copy a parser and change it to create objects of a different type.

4. root is a function that returns the path to the root on the filesystem where all objects of this type reside. I've made this configurable using the settings object. Only objects under root that match the paths regular expression will be parsed.

5. rstparse is a function specific to the RstParser. It takes an environment (e.g. where settings are defined), dictionary of objects (possibly empty), the path to the file from which this object was derived, and the pieces parsed from the object. For now we're only concerned with the docutils publisher object ("pub").

   The rstparse function is expected to append the parsed object to lists in the dictionary. It does this instead of simply returning the object as this will allow multiple objects to be created from a single parser in the future (e.g., LaTeX embedded in a post is parsed into an object when the post is parsed, and then this is written to an image file).

With the new object we must declare several settings:

# The wiki's source files reside in the current directory (assuming the make
# file is invoked from this directory).
CONTENT_ROOT = '.'

# This enables our custom wiki parser.
PARSERS = ['firmantwiki.WikiParser']

# The directory (under CONTENT_ROOT) used for storing wiki documents.
WIKI_SUBDIR = 'wiki'

# The URL mapping.
from firmant.routing import components as c
URLS = [c.TYPE('wiki') /c.PATH]

# Permalinks for our wiki objects must be to the html rendering.
PERMALINK_EXTENSIONS = {'wiki': 'html'
                       }

The only two settings that really need explanation are URLS, and PERMALINK_EXTENSIONS.

URLS

This is used for URL routing. This will be explained more in a later section.

PERMALINK_EXTENSIONS

A dictionary mapping types to the extension that is used for the permalink. For example, declaring "html" will ask the URLMapper (explained later) to map the permalink for this object to an HTML document. A value of None implies that the constructed URL will contain an extension. This will become more clear in the URL routing section.

Creating a Sample Wiki Page

I added the following wiki page as wiki/index.rst (commit 2c825b17eab148bfdf18cde7c805cf1b27adf3e8 for those with a score card):

Firmant
=======

Firmant is a framework for developing static web applications.

Much of today's web development focuses on developing dynamic applications
that regenerate the page for each view.  Firmant takes a different approach
that allows for publishing of static content that can be served by most http
servers.

Some of the benefits of this approach include:

 * Build locally, deploy anywhere.  Many notable server distributions
   (including CentOS 5, and Debian Lenny) still ship old (pre-2.6) versions
   of Python.  With Firmant, this is not an issue as static output may be
   published anywhere independent of the system where it was built.
 * Quicker page load times.  Search engines and viewers expect near-instant
   page load times and static content can meet these expectations.  Dynamic
   content can as well; however, it often requires more than simple hardware
   to do so.
 * Offline publishing capability.  Previewing changes to a website does not
   require Internet access, as the changes are all made locally.  Changes do
   not need to be pushed to a remote server.
 * Store content in revision control.  This is not strictly a feature granted
   by generating static pages.  Firmant is designed to make storing all
   content in a repository a trivial task -- something that web application
   frameworks that are powered by relational databases do not consider.

This wiki page will have a path of index, and a title of Firmant. The content will be an HTML version of the body of the page. If we were to run make right now we would see:

env FIRMANT_OUTPUT_DIR=preview \
    FIRMANT_PERMALINK_ROOT=file://`readlink -f preview` \
    firmant settings
INFO:firmant.application.Firmant:firmantwiki.WikiParser parsing 'index.rst'
xdg-open preview/index.html

It's clear that the parser is actually parsing the index wiki object, but our web browser does not show any output. To actually see the parsed page, we need to create a writer.

Creating a Writer

Firmant makes rendering HTML using Jinja2 as easy as pie. Writers must inherit from firmant.writers.Writer. For convenience, I've also created firmant.writers.j2.Jinja2Base which enables easy rendering of Jinja2 templates to the filesystem.

The entire writer code is:

class WikiWriter(j2.Jinja2Base, writers.Writer):
    extension = 'html'
    template = 'wiki.html'

    def render(self, environment, path, obj):
        context = dict()
        context['path'] = obj.path
        context['page'] = obj
        self.render_to_file(environment, path, self.template, context)

    def key(self, wiki):
        return {'type': u'wiki', 'path': wiki.path}

    def obj_list(self, environment, objects):
        return objects.get('wiki', [])

From top-to-bottom of the class declaration:

1. extension this is the extension the writer will use for writing the wiki objects. This will make more sense when I introduce URL routing.
2. template this is the name of the Jinja2 template that will be used when rendering the wiki objects.
3. render populates the Jinja2 context and calls the render_to_file helper function. Typically this is as short as creating a dictionary and calling render_to_file.
4. key is a function that produces a URL attribute dictionary from a single object. (Once again, more on this in the URL Routing section).
5. obj_list is a function that returns a list of Python objects. In this case it is simply a list of wiki objects. Other writers I've written (e.g., for blogging) return lists of objects that have been grouped (e.g., by date for an archive view). The only requirement is that render and key are able to make sense of each item in the list returned.

Just as with parsers, the writers rely heavily upon the template method pattern to drive the whole process.

Some additional settings are needed as well:

# This enables our custom wiki writer.
WRITERS = ['firmantwiki.WikiWriter']

# The directory (under CONTENT_ROOT) used for storing wiki documents.
WIKI_SUBDIR = 'wiki'

# Load Jinja2 templates from the filesystem.
import jinja2
TEMPLATE_LOADER = jinja2.FileSystemLoader('templates')

Running make shows:

env FIRMANT_OUTPUT_DIR=preview \
    FIRMANT_PERMALINK_ROOT=file://`readlink -f preview` \
    firmant settings
INFO:firmant.application.Firmant:firmantwiki.WikiParser parsing 'index.rst'
INFO:firmant.application.Firmant:firmantwiki.WikiWriter declared 'file:///home/rescriva/projects/firmantwiki/preview/index/'
INFO:firmant.application.Firmant:firmantwiki.WikiWriter rendered 'preview/index/index.html'
xdg-open preview/index.html

Notice how the writer declares a URL for the index document while it writes it to preview/index.html. This logic is powered by the URL routing backend. I'll be elaborating more on this in the next section (as I have promised throughout this post).

URL Routing

The firmant.routing module provides a mapping between a dictionary of attributes and a URL. Those familiar with the lambda calculus or logic-based languages such as Prolog will recognize the behavior to be similar to unification.

Routing revolves around the concept of a path. A path has a set of attributes split between bound, and free attributes. Naturally the bound and free attributes for a path are disjoint while their union is the set of all attributes for the path. If a path matches a set of attributes, it is possible to construct a path from the attributes.

This lends itself to several possibilities:

Single Path Component

An attribute is converted to its string representation, and matches if and only if the attribute names are the same.

Bound Null Path Component

Similar to the SinglePathComponent, but the path is always constructed to be empty.

Static Path Component

The opposite of a bound null path component. This matches when the set of attributes is empty, and always is constructed to the empty string.

Compound Path Component

Several path components joined together. Each component will be constructed using the appropriate set of attributes, and the constructed strings will be joined with '/'. See the documentation for more details. The '/' operator is overloaded to join path components.

The firmant.routing.components package contains several pre-built path components.

If you haven't guessed already, the URLS setting contains a list of components (typically compound components). In our case we just have:

URLS = [c.TYPE('wiki') /c.PATH]

We can see that this declares a URL that has the attributes type=wiki and path. Internally, the firmant.routing.URLMapper object will turn a set of attributes that unifies with this (leaving no free attributes) into a path that is equal to the wiki object's path attribute. Our writer's key method returns just the right set of attributes to unify with these attributes. This is how the writer knows which URL to use for each written object. The URLMapper is able to return both a local filesystem path (relative to the output directory) and a full URL for any set of attributes (with the PERMALINK_ROOT as the base).

This URL mapping system allows for reconfiguration of the output paths and URLS for each set of objects a writer will write. For example, if I wish for the index.rst wiki document to be a special document that doesn't reside at '/index/', but rather at '/', it is as simple as creating a rule with higher priority that only matches this document:

URLS = [c.TYPE('wiki') /r.BoundNullPathComponent('path', 'index')
       ,c.TYPE('wiki') /c.PATH
       ]

The first URL entry will construct to the empty path '/'. Firmant will append the 'index.html' to the local filesystem path so that when an HTTP client requests '/', it will be served '/index.html'. The result is so-called "clean" URLs for all pages.

Creating A Blog

Firmant started as a blogging platform and slowly evolved to support much more. Similar steps to those taken here can be used to create a blog. The Makefile is the same as for a wiki. Firmant conveniently provides a basic configuration suitable for a blogging platform as firmant.settings.

If you're interested in building a blog, consider cloning either the Firmant blog or the CHASM blog. The common configuration requires a bare settings file, and a small template configuration file. I'll be posting more about creating a blog with Firmant in a future post.

Conclusion (AKA Where is this going from here)

Firmant is still a very young platform for development. It's only a year and a half old, but the current iteration (with static files) is less than six months old.

I'm hoping to add polish to both the code and documentation in future releases to make it more easily usable by those who are not inside my head (including myself). Users or developers interested in following Firmant can join the Firmant mailing lists to be alerted to new releases, or participate in the development of new releases.

As usual, you can find 'me' at this domain using SMTP, or 'rescrv' on irc.freenode.net #firmant.

Testing Strategy

Firmant is a highly modular application. Tests are written to a particular interface, and then each implementation of an interface can easily take advantage of a thorough, already written test suite. In general it is only necessary to override the setUp and tearDown methods of test classes.

It takes exactly 100 lines of code to invoke the roughly 1300 lines of test code for the atom backend. Much of this is boilerplate I hope to be able to further reduce (Python is nice for doing this).

Why are these tests written like this? If some day someone wishes to write a CouchDB, MySQL, PostgreSQL or other backend, they can rely upon a test suite to ensure that most corner cases in their code are properly covered so that it can seemlessly be changed with the default Atom interface.

Porting Firmant

Firmant's test suite also makes it relatively easy to maintain several platforms at once without having to develop on all of them. FreeBSD, CentOS, Debian, and Fedora all should work without me having to develop on all these platforms. I need only to run the tests occasionally and verify that all pass.

I've found that CentOS provides the most information about non-portable Python code. CentOS is the only distribution I listed that still is defaulting to Python2.4. Debian is on 2.5 while FreeBSD generally tends to 2.6. From what I've heard, Fedora may go 3.0 sooner than the rest. I'll cross that line when it comes.

:wq