How Dust Works

Posted 2013-12-09 10:18 PM GMT

How Dust Works

When I started working at LinkedIn two years ago, I had never heard of Dust. Since then, I have had the opportunity to work on the LinkedIn team that develops and maintains the LinkedIn fork of Dust. When I started working on Dust, I decided to take some time digging through the code, to try to figure out how all the pieces work together. At the time, I was taking a Udacity course on programming languages, which proved to be very useful in my study of Dust. This is what I learned.

Dust Grammar

The Dust grammar is simple relative to many programming languages, which makes sense considering Dust is a (mostly) logic-less templating language. The grammar defines everything that is significant to Dust, including sections, bodies, references, filters, and helpers. Let's take a look at the grammar that defines a reference:

/-------------------------------------------------------------------------------------------------------------------------------------
   reference is defined as matching a opening brace followed by an identifier plus one or more filters and a closing brace
---------------------------------------------------------------------------------------------------------------------------------------/
reference "reference"
  = ld n:identifier f:filters rd
  { return ["reference", n, f].concat([['line', line], ['col', column]]) }
In other places in the grammar ld (left delimiter) is defined as {, rd is defined as }, identifier is defined as being a key or a path, and key is defined as a series of characters starting with a-z, A-Z, _, or $, followed by zero or more of 0-9, a-z, A-Z, _, $, or - (and a path is a series of dot separated keys). We could follow the same procedure for every other piece of the Dust grammar.

dust.parse

Dust uses uses PEGjs to turn the grammar into a parser. The parser takes any valid Dust template and creates an AST. Let's look at an example. This Dust template:

Hello {name}! I think you're a cool guy.
creates this AST (by running dust.parse("Hello {name}! I think you're a cool guy.");)
[
    "body",
    [
        "buffer",
        "Hello "
    ],
    [
        "reference",
        [
            "key",
            "name"
        ],
        [
            "filters"
        ]
    ],
    [
        "buffer",
        "! I think you're a cool guy."
    ]
]
The highest level of every Dust template is a body, and that's what we see on the 2nd line of the AST. Inside the body we see a buffer, a reference, and another buffer. From the template we can see that a buffer is plain text, and a reference is { some text }. With an AST, each piece of the source code is clearly identified, ready for a compiler to transform into executable code.

dust.compile

Dust templates are compiled to JavaScript. The compiler takes a Dust template (and a template name), runs dust.parse to create an AST, optimizes the AST for compilation, then "walks" through each part of the AST to create the compiled JavaScript. To compile a Dust template:

dust.compile("Hello {name}! I think you're a cool guy.", "home*")

First the compiler creates the first part of an IIFE (immediately invoked function expression), and creates a call to register the template with the given name:

The compiled template thus far:

(function(){
    dust.register("home*",

Then Dust compiles the template starting with the first node (body) of the AST. Each body is a function within the enclosing IIFE, and each of these functions is given a name of the form body_n starting with body_0.

When a buffer is encountered in the AST, it is turned into something like .write("Hello "). When a reference is encountered, it becomes something like .reference(ctx.get("name"),ctx,"h") (note the "h" filter is applied by default to all references). How .write and .reference work will be discussed in the dust.render section below.

The fully compiled template:

(function(){
    dust.register("home*",body_0);
    function body_0(chk,ctx){
        return chk.write("Hello ").reference(ctx._get(false, ["name"]),ctx,"h").write("! I think you're a cool guy.");
    }
    return body_0;
})();

If you include the browser version of Dust and the above JavaScript on any web page, the template will register itself and be ready to be rendered.

dust.render

When you include a compiled Dust template on a web page, the IIFE is immediately executed and template is registered to the Dust cache with the call to dust.register. Now you can call dust.render to render your template. Here's an example:

HTML before calling render:

<div id="container"></div>
JavaScript used to render:
var container = document.getElementById('container');
dust.render('home*', {name: 'Coach Z'}, function(err, out) {
    if (out) {
        container.innerHTML = out;
    }
});
HTML after calling render:
<div id="container">Hello Coach Z! I think you're a cool guy.</div>

How dust.render works

When you call dust.render, Dust looks for a template with the given name in its cache. If such a template is found, the function related to the template is executed. When dust.register("home*", body_0); is executed, the body_0 function is tied to the home* template, so body_0 is executed. Here's that function again, with slightly nicer formatting.

function body_0(chk, ctx){
    return chk.write("Hello ")
           .reference(ctx._get(false, ["name"]), ctx, "h")
           .write("! I think you're a cool guy.");
}
The function accepts two arguments: chk (pronounced "chunk"), and ctx (pronounced "context").

chk is an object that holds data about the template currently being rendered (especially the rendered output), as well as a number of methods used to render the template (such as write, reference, and section).

ctx is an object containing the data used to render the template. It is very similar to the data passed in to dust.render (in this exampled, {name: 'Coach Z'}), but it contains some extra metadata and is restructured to make the render process a bit easier. The example data ends up looking something like this:

{
    "stack": {
        "isObject": true,
        "head": {
            "name": "Coach Z"
        }
    },
    "global": {
        "__templates__": [
            "home*"
        ]
    }
}

Looking back at body_0, we can see how our example template is executed by Dust. chk.write("Hello ") saves "Hello " to chk to be output when the template finishes execution. .reference(ctx._get(false, ["name"]), ctx, "h") first attempts to find the value of key name in the ctx, then applies filters to that value, then calls chk.write on the filtered value. When the execution is finished, Dust returns output that has been saved to chk. It is worth noting, however, that dust.render does return anything. Instead it takes a callback function which accepts the output as its second argument. This is because Dust is an asynchronous templating language.

There are a number of other Dust features that make Dust a very valuable tool for building a web application. If you have questions about this post or Dust, you can send them to me or you can ask them on StackOverflow.