Creating DSLs with Frag: Executing the DSL

As Frag is a dynamic language, we can use the ordinary option for executing a DSL in a dynamic language: Using the interpreter of Frag to execute the language. In addition, as Frag is embedded in Java, we also have the option to use Java classes to implement the DSL execution semantics (remember: Frag imports Java classes automatically via reflection).

In addition to these obvious options for defining the DSL's execution semantics we can also support other options in Frag. Frag additionally supports transformation templates, as they can be found in model-driven language workbenches. Using these transformation templates any code can be generated.

Finally, there is a third option supported by Frag: As Frag combines a dynamic language with a DSL language model, the classes of the DSL language model can also be transformed using Frag's dynamic language features, such as reclassing Frag objects, changing superclasses, renaming objects/classes, and so on.

In Frag, these options can be selected at any stage at which designing execution semantics for the DSL is necessary. They can also be combined. For example, a common combination when using transformation templates is to also use some extensions of the language model in Frag, e.g., using mixins, to externalize interpretation tasks from the transformation templates. In this section, we want to discuss these two options.

Extending a Language Model via Mixins

If we want to add interpretations to a language model, we could simply define methods on the language model classes. However, that would mean to mix the concerns of language model definition and interpretation of the language model. A more elegant way to add behavior to a language model are hence mixins that extend the language model classes.

As an example, consider the following model of a book containing documents.

FMF::Class create Book

FMF::Class create Document -attributes {
    text String
    title String
    description String
    author String
    version String
}

FMF::Composition create Book-Document -ends {
    {Book -roleName book -multiplicity 0..1 -navigable true -aggregatingEnd true}
    {Document -roleName document -multiplicity * -navigable true}
}

Consider we want to generate an HTML documentation using the information in instances of this model. A small instance model is for example the following:

Book create FragDoc
Document create Basics -book FragDoc -author "Uwe Zdun" -title "Basics" -text "..." -description "Frag Documentation: Basics" -version 0.8
Document create Objects -book FragDoc -author "Uwe Zdun" -title "Frag Object System" -text "..." -description "Frag Documentation: Frag Object System" -version 0.8

To generate HTML using mixins, we need to define mixins for transforming such information into HTML code. To do so, we first define a mixin for the book class that offers a method to transform each document into HTML (using a method transformToHTML we will later define on a mixin for documents) and write the HTML code to a file.

Object create BookHTMLMixin
BookHTMLMixin method transformToHTML {genDir} {
    puts "... Make gen dir"
    set f [File create -name $genDir]
    if {![$f exists]} {
        $f mkdir
    }
    puts "... Generating Documents"
    foreach doc [self document] {
        puts "... ... $doc"        
        set htmlCode [$doc transformToHTML]
        set f [File create -name "$genDir/${doc}.html"]
        if {[$f exists]} {
            $f delete
        }
        puts $htmlCode
        $f write $htmlCode
    }
}
Book mixins BookHTMLMixin

Next, we define the a mixin for Document. In its transformToHTML method, it first checks for the mandatory arguments that should not be empty using the method mandatoryNotEmptyArguments. If this method either does not find all mandatory arguments or one of them is empty, it raises an exception. If no exception is raised, transformToHTML proceeds by creating an HTML representation of the document, by concatenating it as a string.

Object create DocumentHTMLMixin
DocumentHTMLMixin method transformText {} {
    # only a simple placeholder, doing nothing to the text
    return [self text]
}
DocumentHTMLMixin method mandatoryNotEmptyArguments {argList} {
    foreach arg $argList {
        if {![self varExists $arg] || [self $arg] == ""} {
            throw [Exception create -msg "Mandatory argument '$arg' missing or empty for document: '[self]'"]
        }
    }
}
DocumentHTMLMixin method transformToHTML {} {
    self mandatoryNotEmptyArguments {title description author version}

    string build `
        <html>
            <head>
                <title>`[self title]`</title>
                <meta NAME="description" CONTENT="`[self description]`">
            </head>
            <body>
                <h1>`[self title]`</h1>
                <h3>`Author: [self author]`</h3>
                <h3>`Version: [self version]`</h3>
                `[self transformText]`
            </body>
        </html>`
}
Document mixins DocumentHTMLMixin

The final thing we need to do is to call the book's transformToHTML method:

FragDoc transformToHTML ./gen

This approach works well for defining behaviors for DSLs that need to be interpreted at runtime and for simple generation tasks, as the ones explained above. It works not so well for complex and tangled code transformation or generation tasks, as it can be pretty hard to understand the structure of the code generated from a number of interdependent mixins. In such cases, it is advisable to use the transformation templates explained in the next section.

Transformation Templates in Frag

Frag provides a simple template engine that can make use of a few special purpose features, but mainly uses the Frag language itself to define the transformations. Hence, it blends in well with other approaches like the mixins based extensions explained in the previous section. The main difference is that the templates directly reflect the code to be generated, and hence make it easier to understand the structure of the generated output.

For example, the template below can be used for generating HTML code from an FMF model. Likewise, other code in other languages or other models can be generated.

<table>
  <~ self applyForeach entry [$collection getEntries] {
     <tr valign="top">
       <td align="right">
       <a name="<~ $entry getID ~>"> <~ $entry getValue ~> </a> 
     </td>
       <td>
         <~ $entry getLink ~>
       </td>
     </tr>
  } ~>
</table>

In templates we use the delimiters <~ and ~> to denote Frag code embedded in the generated code fragment. Using self we refer to the template engine, which offers a number of methods for traversing and interpreting language models in templates, such as applyForeach in the example above. The template engine offers the following methods for traversing and interpreting language models in templates:

<templateEngine> applyForeach <element> <list> <code>: Applies the template code for each element, given in the list. Enables you to traverse a number of elements, e.g., in a an association and perform the same template code for each of them. The output is the concatenated outputs of all template runs triggered by the applyForeach statement.
<templateEngine> applyIf <condition> <code> ?else <code>?: Applies the template code, if condition is true. Otherwise the template code in the option else-part, if present, is executed. The output is the result of the template code evaluation.
<templateEngine> applyByType <element> <typeMappingList>: Based on the type of the element, the first matching type in typeMappingList is chosen. The type mapping list consists pairs of elements of the form <type> <code>. applyByType goes through these elements, one by one, and if element is of a given type, the corresponding template code is evaluated, and its result is returned. Superclass also cause the element to match. Hence, Object can be chosen to define a default type implementation.

When intiating or configuring the template we can provide initialization code. For example, in the example above the variables collection and entry have to be set prior to template execution. The template engine offers the following methods for configuring the engine prior to generation:

<templateEngine> setTemplateDir <templateDir>: Set the directory in which the template engine can find the templates. Defaults to the current directory.
<templateEngine> getTemplateDir: Retrieve the directory in which the template engine looks for the templates.
<templateEngine> setGenDir <genDir>: Set the directory in which the template engine should place the output files of the generation.
<templateEngine> getGenDir: Returns the directory in which the template engine should place the output files of the generation.
<templateEngine> setConfigCode <configCode>: Provide configuration code that the template engine executes before every execution of a template. Is usually used to pass variables that are used in the template to the template instance.
<templateEngine> getConfigCode: Returns the configuration code currently set in the engine.

The following methods are usually used to executes templates:

<templateEngine> transformFile <inputFile> <outputFile>: Reads the input file, and applies the template in the input file to the data configured for the template engine. Then it writes the result to the output file. The input file is read from the templates directory configured in the engine, and the output is written to the gen directory.
<templateEngine> apply <template>: Applies the given template (from the memory, not from a file) to the data configured for the template engine.
<templateEngine> applyFile <inputFile>: Same as apply, but first reads the template from the file inputFile.

Example

To illustrate the use of template, let us consider the same example, as discussed in the previous section. We use the same language model and instance code. In order to run the templates, we first need to import the template engine via the package mdd.Templates. Next, we instantiate the template engine and configure the template directory and the gen directory for the generated output. Finally, we use a foreach-loop to walk through the documents. Here we first set the variable doc to "$document" in the setConfigCode method, which is executed prior to the template evaluation; hence, in the template we can use the variable doc to refer to the current document. Finally, we use transformFile to transform the template stored in the file Document.ft to ${document}.html.

import mdd.Templates

puts "... Generating HTML"
Templates::TemplateEngine create templateEngine \
    -setTemplateDir "./bookGenTemplates/" \
    -setGenDir "./gen/"

foreach document [FragDoc document] {
    puts "... ... $document"
    templateEngine setConfigCode "set doc $document"
    templateEngine transformFile Document.ft ${document}.html
}

In the file Document.ft, the template is defined. It has the following structure. As you can see, it basically looks like the html text that should get generated and calls methods for the attributes where appropriate.

<html>
    <head>
          <title> <~ $doc title ~></title>
           <meta NAME="description" CONTENT="<~ $doc description ~>">
    </head>
    <body>
        <h1><~ $doc title ~></h1>
        <h3>Author: <~ $doc author ~></h3>
        <h3>Version: <~ $doc version ~></h3>
        <~ $doc text ~>
    </body>
</html>

This template is pretty simple. In many cases we need more complex operations as well. This can be done using the mixin approach we have seen before. For example, in the previous example, we have checked for mandatory and not empty arguments. Such extensions can be placed in appropriate mixins that are called from the template:

Object create DocumentHTMLMixin
DocumentHTMLMixin method mandatoryNotEmptyArguments {argList} {
    foreach arg $argList {
        if {![self varExists $arg] || [self $arg] == ""} {
            throw [Exception create -msg "Mandatory argument '$arg' missing or empty for document: '[self]'"]
        }
    }
    return ""
}
Document mixins DocumentHTMLMixin

Now we can add a call to this extension in the template. It can be placed anywhere, as it has no result (empty string). If it would have a result, it would be appended to the template result.

<html>
    <~ $doc mandatoryNotEmptyArguments {title description author version} ~>
    <head>
          <title> <~ $doc title ~></title>
           <meta NAME="description" CONTENT="<~ $doc description ~>">
    </head>
    <body>
        <h1><~ $doc title ~></h1>
        <h3>Author: <~ $doc author ~></h3>
        <h3>Version: <~ $doc version ~></h3>
        <~ $doc text ~>
    </body>
</html>

Please note that another option than placing extensions in the template is to place them in the configuration code. As the mandatoryNotEmptyArguments method has no results this might be more appropriate in this case:

foreach document [FragDoc document] {
    puts "... ... $document"
    $document mandatoryNotEmptyArguments {title description author version}
    templateEngine setConfigCode "set doc $document"
    templateEngine transformFile Document.ft ${document}.html
}

Now we have basically implemented all features of the previous example. To illustrate the further features of templates, lets consider instead of just containing text, documents can contain all kinds of elements, and there is a association doc-elt which lets one document contain * elements (accessed using the element role name):

FMF::Class create DocElement

FMF::Class create TOC -superclasses DocElement 

FMF::Class create DocElementContainer -superclasses DocElement

FMF::Composition create DocElementContainer-DocElement -ends {
    {DocElementContainer -roleName container -multiplicity 0..1 -navigable true -aggregatingEnd true}
    {DocElement -roleName element -multiplicity * -navigable true}
}

FMF::Association create doc-elt -ends {
    {Document -roleName document -multiplicity 0..1 -navigable true}
    {DocElement -roleName element -multiplicity * -navigable true}
}

FMF::Class create Link -superclasses DocElement -attributes {
    target String
}
FMF::Class create DocumentLink -superclasses Link 
FMF::Class create SectionLink -superclasses DocumentLink -attributes {
    section String
}
FMF::Class create URLLink -superclasses Link -attributes {
    text String
}


FMF::Class create SectionContainer -superclasses DocElementContainer

FMF::Class create Section -superclasses SectionContainer
FMF::Class create SubSection -superclasses SectionContainer
FMF::Class create SubSubSection -superclasses SectionContainer
FMF::Class create Paragraph -superclasses SectionContainer

FMF::Class create List -superclasses DocElement
FMF::Class create ListItem -superclasses DocElementContainer
FMF::Composition create List-ListItem -ends {
    {List -roleName list -multiplicity 0..1 -navigable true -aggregatingEnd true}
    {ListItem -roleName element -multiplicity * -navigable true}
}

FMF::Class create Em -superclasses DocElementContainer
FMF::Class create Bf -superclasses DocElementContainer
FMF::Class create Tt -superclasses DocElementContainer


FMF::Class create Text -superclasses DocElement -attributes {
    text String
}

FMF::Class create Code -superclasses Text

FMF::Class create Separator -superclasses DocElement

FMF::Class create Figure -superclasses DocElement -attributes {
    fileName String
}

Instead of just printing the text, we want to step-by-step go through all document elements and append all of them to the output. This can be done as follows using applyForeach:

<~ self applyForeach elt [$doc element] {
    ...
} ~>

At ... we need to insert the code that handles the individual elements. In our case, we need to handle all elements differently - guided by the type of the element. We could do this using applyIf as follows:

<~ self applyForeach elt [$doc element] {
    <~ self applyIf {[$elt isType Separator]} {
        <p></p>
    } ~>
} ~>

But with the many types of our meta-model, we would need many applyIf's. Hence, applyByType is useful here:

<~ self applyForeach elt [$doc element] {
    <~ self applyByType $elt {
        Separator {
            <p></p>
        }
        Code {<pre CLASS="code"><~ $elt convertSpecialCharactersToHTML [$elt text] ~></pre>}
        Text {<~
            $elt convertSpecialCharactersToHTML [$elt text]
        ~>}
        Section {
            <h2><~ $elt convertContainedSectionEltsToHTML [self] ~></h2>
        }
        SubSection {
            <center><h3><~ $elt convertContainedSectionEltsToHTML [self] ~></h3></center>
        }
        SubSubSection {
            <h3><~ $elt convertContainedSectionEltsToHTML [self] ~></h3>
        }
        Paragraph {
            <h4><~ $elt convertContainedSectionEltsToHTML [self] ~></h4>
        }
        Tt {<tt><~ $elt convertContainedEltsToHTML [self] ~></tt>}
        Em {<em><~ $elt convertContainedEltsToHTML [self] ~></em>}
        Bf {<b><~ $elt convertContainedEltsToHTML [self] ~></b>}
        List {
            <ul><~ $elt convertListItemsToHTML [self] ~></ul>    
        }
        ListItem {
            <li><~ $elt convertContainedEltsToHTML [self] ~></li>    
        }
        DocumentLink {<~
            set doc [$elt target]
            if {[interp isObject $doc] && [$doc isType Document]} {
                $elt makeLink [self] $doc
            } else {
                puts "Warning: undefined document link: the target '$doc' is not existing"
            }
        ~>}
        URLLink { <a href="<~ $elt get target~>"><~ $elt get text~></a>}
        TOC {
            <~ $elt getTocInHTML [self] ~>
        }
        Figure {
            <p></p>
            <img src="<~ $elt fileName ~>" alt="<~ $elt fileName ~>" />
            <p></p>
        }
        Object {
            <~ throw [Exception create -msg "Don't know document element type: [$elt getClass]"] ~>
        }
    } ~>
} ~>

Please note that we use all kinds of code invocations here. Some use the meta-model elements directly, others call methods on mixins (like the convertContainedSectionEltsToHTML) methods. Object is used to implement a default behavior at the end of the type mapping list.

To modularize the templates, we can place parts of templates in other template files. For example, we could externalize the applyByType block, as it can be used from different other templates. Let's assume that we have stored it in a file DocumentElements.ft. Then we can rewrite the applyForeach loop as follows:

<~ self applyForeach elt [$doc element] {<~
        self applyFile DocumentElements.ft 
    ~>} 
~>

index | contents | previous