YML - Why a Markup Language?!

Any time a formal language is created for computing, a compromise has to be found: whether the language is perfect for the computer but terrible for the human or vice versa. XML is very good for the computer ;-)

Using XML for the reasons mentioned above, but for programming? "Why a Markup Language?!" was what I was shouting some time playing around with a code generator in XSLT. That gave the idea.

Motivation

Many people don't write XSLT directly, because they don't like writing programs in angle brackets. The result usually is, that people are writing Java programs, and are processing XML from Java, or are using XSLT features from Java programs.

But that is very inefficient - there are small and quick XSLT processors, and there are no advantages at all to implement that in Java or C++ usually.

So I wanted to have something like a Java or C like language, which can be easily translated into XSLT. Then a common XSLT processor can process the program, and XML can be processed very quickly.

I started this, because I saw, that code generation for Automated Software Engineering can be implemented very easily in XSLT - but writing XSLT is annoying.

You can find some sample code here how to do that with YML and YSLT.

How it works: Replacing angle brackets with some Python

Just writing down what I wanted to have instead of XML for a sample:

<list name="List of goods">
    <head>
        <columTitle>
            Goods
        </columnTitle>
        <columnTitle>
            Price
        </columnTitle>
    </head>
    <row>
        <value>
            Beer
        </value>
        <value>
            20
        </value>
    </row>
    <row>
        <value>
            Wine
        </value>
        <value>
            30
        </value>
    </row>
</list>

Something like that should be more easy, say, like this:

list("List of goods") {
    head {
        title "Goods"
        title "Price"
    }
    row {
        value "Beer"
        value 20
    }
    row {
        value "Wine"
        value 30
    }
}

Y Languages

The latter is what I call an Y language - a language specified in YML. How could this be achieved? Well, what's to do? To have the required information, how to build XML from the script above, we need:

the information, that "list of goods" is an attribute named "name", while "Goods" is the text value of a tag
"20" and "30" have to be converted to text, and may not be different from "Beer" and "Wine" in typing
"title" shout be written out as "columnTitle"

How to do that? Let's invent a simple definition language for that information:

decl list("name") alias list
decl head() alias head
decl row() alias row
decl title("=>body") alias columnTitle
decl value("=>body") alias value

These four lines have all information we need. If you declare a parameter "name", then it's clear, that what's going in there as value will be attached to an attribute "name".

The alias keyword will connect the declared function with an XML tag name.

The notation =>body means, that what's given here as a parameter, will be put into the value of a tag directly. Because <=body is default, you may drop this part. The alias keyword is optional, if tag name and function name match. The following declarations are equivalent to the ones above:

decl list("name")
decl head
decl row
decl title alias columnTitle
decl value

The difference between =>body and <=body is, that <=body does not move a given attribute into the body, if the attribute text contains a '=' sign. The reason for doing so is, that you can simply add plain attributes to "empty tags" using the syntax i.e.: decl div alias div in YHTML and then calling it with manually set attributes, i.e. div 'style="color=red;"'If you don't want this behaviour, you should declare using =>body.

Here you can download the complete list sample.

Hint: Every declared function generates a Python function named yml_ plus the function name. decl head() alias head for example defines a Python function yml_head().

Some Features

Function Calls

It does not matter, if you're calling your function using parantheses or without, or if you want to add a semicolon to the end of line.

So these statements are all equal:

foo "hello, world"
foo("hello, world")
foo("hello, world");

Hint: What comes behind the function name (or in the parantheses) is just a comma separated list of Python expressions. That means, text(' ' * 10) will give you a string of ten spaces as first parameter of function text.

Sub Trees

YML sub trees are opened and closed with braces:

foo {
    bar {
        something();
    }
}

attr(): Enforcing a manual set attribute

If you have defined an attribute, and want to override it setting another attribute, you can use the attr() class for doing so. Here a sample:

// declare an attribute
decl when("test")
// using it
when ("type=1") {
// ...
// overriding it
when (attr("myAttr='something'"), "type=1")
// of course, you just can write this, too:
when "type=1", "myAttr='something'"

The latter two define myAttr="something" as XML attribute additionally to "test='type=1'".

x(): Disabling XML Escaping for attribute values

YML automatically escapes text of attribute values.

If you don't want this, use x(value) as attribute value. This prevents YML from escaping your text.

// text escaped
text("> hello, world <")

// text not escaped
text(x('this is "real"'))

xattr(): Enforcing a manual set attribute and disabling XML Escaping for it

If you want to have both the functionality of x() and attr(), use the combined xattr().

body(): Bringing text into the body of a tag

To make a "one-liner", you can use body(value) as a YML function parameter. Here is a sample:

decl staff
decl director("name")

staff {
    director "Herbert Meier" {
        > Head of Development
    }
    director "Alfred E. Newman", body("Head of Sales")
}

This results in:

<?xml version="1.0" encoding="iso-8859-1"?>
<staff>
  <director name="Herbert Meier">Head of Development</director>
  <director name="Alfred E. Newman">Head of Sales</director>
</staff>

xbody(): Bringing text into the body of a tag and disabling XML Escaping for it

If you alread have XML text, you can switch off XML escaping by using xbody(value) instead of body(value).

Quoting Operators

Three different quoting operators implement different functionality:

Quote >

The '>' operator quotes into text nodes, doing XML escaping of text. An example:

> this text will be put into a text node and these angle brackets <> will be quoted

Line Quote |

The '|' operator does the same as the '>' operator, adding a new line character to the text node.

Additionally, it can be used to implement an indention system, see YSLT below.

Then it's used together with additional '>' symbols showing the grade of indention:

| not indented
|> single indent
|>> double indent
(...)

Block Line Quote ||

The '||' operator opens and closes a block of lines, which then are handled like if each of them would be preceeded with a Line Operator '|'.

Sample:

||
this is code being quoted through
this is the second line
||

is equivalent to:

| this is code being quoted through
| this is the second line

Inserting Commands

Just like with a Unix shell, you can insert statements into text by using back quotes:

| Click `a("http://fdik.org/yml/", "this link")`, please!

For abbreviation purposes, you can define a short cut insert (see YSLT below).

Here you can have a default tag for inserts using the « » operators. In YSLT you can use them for xsl:value-of tags, for example.

The following two lines are equivalent in YSLT:

| Gimme some «$value» hereby.
| Gimme some `value("$value")` hereby.

They are, because there is a define special_command_name as value command in the YSLT definition file.

Hint: If you're running into troubles because of character set problems with the '»' symbols, then use these two lines on top of your YSLT script:

define special_command_left as «
define special_command_right as »

My samples all use iso_8859-15 charset, if you're using something else you may need that.

Being in a Block Line Quote '||', you additionally can use the Line Command operator (two backquotes, '``');

This is very interesting to have in YSLT, for example:

||
some code
``apply "myTemplate"
some other code
||

Quote Through ]

The Apple ][ prompt operator just quotes through directly into XML what it gets.

If the first non whitespace character of a line is a '<', then quote through is applied automatically.

This is the preferred way to output XML tags directly in YML:

<output me="directly" />

] <!--
]     add some comment, which then appears in XML
] -->

Importing and Including

Because YML first is translated into Python (what I call Python YML - PYML), you can use simple Python modules together with your YML code (see next section, too):

!from module import *
!import module2

You can include a second YML script file into an existing YML script file at any place using:

include something.yml

File globbing using * and ? placeholders is allowed to include more than one file at a time:

include part*.yml

File globbing also can be used reverted; that means, the files are included in reverse order:

include reverse part*.yml

If there are the files part1.yml, part2.yml and part3.yml, part3.yml is included first now.

For including files, there are two keywords: include and section. They're equivalent.

if you have text in a file, which you want to include between '||' operators, instead of writing:

||
include some.txt
||

you also could write:

include text some.txt

Escaping into Python directly - the Escape Operator !

As an alternative, you can insert a Python command into the generated PYML script at any place by using the ! operator:

!print "hello, world"

Comments //

Comments are written like in Java or C++:

// this is a comment

something() { // this is a comment after a tag

After quoting operators, comments are not possible. Instead, they're quoted through:

// the following line you'll find in the output document
> this text is being output // and this too

Hint: Because comments are removed with the lexical analyzer already, you should not use // in parameters when using functions without parantheses. This will cause an error, because everthing behind // will be treated as a comment:

callMe "with // in text parameter"

There is no such problem, when you're using parantheses. This line will work:

callMe("with // in text parameter")

Macros

Macros can be defined using he define keyword, i.e.:

define ABC as "ABCDEFGHIFJKLMNOPQRSTUVWXYZ"
define abc as "abcdefghifjklmnopqrstuvwxyz"

Some macros have special meanings:

quote_text

quote_text is an PYML (Python) expression, how to insert plain text. %n is a placeholder for the inserted text.

YSLT defines it as following:

define quote_text as yml_text("""%n""")

quote_indention

quote_indention is an YML function call, which is called by the | operator for indenting. There is the place holder %n, which is replaced by the indention level.

YSLT defines quote_indention as following:

define quote_indention as indent("%n")

special_command_left and special_command_right

special_command_left and special_command_right are operators, which can be used inside quoted text for special functions. By default they're defined as following:

define special_command_left as «
define special_command_right as »

They're used by YSLT for value() function calls. By YHTML they're used for <strong> text.

special_command_name

special_command_name is the YML function, which is called by using the operators, which were defined with special_command_left and special_command_right.

generate_comments and comment_mask

If you define generate_comments as True then a comment containing the include file name (or the empty string for the main script) and the line number are added to each line, which is output using the pipe operator '|'.

comment_mask defines a pattern for generating the output. The default pattern is /* %f:%n */. %f is replaced by the name of the current include file (or an empty string if we're in the main script), and %n is replaced by the line number of the pipe operator in the script, which generated the line.

This feature is intended for debugging purposes.

YSLT - XSLT as Y language

Basic idea

XSLT already is a functional programming language. But it's ugly. So one of the first approaches to use YML was to define YSLT. YSLT is just a Y language for XSLT.

Some YSLT samples

Hello, world

The hello world looks plain and simple:

include yslt.yml

textstylesheet {
    template "/" {
        | hello, world
    }
}

You can run that with:

vb@ostrich:~ % ysltproc hello.ysl 
hello, world
vb@ostrich:~ %

Java code generator in YHTML - a sketch

This works very well - see this sample program, which compiles Java interfaces out of XMI files.

You can compile it using:

vb@www:~/fdik.org/yml $ ysltproc xmi2JavaInterface.ysl sample.xmi > sample.java 
vb@www:~/fdik.org/yml $

Generating HTML

This sample generates a table view in HTML out of the list sample of the first chapter. I'm using YHTML for it, the same Y language I'm writing this documentation in:

include yhtml.yml
include yslt.yml

stylesheet {
    template ("list") {
        _html {
            head {
                title {
                    > «@name»
                }
                style ("text/css") {
                    comment {
                        > th { border-style:solid; border-width:1; }
                        > td { border-style:dotted; border-width:1; }
                    }
                }
            }
            body {
                table {
                    apply "head"
                    apply "row"
                }
            }
        }
    }

    template ("head") {
        tr {
            apply "columnTitle"
        }
    }

    template ("columnTitle") {
        th {
            > «.»
        }
    }

    template ("row") {
        tr {
            apply "value"
        }
    }

    template ("value") {
        td {
            > «.»
        }
    }
}

This YSLT program you can download here. The result looks like this:

CYY sample

I added a sample for CYY, a language, which shows how to use YML/YSLT as a macro preprocessor to C++.

section cyy_templates

template "type", "enum" {
    | T«@name»`if("position()=1", " = 0")`,
}

section cyy_code
#include "types.hh"

enum Types {
`` apply "types/type", "enum"
    NUMBEROF_TYPES
};

class TileCache { };

template <Types SELECT, class T>
class TileCacheAdapter : public TileCache {
public:
    enum { adapter_type = SELECT };
    T obj;
};

class CacheController {
public:
    CacheController() {
    `` foreach "types/type" {
    ``      |>> adapter[T«@name»] = new TileCacheAdapter< T«@name», «@name» >;
    `` }
    }
    ~CacheController() {
        for (int i=0; i<NUMBEROF_TYPES; i++)
            delete adapter[i];
    }
private:
    TileCache* adapter[NUMBEROF_TYPES]; 
};

section cyy_end

You can use this together with this data file as input:

decl types
decl type("name") 

types {
    type "Street"
    type "Edge"
    type "Vertex"
}

So for getting C++ code out of this, use:

vb@www:~/fdik.org/yml $ ysltproc sample.cyy types.yml > sample.cc
vb@www:~/fdik.org/yml $

The content of the CYY definition files you can find here: cyy_templates, cyy_code and cyy_end.

if you don't define templates, you can use cyy_begin instead of writing two lines with cyy_templates and cyy_code.

As a result of the sample above, this C++ program is being generated.

YC - a C like Y language

YC is just for fun ;-) I wrote a faculty sample and Towers of Hanoi in it to demonstrate, how to compute while compile time.

For easier translating of YC code into object code, you can use this script.

YBlog - another sample

With YBlog you have a sample how to hack a little Blog engine using YML.

Features

For features, just read the YSLT definition and have a look on the samples above.

Tool Chain

What it is

This documentation was written in YHTML - see the source and the Makefile as well as the YML translator script.

It was mostly generated out of the XML Schema of XHTML using the Default Compiler using YSLT and an YSLT processor. As XML tool chain, I'm using xsltproc and XMLStarlet.

YSLT is using the YPath python module as an addition to XPath.