LALR Parser Generator

Visual Prolog 7.5 Commercial Edition (build 7501+) contains an LALR(1) parser generator in the examples (in the directory vipLalrGen). The example directory also contains a program exprEval, which uses a generated parser to parse arithmetical expressions.

The parser generator itself also uses such parser to parse grammar files, so it can be seen as another example.

The directory $(ProDir)/vip/vipSyntax contains a grammar file vipSyntax.vipgrm defining the Visual Prolog syntax. The directory also contains the corresponding generated files, and all other files necessary to parse Visual Prolog programs. But that is not the topic of this article, it is only mentioned for reference.

The theory of parsers is huge and cannot be covered here. An LALR(1) parser parses from Left to right, and produces an inverse Rightmost derivation using 1 Look-Ahead symbol. See for example wikipedia:LALR parser for more information.

Parsing

The purpose of a parser is to validate that an input string fulfills a certain grammar specification, and:

• if the grammar is fulfilled: construct a corresponding parse tree.
• if the grammar is not fulfilled: output "syntax error" messages describing the problems.

The overall parsing process consist of splitting the input into lexical elements known as <vpgrmgrmgrm>terminals</vpgrmgrmgrm> and parsing these according to the grammar specification.

The lexing and the parsing is performed intermixed in a single start-to-end scan of the text (without backtracking).

Parser structure

The parser of the exprEval demo can be used as illustration of the parser components and how the overall parsing works.

The grammar file and the parser components are in the sub directory/package <vpgrmgrmgrm>expressionGrm</vpgrmgrmgrm>.

• <vpgrmgrmgrm>expcessionGrm.vipgrm</vpgrmgrmgrm> the grammar specification.
• <vpgrmgrmgrm>expcessionGrm.i</vpgrmgrmgrm>, <vpgrmgrmgrm>expcessionGrm.cl</vpgrmgrmgrm> and <vpgrmgrmgrm>expcessionGrm.pro</vpgrmgrmgrm> the parser.
• <vpgrmgrmgrm>expressionLexer</vpgrmgrmgrm> contains the lexer.
• <vpgrmgrmgrm>expressionSem</vpgrmgrmgrm> contains a support class for building the resulting parse tree.

<vpgrmgrmgrm>expcessionGrm.i</vpgrmgrmgrm>, <vpgrmgrmgrm>expcessionGrm.cl</vpgrmgrmgrm> and <vpgrmgrmgrm>expcessionGrm.pro</vpgrmgrmgrm> are generated by the <vpgrmgrmgrm>vipLalrGen</vpgrmgrmgrm> program from the grammar specification <vpgrmgrmgrm>expcessionGrm.vipgrm</vpgrmgrmgrm>.

The directory sub directory/package <vpgrmgrmgrm>expressionGrmSem</vpgrmgrmgrm> contains support predicates for the semantic actions in the grammar.

The directory sub directory/package <vpgrmgrmgrm>expressionLexer</vpgrmgrmgrm> contains the lexical analyzer. It is based on the class <vpgrmgrmgrm>pfc\syntax\lexer_string</vpgrmgrmgrm>, which makes it very easy to implement lexers uses Visual Prolog number, string and comment syntax. Basically the programmer only needs to define things like keywords and operators.

vipLalrGen

The parser generator itself is in the <vpgrmgrmgrm>vipLalrGen</vpgrmgrmgrm> sub directory (i.e. in <examples root>\vipLalrGen\vipLalrGen, and it should be built before it can be used.

The vipLalrGen program will read grammar files and generate LALR(1) parsers as Visual Prolog source code.

vipLalrGen.exe - LALR parser generator for Visual Prolog
 
Usage:
 
    vipLalrGen.exe [options] <grammar files>
 
Options:
 
    @<File>     Read options from <File>
    -help       Displays the help message
    -out <OutDir>       Generate files in <OutDir> (default: the directory containing the grammar file)
    -details    Add details to the log file
    -nodetails  Don't add details to the log file (default)

It is recommended to have grammars in files with extension vipgrm; the IDE will token color files with that extension.

Example To run vipLalrGen on the grammar file in the exprEval example from a command console:

>cd <example path>\vipLalrGen
>vipLalrGen\Exe\vipLalrGen.exe exprEval\expressionGrm\expressionGrm.vipgrm
OK: exprEval\expressionGrm\expressionGrm.vipgrm

When running vipLalrGen on a grammar file it always produces the file:

• <vpgrmgrmgrm>log/<grammar>.log</vpgrmgrmgrm> containing detailed information about the grammar
• If successful: <vpgrmgrmgrm><grammar>.i</vpgrmgrmgrm>, <vpgrmgrmgrm><grammar>.cl</vpgrmgrmgrm>, <vpgrmgrmgrm><grammar>.pro</vpgrmgrmgrm> containing the generated parser

Grammar files

The input <vpgrmgrmgrm>vipLalrGen</vpgrmgrmgrm> is a grammar file. As mentioned the IDE supports token coloring if the extension is vipgrm.

A grammar file contains a named grammar:

grammar expressionGrm
    open expression, expressionGrmSem
 
nonassoc t_cmp.
left t_plus.
left t_mult.
right t_power.
 
nonterminals
    exp : expression.
rules
    exp { mkBinOp(Op, A, B) } ==>
        exp { A },
        [t_cmp] { Op },
        exp { B }.
 
    exp { mkBinOp(Op, A, B) } ==>
        exp { A },
        [t_power] { Op },
        exp { B }.
 
    exp { mkBinOp(Op, A, B) } ==>
        exp { A },
        [t_mult] { Op },
        exp { B }.
 
    exp { mkBinOp(Op, A, B) } ==>
        exp { A },
        [t_plus] { Op },
        exp { B }.
 
    exp { E } ==>
        [t_lpar],
        exp { E },
        [t_rpar].
 
    exp { number(toTerm(real, N)) } ==>
        [t_number] { N }.
 
    exp { bool(toTerm(boolean, N)) } ==>
        [t_boolean] { N }.
 
end grammar expressionGrm

nonterminals & rules

The grammar file (among other) contains <vpgrmgrmgrm>nonterminals</vpgrmgrmgrm> and <vpgrmgrmgrm>rules</vpgrmgrmgrm> sections.

<vpgrmgrmgrm>nonterminals</vpgrmgrmgrm> sections declares nonterminal symbols and the type of the corresponding parse trees. The nonterminals section above states that <vpgrmgrmgrm>exp</vpgrmgrmgrm> is a nonterminal symbol and that it produces parse trees of the (Visual Prolog) type <vpgrmgrmgrm>expression</vpgrmgrmgrm>.

<vpgrmgrmgrm>rules</vpgrmgrmgrm> sections contains rules that both defines how valid derivations of nonterminal system looks and how the corresponding parse tree is constructed.

Everything in braces have to do with the parse trees. If we initially disregard it, we only see what have to do with the derivations

rules
    exp ==> exp, [t_cmp], exp.
 
    exp ==> exp, [t_power], exp.
 
    exp ==> exp, [t_mult], exp.
 
    exp ==> exp, [t_plus], exp.
 
    exp ==> [t_lpar], exp, [t_rpar].
 
    exp ==> [t_number].
 
    exp ==> [t_boolean].

The first rule says that from the nonterminal symbol <vpgrmgrmgrm>exp</vpgrmgrmgrm> we can derive <vpgrmgrmgrm>exp</vpgrmgrmgrm> followed by <vpgrmgrmgrm>[t_cmp]</vpgrmgrmgrm> followed by <vpgrmgrmgrm>exp</vpgrmgrmgrm>, where <vpgrmgrmgrm>[t_cmp]</vpgrmgrmgrm> is the terminal symbol <vpgrmgrmgrm>t_cmp</vpgrmgrmgrm>.

Example

Here is a derivation that corresponds to the expression <vpgrmgrmgrm>7 < 5</vpgrmgrmgrm> (assuming that <vpgrmgrmgrm>t_number</vpgrmgrmgrm> are numbers and <vpgrmgrmgrm>t_cmp</vpgrmgrmgrm> are comparison operators):

exp
    ==> exp, [t_cmp], exp 
    ==> exp, [t_cmp], [t_number] 
    ==> [t_number], [t_cmp], [t_number]

The derivation in the example is a rightmost derivation, because in each step we derive something from the rightmost nonterminal symbol. An LR parser will make reductions in the inverse order of the derivations in a rightmost derivation. LR parsing means that we scan tokens from Left to right and produce an inverse Rightmost derivation.

As mentioned the braces describes how to construct a corresponding parse tree. The braces on the left hand side contains a Visual Prolog expression that constructs the node in the parse tree. The braces in the right hand side of the rules defines variable names for the corresponding sub-trees.

Given the rule

   exp { mkBinOp(Op, A, B) } ==>
        exp { A },
        [t_cmp] { Op },
        exp { B }.

<vpgrmgrmgrm>A</vpgrmgrmgrm> and <vpgrmgrmgrm>B</vpgrmgrmgrm> contains the parse-trees of the two <vpgrmgrmgrm>exp</vpgrmgrmgrm>s and <vpgrmgrmgrm>Op</vpgrmgrmgrm> contains the string of the terminal symbol <vpgrmgrmgrm>t_cmp</vpgrmgrmgrm>. The resulting parse tree is calculated as <vpgrmgrmgrm>mkBinOp(Op, A, B)</vpgrmgrmgrm>.

Precedence

The grammar rules in the expression grammar are by themselves ambiguous, in the sense that 3 + 4 * 5 can derived by both these rightmost derivations:

exp 
    ==> exp, [t_add], exp 
    ==> exp, [t_add], exp, [t_mult], exp 
    ==> exp, [t_add], exp, [t_mult], [t_number] 
    ==> exp, [t_add], [t_number], [t_mult], [t_number] 
    ==> [t_number], [t_add], [t_number], [t_mult], [t_number]

exp 
    ==> exp, [t_mult], exp 
    ==> exp, [t_mult], [t_number] 
    ==> exp, [t_add], exp, [t_mult], [t_number] 
    ==> exp, [t_add], [t_number], [t_mult], [t_number] 
    ==> [t_number], [t_add], [t_number], [t_mult], [t_number]

The ambiguity is not relevant with regards to whether the expression is a valid expression or not, but the two derivations corresponds to two different parse trees (corresponding to the expressions <vpgrmgrmgrm>(3 + 4) * 5</vpgrmgrmgrm> and <vpgrmgrmgrm>3 + (4 * 5)</vpgrmgrmgrm>, respectively). Obviously, we are interested in a particular parse tree (i.e. the latter).

Moreover the ambiguity in the grammar actually makes it impossible for the parser generator to produce a parser (see next section).

This particular kind of ambiguity can be solved by stating the precedence of the terminal symbols corresponding to the operators:

 
nonassoc t_cmp.
left t_plus.
left t_mult.
right t_power.

a precedence declaration states one of the associativites (<vpgrmgrmgrm>nonassoc</vpgrmgrmgrm>, <vpgrmgrmgrm>left</vpgrmgrmgrm> or <vpgrmgrmgrm>right</vpgrmgrmgrm>) followed by a comma separated sequence of terminal symbols and terminated by a dot.

The terminals in a precedence declaration have the same precedence level and associativity. The declarations are stated in increasing precedence (i.e. the first declaration have lowest precedence).

Given <vpgrmgrmgrm>A [t1] B [t2] C</vpgrmgrmgrm>

• If <vpgrmgrmgrm>t1</vpgrmgrmgrm> has higher precedence than <vpgrmgrmgrm>t2</vpgrmgrmgrm> then <vpgrmgrmgrm>A [t1] B</vpgrmgrmgrm> will be reduced to <vpgrmgrmgrm>AB</vpgrmgrmgrm> and then <vpgrmgrmgrm>AB [t2] C</vpgrmgrmgrm> will be reduced.
• If <vpgrmgrmgrm>t1</vpgrmgrmgrm> has lower precedence than <vpgrmgrmgrm>t2</vpgrmgrmgrm> then <vpgrmgrmgrm>B [t2] C</vpgrmgrmgrm> will be reduced to <vpgrmgrmgrm>BC</vpgrmgrmgrm> and then <vpgrmgrmgrm>a [t1] BC</vpgrmgrmgrm> will be reduced.
• If <vpgrmgrmgrm>t1</vpgrmgrmgrm> and <vpgrmgrmgrm>t2</vpgrmgrmgrm> have same precedence, then
  • If they are <vpgrmgrmgrm>left</vpgrmgrmgrm> then <vpgrmgrmgrm>A [t1] B</vpgrmgrmgrm> will be reduced to <vpgrmgrmgrm>AB</vpgrmgrmgrm> and then <vpgrmgrmgrm>AB [t2] C</vpgrmgrmgrm> will be reduced.
  • If they are <vpgrmgrmgrm>right</vpgrmgrmgrm> then <vpgrmgrmgrm>B [t2] C</vpgrmgrmgrm> will be reduced to <vpgrmgrmgrm>BC</vpgrmgrmgrm> and then <vpgrmgrmgrm>a [t1] BC</vpgrmgrmgrm> will be reduced.
  • If they are <vpgrmgrmgrm>nonassoc</vpgrmgrmgrm> the construction is illegal and the parser will issue s syntax error.

Error recovery

During parsing the parser will look at the next terminal symbol and:

• Decide that parsing is successfully ended and thus <vpgrmgrmgrm>accept</vpgrmgrmgrm> the input
• <vpgrmgrmgrm>shift</vpgrmgrmgrm> the terminal symbol on its stack
• <vpgrmgrmgrm>reduce</vpgrmgrmgrm> the top of the stack
• or else conclude that there is a syntax <vpgrmgrmgrm>error</vpgrmgrmgrm> in the input

In the last case the parser will report the error and it will not produce a parse tree. It is however desirable to try to detect additional syntax errors in the remaining input. Error recovery is attempting to get the parser and input back in synchronization, so that additional syntax errors can be detected.

The parsing machinery in PFC can use a technique involving an special error-terminal. In the grammar files the terminal <vpgrmgrmgrm>[error]</vpgrmgrmgrm> is reserved for the error terminal.

The basic use of the <vpgrmgrmgrm>[error]</vpgrmgrmgrm> terminal is to add grammar rules of the form:

rules
    ...
    something ==>
        [error].

When the parser detects an error it will do recovery in the following way:

• It will pop entities from its stack (corresponding to skipping backwards in the things that are in progress of being parsed), until it get to a state where the <vpgrmgrm>[error]</vpgrmgrm> terminal can be shifted on the stack. Given the grammar rule above this would be in a place where <vpgrmgrm>something</vpgrmgrm> is a valid "next thing".
• It will then perform actions corresponding to the grammar rule (shift the error terminal and perform the reduction)
• Finally it will skip input until it finds a terminal that can be shifted onto the stack

If all that succeeds recovery is completed and parsing resumes normally.

Example

This strategy works quite good if the language have some clear "synchronization" points. In Visual Prolog the section keywords <vpgrmgrm>predicates</vpgrmgrm>, <vpgrmgrm>clauses</vpgrmgrm>, etc are very good synchronization points: It is quite certain how to parse things after one of these words.

rules
    sectionList ==> .
    sectionList ==> sectionList, section.
rules
    section ==> [t_predicates], ...
    section ==> [t_clauses], ...
    ...
    section ==> [error].

If a syntax error is detected in a <vpgrmgrm>section</vpgrmgrm> the parser stack will contain a <vpgrmgrm>sectionList</vpgrmgrm> followed by whatever it is in the middle of parsing in the current section.

It will then pop until it reaches the <vpgrmgrm>sectionList</vpgrmgrm>, because at that point our error-section (and thus the <vpgrmgrm>[error]</vpgrmgrm> terminal) would be a valid next thing.

It will shift and reduce the error-section and then skip terminal symbols until we meet one that can legally follow a section, like a section keyword, <vpgrmgrm>end interface</vpgrmgrm>, ...

Cursor

For many reasons (for example to give error messages and create browse and debug info) the compiler deals with positions in the input. Every rule have access to a (predefined) variable named <vpgrmgrm>Cursor</vpgrmgrm>. This variable holds four pieces of information:

• The start position of the construction
• The end position of the construction
• Comments before in the construction (comments are discussed in the next section)
• Comments after the construction

The parser calculates such a cursor for a rule from the cursors of the components of the rule. The position handling is quite simple, given a rule:

b  ==> a1, a2, ... an.

• The start position of <vpgrmgrm>b</vpgrmgrm> is the start position of <vpgrmgrm>a1</vpgrmgrm>
• The end postion of <vpgrmgrm>b</vpgrmgrm> is the end position of <vpgrmgrm>an</vpgrmgrm>.

The <vpgrmgrm>Cursor</vpgrmgrm> variable can be used in the semantic action in the rule head together with the variables defined for the body components:

b { mkB(Cursor, A1, A2, ... An ) } ==> a1 { A1 }, a2 { A2 }, ... an { An }.

Comments

In most parsing schemes comments are simply discarded by the lexer, so the parser never see any comments at all. This is fine for many purposes, but there may be situations where you want to use the comments. Visual Prolog programs can for example contain documentation comments which it may be nice to deal with. And if you want to pretty print or restructure a program programmatically you don't want to discard the comments.

As mentioned above comments are placed in the cursors. Many cursors are however discarded during parsing, but the comments in these cursors should not be discarded. Therefore the parser moves comments from discarded cursors to cursors that stay are not discarded.

The parser assumes that cursors that is used in a semantical action by means of the <vpgrmgrm>Cursor</vpgrmgrm> variable will stay alive, and will therefore assume that comments on such a cursor will survive. Subsequently it may move additional comments to such a cursor.

Consider a rule for an if-then (for simplicity without an else part) construction:

term { mkIfThen(Cursor, Cond, Then ) } ==>
     [t_if],
     term { Cond },
     [t_then]
     term { Then },
     [t_end],
     [t_if].

When the parser have to reduce this rule it have access to the six cursors corresponding to the sub-components, it also knows which of them that can carry comments and which will be discarded.

In general pre-comments are moved leftwards and post comments are moved rightwards. So pre-comment of [t_then] will be moved to become post-comments of Cond (given that Cond can carry comments). Likewise the post-comments of [t_then] will be moved to become pre-comments of Then (given that Then can carry comments).

If the first symbol is going to be discarded its pre comments will move to become pre-comments in the parent cursor. And likewise the the post comments of the last symbol will be moved to the post-comments of the parent cursor.

If several adjacent symbols cannot carry comments then the exact movement of comments depends on the placement in the rule. For the if-then rule above:

• the pre-comments of [t_end] will become post comments in Then, and
• the post-comments of vpgrm>[t_end]</vpgrm> and all comments of [t_if] will go to the parent cursor

The same principle of "most goes to the parent" will be used in the beginning of the rule. In the middle of a rule most comments will move to the left.

It is important to notice that comments end's up on cursors that have been used with the Cursor variable, so to preserve all comments it is important to retain all such referenced cursors. Or to put it differently if a semantic action uses the Cursor variable it also takes responsibility for the comments on that cursor.

It is also important to notice that the parser moves comments around by updating making destructive updates to the cursor structs, so it is not only important to retain cursors received during parsing, you must retain exactly those cursors your semantic actions receive. To be safe you should not assume that comment are stable until the entire parsing is completed.

Conflicts

As briefly mentioned above there are situations where the parser generator cannot produce a parser. Briefly speaking the parser generator generates parset tables that in a given situation instructs the parser either to shift the next terminal symbol onto an parser stack or to reduce the top elements of the parser stack using one of the grammar rules (aka production rule).

But when calculating the parser table it may turn out that it would both be equally good/bad to shift the next terminal onto the stack and to reduce the stack top. This is a so called shift-reduce conflict. Likewise it can turn out that there are two equally good reductions that could be applied to the top of the parse stack, i.e. a so called reduce-reduce conflict.

Details about parsing conflicts can be found in the produced log file.

A grammar for a certain language can be written in many ways (just like a program with a certain functionality can be written in many ways), and if may often be possible to rewrite a grammar that have conflicts to another one that doesn't have conflicts (but recognizes the same language).

It is outside the scope of this article to go into details about conflicts and how to resolve them, unfortunately it may require relatively deep understanding of parsing. As mentioned above vipLalrGen is an LALR(1) parser generator like YACC and the like, so there is lots of existing material about the subject "out there".

External References

Basic reading is the "Dragon Book".