TWEAST: a simple and effective technique to implement concrete-syntax AST rewriting using partial parsing

• Authors: Akim Demaille;Roland Levillain;Benoît Sigoure
• Venue: Proceedings of the 2009 ACM symposium on Applied Computing
• Year: 2009

Abstract Syntax Trees (ASTs) are commonly used to represent an input/output program in compilers and language processing tools. Many of the tasks of these tools consist in generating and rewriting ASTs. Such an approach can become tedious and hard to maintain for complex operations, namely program transformation, optimization, instrumentation, etc. On the other hand, concrete syntax provides a natural and simpler representation of programs, but it is not usually available as a direct feature of the aforementioned tools. We propose a simple technique to implement AST generation and rewriting in general purpose languages using concrete syntax. Our approach relies on extensions made in the scanner and the parser and the use of objects supporting partial parsing called Texts With Embedded Abstract Syntax Trees (TWEASTs). A compiler for a simple language (Tiger) written in C++ serves as an example, featuring transformations in concrete syntax: syntactic desugaring, optimization, code instrumentation such as bounds-checking, etc. Extensions of this technique to provide a full-fledged concrete-syntax rewriting framework are presented as well.