AST is an abstraction of the CST (concrete syntax tree, or, parse tree). The concrete syntax tree is the tree resulting from the productions (in the grammar) used to parse the file. So your AST is basically derived from your grammar definition, but has for transformed Exp / | \ / | \ * Ident … Read more
Pythoscope does this to the test cases it automatically generates as does the 2to3 tool for python 2.6 (it converts python 2.x source into python 3.x source). Both these tools uses the lib2to3 library which is an implementation of the python parser/compiler machinery that can preserve comments in source when it’s round tripped from source … Read more
Raphael wrote: Does antlr build the AST for us automatically by option{output=AST;}? Or do I have to make the tree myself? If it does, then how to spit out the nodes on that AST? No, the parser does not know what you want as root and as leaves for each parser rule, so you’ll have … Read more
Ok, let’s build a simple math example. Building an AST is totally overkill for such a task but it’s a nice way to show the principle. I’ll do it in C# but the Java version would be very similar. The grammar First, let’s write a very basic math grammar to work with: grammar Math; compileUnit … Read more
A concrete syntax tree represents the source text exactly in parsed form. In general, it conforms to the context-free grammar defining the source language. However, the concrete grammar and tree have a lot of things that are necessary to make source text unambiguously parseable, but do not contribute to actual meaning. For example, to implement … Read more
Here’s two examples, a simple one and a nasty one (C++’s “most vexing parse”). A simple Fibonacci program from http://talkbinary.com/programming/c/fibonacci-in-c/ parsed as C++ code: int fib(int n) { if ( n == 0 || n == 1 ) return n; int fib1 = 0; int fib2 = 1; int fib = 0; for ( int … Read more
I’ve been building tools (DMS Software Reengineering Toolkit) to do general purpose program manipulation (with language translation being a special case) since 1995, supported by a strong team of computer scientists. DMS provides generic parsing, AST building, symbol tables, control and data flow analysis, application of translation rules, regeneration of source text with comments, etc., … Read more
The problem of converting an AST back into source code is generally called “prettyprinting”. There are two subtle variations: regenerating the text matching the original as much as possible (I call this “fidelity printing”), and (nice) prettyprinting, which generates nicely formatted text. And how you print matters depending on whether coders will be working on … Read more
datamap = eval(input(‘Provide some data here: ‘)) means that you actually evaluate the code before you deem it to be unsafe or not. It evaluates the code as soon as the function is called. See also the dangers of eval. ast.literal_eval raises an exception if the input isn’t a valid Python datatype, so the code … Read more