ANTLR 4.5 – Mismatched Input ‘x’ expecting ‘x’

by

This seems to be a common misunderstanding of ANTLR:

Language Processing in ANTLR:

The Language Processing is done in two strictly separated phases:

  • Lexing, i.e. partitioning the text into tokens
  • Parsing, i.e. building a parse tree from the tokens

Since lexing must preceed parsing there is a consequence: The lexer is independent of the parser, the parser cannot influence lexing.

Lexing

Lexing in ANTLR works as following:

  • all rules with uppercase first character are lexer rules
  • the lexer starts at the beginning and tries to find a rule that matches best to the current input
  • a best match is a match that has maximum length, i.e. the token that results from appending the next input character to the maximum length match is not matched by any lexer rule
  • tokens are generated from matches:
    • if one rule matches the maximum length match the corresponding token is pushed into the token stream
    • if multiple rules match the maximum length match the first defined token in the grammar is pushed to the token stream

Example: What is wrong with your grammar

Your grammar has two rules that are critical:

FILEPATH: ('A'..'Z'|'a'..'z'|'0'..'9'|':'|'\\'|"https://stackoverflow.com/"|' '|'-'|'_'|'.')+ ;
TITLE: ('A'..'Z'|'a'..'z'|' ')+ ;

Each match, that is matched by TITLE will also be matched by FILEPATH. And FILEPATH is defined before TITLE: So each token that you expect to be a title would be a FILEPATH.

There are two hints for that:

  • keep your lexer rules disjunct (no token should match a superset of another).
  • if your tokens intentionally match the same strings, then put them into the right order (in your case this will be sufficient).
  • if you need a parser driven lexer you have to change to another parser generator: PEG-Parsers or GLR-Parsers will do that (but of course this can produce other problems).

More Related Contents:

  1. What is a ‘semantic predicate’ in ANTLR?
  2. Negating inside lexer- and parser rules
  3. ANTLR 4 tree inject/rewrite operator
  4. ANTLR: Is there a simple example?
  5. How to create AST with ANTLR4?
  6. If/else statements in ANTLR using listeners
  7. What does “fragment” mean in ANTLR?
  8. How does the ANTLR lexer disambiguate its rules (or why does my parser produce “mismatched input” errors)?
  9. When is EOF needed in ANTLR 4?
  10. When is EOF needed in ANTLR 4?
  11. ANTLR What is simpliest way to realize python like indent-depending grammar?
  12. How to output the AST built using ANTLR?
  13. How to convert a String to its equivalent LINQ Expression Tree?
  14. Practical difference between parser rules and lexer rules in ANTLR?
  15. lexers vs parsers
  16. Using ANTLR 3.3?
  17. Antlr 4.5 parser error during runtime
  18. Visualizing an AST created with ANTLR (in a .Net environment)
  19. Build symbol table from grammar [closed]
  20. How can I build an AST using ANTLR4? [duplicate]
  21. Compiling sample ANTRL4 output
  22. Handling errors in ANTLR4

Leave a Comment