The lexer splits an input stream into tokens. These are handed to the parser. Ritex::Parser takes care of setting up and configuring the lexer.

In order to support macros, the lexer maintains a stack of strings. Pushing a string onto the stack will cause lex to yield tokens from that string, until it reaches the end, at which point it will discard the string and resume yielding tokens from the previous string.

To handle macros, the lexer is stateful. Normally it ignores all spacing. After hitting an ENV token it will start returning SPACE tokens for each space until it hits a ’}’.