Discussion forum for David Beazley

Conditional lexing and start conditions in SLY?


I am not sure if this is the right place to ask, but is there conditional lexing in SLY? Currently I am using this quite a lot in PLY.

Many thanks!


Ah yes. This is something that I’ve been thinking about in SLY, but the exact API hasn’t been fully determined yet. One way to do it is to force a lexer state change by raising an exception. The attached code sample shows how this works.

import sly

class CalcLexer(sly.Lexer):
    ignore = ' \t\n'

    NUMBER = r'\d+'
    PLUS = r'\+'
    TIMES = r'\*'
    MINUS = r'-'
    DIVIDE = r'/'
    LBRACE = r'\{'

    def LBRACE(self, t):
        raise sly.LexerStateChange(BlockLexer, t)

class BlockLexer(sly.Lexer):
    tokens = { RBRACE, NAME, VALUE }
    ignore = ' \t\n'

    NAME = r'[a-zA-Z_][a-zA-Z0-9_]+'
    VALUE = r'\d+'
    RBRACE = r'\}'

    def RBRACE(self, t):
        raise sly.LexerStateChange(CalcLexer, t)

if __name__ == '__main__':
    lexer = CalcLexer()
    for tok in lexer.tokenize('3 + 4 { foo bar 1234 } * 6'):

The notion of “inclusive” and “exclusive” states really hasn’t been formalized here. However, I could imagine adding it via some kind of inheritance mechanism. This message is probably a good excuse to go look at this further.


Many thanks for your quick reply.

And in particular for implementing this via the begin(cls), push_state(cls), and pop_state(cls) methods in v 0.3 already!



I’ve been meaning to add this feature for awhile, but first had to resolve the overall mechanism by which I was going to make “inclusive states” work. SLY does it via inheritance, but there are some tricky facets to it. The current version should be a start in that direction so I’ll be curious to hear how it works.