Discussion forum for David Beazley

Conditional lexing and start conditions in SLY?


#1

I am not sure if this is the right place to ask, but is there conditional lexing in SLY? Currently I am using this quite a lot in PLY.

Many thanks!
Pascal


#2

Ah yes. This is something that I’ve been thinking about in SLY, but the exact API hasn’t been fully determined yet. One way to do it is to force a lexer state change by raising an exception. The attached code sample shows how this works.

import sly

class CalcLexer(sly.Lexer):
    tokens = { NUMBER, PLUS, MINUS, TIMES, DIVIDE, LBRACE }
    ignore = ' \t\n'

    NUMBER = r'\d+'
    PLUS = r'\+'
    TIMES = r'\*'
    MINUS = r'-'
    DIVIDE = r'/'
    LBRACE = r'\{'

    def LBRACE(self, t):
        raise sly.LexerStateChange(BlockLexer, t)

class BlockLexer(sly.Lexer):
    tokens = { RBRACE, NAME, VALUE }
    ignore = ' \t\n'

    NAME = r'[a-zA-Z_][a-zA-Z0-9_]+'
    VALUE = r'\d+'
    RBRACE = r'\}'

    def RBRACE(self, t):
        raise sly.LexerStateChange(CalcLexer, t)


if __name__ == '__main__':
    lexer = CalcLexer()
    for tok in lexer.tokenize('3 + 4 { foo bar 1234 } * 6'):
        print(tok)

The notion of “inclusive” and “exclusive” states really hasn’t been formalized here. However, I could imagine adding it via some kind of inheritance mechanism. This message is probably a good excuse to go look at this further.


#3

Many thanks for your quick reply.

And in particular for implementing this via the begin(cls), push_state(cls), and pop_state(cls) methods in v 0.3 already!

Pascal


#4

I’ve been meaning to add this feature for awhile, but first had to resolve the overall mechanism by which I was going to make “inclusive states” work. SLY does it via inheritance, but there are some tricky facets to it. The current version should be a start in that direction so I’ll be curious to hear how it works.