Discussion forum for David Beazley

What in the heck is the SLY project?


#1

In a nutshell, SLY is a modern replacement for the PLY library (http://www.dabeaz.com/ply/index.html). You’d use it to solve the same sorts of lexing and parsing problems. A bit of history though…

The PLY project started as a very hastily put together “hack” that was created over a two week period during spring break in 2001. I was scheduled to teach Compilers in the Spring quarter and wanted to see what it would be like to write a compiler in Python. At that time, there were very few available tools–and none that I really felt were suitable for a class. PLY was an attempt at making something that looked pretty similar to Unix lex/yacc.

At this time, Python would have been about version 2.1. There were no new-style classes, no generators, no metaclasses, no decorators, and whole bunch of other modern Python idioms that we now take for granted. The code was designed to run fast on a 300mhz SPARC processor and employed all sorts of crazy caching. There is madness contained in the code. For instance, I once found a loop like this::

for n in range(len(C)):
     x = C[n]
     n += 1

Who writes garbage like that?!?!?! Well, apparently me. I digress.

So, all of that said. SLY is a rethinking of the entire PLY project using Python 3.5+ as a starting point. It discards a lot of flaky API features and makes used of more modern Python idioms such as the iteration protocol. By far the biggest change is that SLY relies heavily on metaclasses to create a kind of domain specific language for writing lexers and parsers. In some sense, PLY was doing the same thing, but it was layered on top of doc-strings and other hacks. Had metaclasses evolved to their current state back then, I might have used them. However, I had to wait until 2016 instead.

So that’s about it. Oh yeah, I also needed a recursive acronym. So, SLY stands for “Sly Lex Yacc”. Once you see its abuse of metaclasses, the “Sly” part of that will probably make more sense ;-).