pyre.parsing.Token

Module Contents

class pyre.parsing.Token.Token(lexeme='', locator=None, **kwds)

Base class for tokens, the atomic units of recognizable text in a stream

name =

head =

tail =

pattern

regex =

scanner

__slots__ = ['lexeme', 'locator']

__len__(self)

Compute the length of the lexeme

N.B.: this is not the same as the footprint of the token in the input stream, which includes the {head} and {tail} of the token

__str__(self)

Textual representation, mostly for debugging purposes