sltokenize -- Tokenizer for the RenderMan Shading Language

tokenize(readline, tokeneater)

Reads an input stream and creates tokens.

The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string.

The second parameter, tokeneater, must also be a callable object. It is called with six parameters: the token type, the token string, a tuple (srow, scol) specifying the row and column where the token begins in the source, a tuple (erow, ecol) giving the ending position of the token, the line on which the token was found and the filename of the current file.

The token type can be one of

• WHITESPACE: This is a series of blanks and/or tabs.
• NAME: A valid identifier name or keyword.
• NUMBER: An integer or float.
• STRING: A string enclosed in '"'.
• NEWLINE: A newline character.
• OPERATOR: An operator such as '+', '-', '!', '==', '!=', etc.
• CHARACTER: A single character that doesn't fit anything else.
• TYPE: A Shading Language type (float, point, vector, normal, matrix, color)

By default the filename argument is an empty string. It will only be the actual filename if you provide a preprocessed file stream as input (so you should first run cpp on any shader). The tokenizer actually expects preprocessed data as it doesn't handle comments.

The RenderMan (R) Interface Procedures and Protocol are:
Copyright 1988, 1989, 2000, Pixar
All Rights Reserved

RenderMan (R) is a registered trademark of Pixar