4.9. sltokenize — Tokenizer for the RenderMan Shading Language

This module provides a lexical scanner for the RenderMan Shading Language. You use it in the same way as the Python tokenize module is used.

cgkit.sltokenize.tokenize(readline, tokeater)

Reads a Shading Language input stream and creates tokens.

The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string.

The second parameter, tokeater, must also be a callable object. It is called with six parameters: the token type, the token string, a tuple (srow, scol) specifying the row and column where the token begins in the source, a tuple (erow, ecol) giving the ending position of the token, the line on which the token was found and the filename of the current file.

The token type can be one of:

• WHITESPACE: This is a series of blanks and/or tabs
• NAME: A valid identifier name or keyword
• NUMBER: An integer or float
• STRING: A string enclosed in '"'.
• NEWLINE: A newline character.
• OPERATOR: An operator such as '+', '-', '!', '==', '!=', etc.
• CHARACTER: A single character that doesn’t fit anything else.
• TYPE: A Shading Language type (float, point, vector, normal, matrix, color)

By default, the filename argument is an empty string. It will only be the actual filename if you provide a preprocessed file stream as input (so you should first run cpp on any shader). The tokenizer actually expects preprocessed data as it doesn’t handle comments.