This module provides a lexical scanner for the RenderMan Shading Language. You use it in the same way as the Python tokenize module is used.
Reads a Shading Language input stream and creates tokens.
The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string.
The second parameter, tokeater, must also be a callable object. It is called with six parameters: the token type, the token string, a tuple (srow, scol) specifying the row and column where the token begins in the source, a tuple (erow, ecol) giving the ending position of the token, the line on which the token was found and the filename of the current file.
The token type can be one of:
By default, the filename argument is an empty string. It will only be the actual filename if you provide a preprocessed file stream as input (so you should first run cpp on any shader). The tokenizer actually expects preprocessed data as it doesn’t handle comments.