tokenizemodule is used.
The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as a string.
The second parameter, tokeneater, must also be a callable object. It is called with six parameters: the token type, the token string, a tuple (srow, scol) specifying the row and column where the token begins in the source, a tuple (erow, ecol) giving the ending position of the token, the line on which the token was found and the filename of the current file.
The token type can be one of
WHITESPACE: This is a series of blanks and/or tabs.
NAME: A valid identifier name or keyword.
NUMBER: An integer or float.
STRING: A string enclosed in '"'.
NEWLINE: A newline character.
OPERATOR: An operator such as '+', '-', '!', '==', '!=', etc.
CHARACTER: A single character that doesn't fit anything else.
TYPE: A Shading Language type (float, point, vector, normal, matrix, color)
cppon any shader). The tokenizer actually expects preprocessed data as it doesn't handle comments.
The RenderMan (R) Interface Procedures and Protocol are:
Copyright 1988, 1989, 2000, Pixar
All Rights Reserved
RenderMan (R) is a registered trademark of Pixar