cpp.tokenize Namespace Reference

Classes
class	Token

Functions
def	GetTokens (source)
def	main (argv)

Variables
	VALID_IDENTIFIER_CHARS = set(_letters + _letters.upper() + '_0123456789$')
	HEX_DIGITS = set('0123456789abcdefABCDEF')
	INT_OR_FLOAT_DIGITS = set('01234567890eE-+')
string	UNKNOWN = 'UNKNOWN'
string	SYNTAX = 'SYNTAX'
string	CONSTANT = 'CONSTANT'
string	NAME = 'NAME'
string	PREPROCESSOR = 'PREPROCESSOR'
	WHENCE_STREAM
	WHENCE_QUEUE

Function Documentation

GetTokens()

def cpp.tokenize.GetTokens

(

source

)

Returns a sequence of Tokens.

Args:
  source: string of C++ source code.

Yields:
  Token that represents the next token in the source.

main()

def cpp.tokenize.main

(

argv

)

Driver mostly for testing purposes.

Variable Documentation

CONSTANT

string cpp.tokenize.CONSTANT = 'CONSTANT'

HEX_DIGITS

cpp.tokenize.HEX_DIGITS = set('0123456789abcdefABCDEF')

INT_OR_FLOAT_DIGITS

cpp.tokenize.INT_OR_FLOAT_DIGITS = set('01234567890eE-+')

NAME

string cpp.tokenize.NAME = 'NAME'

PREPROCESSOR

string cpp.tokenize.PREPROCESSOR = 'PREPROCESSOR'

SYNTAX

string cpp.tokenize.SYNTAX = 'SYNTAX'

UNKNOWN

string cpp.tokenize.UNKNOWN = 'UNKNOWN'

VALID_IDENTIFIER_CHARS

cpp.tokenize.VALID_IDENTIFIER_CHARS = set(_letters + _letters.upper() + '_0123456789$')

WHENCE_QUEUE

cpp.tokenize.WHENCE_QUEUE

WHENCE_STREAM

cpp.tokenize.WHENCE_STREAM