Add tokenization and truncation funcionality to Tokenize() function.

Add tokenization and truncation funcionality to Tokenize() function. #227