When I tested this particular scenario(that it shows above), it returned 13 instead of 14. I believe this may be because in the tokens that are mentioned above, it shows that there are two spaces after "and" when there appears to be only one space after the "and" in the actual code.Default delimiters i.e. " \t\n\r\f" will be used to tokenize the string. Further, the delimiters themselves will not be returned as tokens. Thus, it will return the following tokens: tokens, with, tabs, and, new, lines. Had it been: StringTokenizer st = new StringTokenizer(str, " \t\n\r\f", true), it would have returned 14 tokens: tokens, (tab character), with, (space character) , tabs, (space character), and, (space character), (space character), (CR character), (Line Feed character), new, (space character), lines. (Note: comma and the leading space in the above list of tokens are not part of the tokens but are there just to present the output here.)
I could be wrong though, just want to make sure that I'm not testing or understanding something incorrectly.