Package com.wcohen.secondstring.tokens

Interface Summary

Tokenizer

Split a string into tokens.

Class Summary

NGramTokenizer	Wraps another tokenizer, and adds all computes all ngrams of characters from a single token produced by the inner tokenizer.
SimpleTokenizer	Simple implementation of a Tokenizer.
Token	An interned version of a string.