Overview  Package   Class  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD

com.wcohen.secondstring.tokens
Interface Tokenizer

All Known Implementing Classes:
NGramTokenizer, SimpleTokenizer
public interface Tokenizer

Split a string into tokens.

Method Summary
 Token intern(java.lang.String s)
          Convert a given string into a token
 Token[] tokenize(java.lang.String input)
          Return tokenized version of a string
 

Method Detail

tokenize

public Token[] tokenize(java.lang.String input)
Return tokenized version of a string

intern

public Token intern(java.lang.String s)
Convert a given string into a token

Overview  Package   Class  Tree  Deprecated  Index  Help 
 PREV CLASS   NEXT CLASS FRAMES    NO FRAMES    
SUMMARY: NESTED | FIELD | CONSTR | METHOD DETAIL: FIELD | CONSTR | METHOD