gov.sandia.cognition.text.token
Interface Tokenizer

All Superinterfaces:
Cloneable, CloneableSerializable, Serializable
All Known Implementing Classes:
AbstractCharacterBasedTokenizer, AbstractTokenizer, LetterNumberTokenizer

public interface Tokenizer
extends CloneableSerializable

Interface for a class that converts strings into tokens.

Since:
3.0
Author:
Justin Basilico

Method Summary
 Iterable<Token> tokenize(Reader reader)
          Converts the string from the given reader into an ordered list of tokens.
 Iterable<Token> tokenize(String s)
          Convert the given string into a corresponding ordered list of tokens.
 Iterable<Token> tokenize(Textual textual)
          Convert the given string into a corresponding ordered list of tokens.
 
Methods inherited from interface gov.sandia.cognition.util.CloneableSerializable
clone
 

Method Detail

tokenize

Iterable<Token> tokenize(String s)
Convert the given string into a corresponding ordered list of tokens.

Parameters:
s - The string to tokenize.
Returns:
The ordered list of tokens.

tokenize

Iterable<Token> tokenize(Reader reader)
Converts the string from the given reader into an ordered list of tokens.

Parameters:
reader - The reader to tokenize the data from.
Returns:
The ordered list of tokens.

tokenize

Iterable<Token> tokenize(Textual textual)
Convert the given string into a corresponding ordered list of tokens.

Parameters:
textual - The object to tokenize.
Returns:
The ordered list of tokens.