gov.sandia.cognition.text.token
Class AbstractTokenizer

java.lang.Object
  extended by gov.sandia.cognition.util.AbstractCloneableSerializable
      extended by gov.sandia.cognition.text.token.AbstractTokenizer
All Implemented Interfaces:
Tokenizer, CloneableSerializable, Serializable, Cloneable
Direct Known Subclasses:
AbstractCharacterBasedTokenizer

public abstract class AbstractTokenizer
extends AbstractCloneableSerializable
implements Tokenizer

Abstract implementation of the Tokenizer interface. It turns the tokenize call for a String into a Reader.

Since:
3.0
Author:
Justin Basilico
See Also:
Serialized Form

Constructor Summary
AbstractTokenizer()
          Creates a new AbstractTokenizer.
 
Method Summary
 Iterable<Token> tokenize(String s)
          Convert the given string into a corresponding ordered list of tokens.
 Iterable<Token> tokenize(Textual textual)
          Convert the given string into a corresponding ordered list of tokens.
 
Methods inherited from class gov.sandia.cognition.util.AbstractCloneableSerializable
clone
 
Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface gov.sandia.cognition.text.token.Tokenizer
tokenize
 
Methods inherited from interface gov.sandia.cognition.util.CloneableSerializable
clone
 

Constructor Detail

AbstractTokenizer

public AbstractTokenizer()
Creates a new AbstractTokenizer.

Method Detail

tokenize

public Iterable<Token> tokenize(String s)
Description copied from interface: Tokenizer
Convert the given string into a corresponding ordered list of tokens.

Specified by:
tokenize in interface Tokenizer
Parameters:
s - The string to tokenize.
Returns:
The ordered list of tokens.

tokenize

public Iterable<Token> tokenize(Textual textual)
Description copied from interface: Tokenizer
Convert the given string into a corresponding ordered list of tokens.

Specified by:
tokenize in interface Tokenizer
Parameters:
textual - The object to tokenize.
Returns:
The ordered list of tokens.