gov.sandia.cognition.text.token
Class LetterNumberTokenizer

java.lang.Object
  extended by gov.sandia.cognition.util.AbstractCloneableSerializable
      extended by gov.sandia.cognition.text.token.AbstractTokenizer
          extended by gov.sandia.cognition.text.token.AbstractCharacterBasedTokenizer
              extended by gov.sandia.cognition.text.token.LetterNumberTokenizer
All Implemented Interfaces:
Tokenizer, CloneableSerializable, Serializable, Cloneable

public class LetterNumberTokenizer
extends AbstractCharacterBasedTokenizer

A tokenizer that creates tokens from sequences of letters and numbers, treating everything else as a delimiter.

Since:
3.0
Author:
Justin Basilico
See Also:
Serialized Form

Constructor Summary
LetterNumberTokenizer()
          Creates a new LetterNumberTokenizer.
 
Method Summary
 boolean isTokenMember(char c)
          Determines if the given character is considered to be part of a token.
 
Methods inherited from class gov.sandia.cognition.text.token.AbstractCharacterBasedTokenizer
tokenize
 
Methods inherited from class gov.sandia.cognition.text.token.AbstractTokenizer
tokenize, tokenize
 
Methods inherited from class gov.sandia.cognition.util.AbstractCloneableSerializable
clone
 
Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface gov.sandia.cognition.util.CloneableSerializable
clone
 

Constructor Detail

LetterNumberTokenizer

public LetterNumberTokenizer()
Creates a new LetterNumberTokenizer.

Method Detail

isTokenMember

public boolean isTokenMember(char c)
Description copied from class: AbstractCharacterBasedTokenizer
Determines if the given character is considered to be part of a token.

Specified by:
isTokenMember in class AbstractCharacterBasedTokenizer
Parameters:
c - A character.
Returns:
True if the character can be part of a token; otherwise, false.