gov.sandia.cognition.learning.function.vector
Class ThreeLayerFeedforwardNeuralNetwork

java.lang.Object
  extended by gov.sandia.cognition.util.AbstractCloneableSerializable
      extended by gov.sandia.cognition.util.AbstractRandomized
          extended by gov.sandia.cognition.learning.function.vector.ThreeLayerFeedforwardNeuralNetwork
All Implemented Interfaces:
Evaluator<Vector,Vector>, GradientDescendable, ParameterGradientEvaluator<Vector,Matrix>, VectorFunction, VectorInputEvaluator<Vector,Vector>, Vectorizable, VectorizableVectorFunction, VectorOutputEvaluator<Vector,Vector>, CloneableSerializable, Randomized, Serializable, Cloneable

@PublicationReference(author="Wikipedia",
                      title="Multilayer perceptron",
                      type=WebPage,
                      year=2009,
                      url="http://en.wikipedia.org/wiki/Multilayer_perceptron")
public class ThreeLayerFeedforwardNeuralNetwork
extends AbstractRandomized
implements VectorizableVectorFunction, VectorInputEvaluator<Vector,Vector>, VectorOutputEvaluator<Vector,Vector>, GradientDescendable

This is a "standard" feedforward neural network with a single hidden layer. There are bias terms on the input and hidden layers. The hidden units are each applied with the same smooth (differentiable) squashing function and the outputs are a linear combination of the hidden units and the hidden-layer bias term.

Since:
3.0
Author:
Kevin R. Dixon
See Also:
Serialized Form

Field Summary
static double DEFAULT_INITIALIZATION_RANGE
          Default initialization range, 0.0010.
static int DEFAULT_RANDOM_SEED
          Default random seed, 1.
static DifferentiableUnivariateScalarFunction DEFAULT_SQUASHING_FUNCTION
          Default squashing function, AtanFunction.
protected  Vector hiddenToOutputBiasWeights
          Bias weights to add to each of the output units.
protected  Matrix hiddenToOutputWeights
          Matrix of weights to pre-multiply the hidden-unit activations by.
protected  Vector inputToHiddenBiasWeights
          Bias weights to add to each of the hidden units.
protected  Matrix inputToHiddenWeights
          Matrix of weights to pre-multiply the inputs by.
 
Fields inherited from class gov.sandia.cognition.util.AbstractRandomized
random
 
Constructor Summary
ThreeLayerFeedforwardNeuralNetwork()
          Creates a new instance of ThreeLayerFeedforwardNeuralNetwork
ThreeLayerFeedforwardNeuralNetwork(int numInputs, int numHidden, int numOutputs)
          Creates a new instance of ThreeLayerFeedforwardNeuralNetwork
ThreeLayerFeedforwardNeuralNetwork(int numInputs, int numHidden, int numOutputs, DifferentiableUnivariateScalarFunction squashingFunction)
          Creates a new instance of ThreeLayerFeedforwardNeuralNetwork
ThreeLayerFeedforwardNeuralNetwork(int numInputs, int numHidden, int numOutputs, DifferentiableUnivariateScalarFunction squashingFunction, int randomSeed, double initializationRange)
          Creates a new instance of ThreeLayerFeedforwardNeuralNetwork
 
Method Summary
 ThreeLayerFeedforwardNeuralNetwork clone()
          This makes public the clone method on the Object class and removes the exception that it throws.
 Matrix computeParameterGradient(Vector input)
          Computes the derivative of the function about the input with respect to the parameters of the function
 void convertFromVector(Vector parameters)
          Converts the object from a Vector of parameters.
 Vector convertToVector()
          Converts the object to a vector.
 Vector evaluate(Vector input)
          Evaluates the function on the given input and returns the output.
protected  Vector evaluateHiddenLayerActivation(Vector input)
          Computes the raw (unsquashed) activation at the hidden layer for the given input.
protected  Vector evaluateOutputFromSquashedHiddenLayerActivation(Vector squashedHiddenActivation)
          Evaluates the output from the squashed hidden-layer activation.
protected  Vector evaluateSquashedHiddenLayerActivation(Vector hiddenActivation)
          Evaluates the squashed hidden-layer activation from its raw activation value.
 int getHiddenDimensionality()
          Gets the number of hidden units, not including the bias term.
 double getInitializationRange()
          Getter for initializationRange
 int getInputDimensionality()
          Gets the expected dimensionality of the input vector to the evaluator, if it is known.
 int getNumParameters()
          Gets the total number of parameters in the neural net, including the bias weights.
 int getOutputDimensionality()
          Gets the expected dimensionality of the output vector of the evaluator, if it is known.
 DifferentiableUnivariateScalarFunction getSquashingFunction()
          Getter for squashingFunction
 void initializeWeights(int inputDimensionality, int hiddenDimensionality, int outputDimensionality)
          Initializes the neural net parameters for the given dimensions, not including the bias terms, using the object's random-number generator uniformly between the initialization range (and its negative value).
 void reinitializeWeights()
          Reinitializes the neural network parameters based on its current setup.
 void setHiddenDimensionality(int hiddenDimensionality)
          Sets the number of hidden units, not including the bias term, by re-initializing the neural net's weights.
 void setInitializationRange(double initializationRange)
          Setter for initializationRange
 void setInputDimensionality(int inputDimensionality)
          Sets the number of input units, not counting the bias term, by re-initializing the neural net's parameters.
 void setOutputDimensionality(int outputDimensionality)
          Sets the output dimensionality of the neural net by re-initializing the weights.
 void setSquashingFunction(DifferentiableUnivariateScalarFunction squashingFunction)
          Setter for squashingFunction
 
Methods inherited from class gov.sandia.cognition.util.AbstractRandomized
getRandom, setRandom
 
Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

DEFAULT_INITIALIZATION_RANGE

public static final double DEFAULT_INITIALIZATION_RANGE
Default initialization range, 0.0010.

See Also:
Constant Field Values

DEFAULT_SQUASHING_FUNCTION

public static final DifferentiableUnivariateScalarFunction DEFAULT_SQUASHING_FUNCTION
Default squashing function, AtanFunction.


DEFAULT_RANDOM_SEED

public static final int DEFAULT_RANDOM_SEED
Default random seed, 1.

See Also:
Constant Field Values

inputToHiddenWeights

protected Matrix inputToHiddenWeights
Matrix of weights to pre-multiply the inputs by.


inputToHiddenBiasWeights

protected Vector inputToHiddenBiasWeights
Bias weights to add to each of the hidden units.


hiddenToOutputWeights

protected Matrix hiddenToOutputWeights
Matrix of weights to pre-multiply the hidden-unit activations by.


hiddenToOutputBiasWeights

protected Vector hiddenToOutputBiasWeights
Bias weights to add to each of the output units.

Constructor Detail

ThreeLayerFeedforwardNeuralNetwork

public ThreeLayerFeedforwardNeuralNetwork()
Creates a new instance of ThreeLayerFeedforwardNeuralNetwork


ThreeLayerFeedforwardNeuralNetwork

public ThreeLayerFeedforwardNeuralNetwork(int numInputs,
                                          int numHidden,
                                          int numOutputs)
Creates a new instance of ThreeLayerFeedforwardNeuralNetwork

Parameters:
numInputs - Input dimensionality, not including the bias term, must be greater than zero.
numHidden - Number of hidden units, not including the bias term, must be greater than zero.
numOutputs - Output dimensionality, must be greater than zero.

ThreeLayerFeedforwardNeuralNetwork

public ThreeLayerFeedforwardNeuralNetwork(int numInputs,
                                          int numHidden,
                                          int numOutputs,
                                          DifferentiableUnivariateScalarFunction squashingFunction)
Creates a new instance of ThreeLayerFeedforwardNeuralNetwork

Parameters:
numInputs - Input dimensionality, not including the bias term, must be greater than zero.
numHidden - Number of hidden units, not including the bias term, must be greater than zero.
numOutputs - Output dimensionality, must be greater than zero.
squashingFunction - Function to apply at the hidden layer.

ThreeLayerFeedforwardNeuralNetwork

public ThreeLayerFeedforwardNeuralNetwork(int numInputs,
                                          int numHidden,
                                          int numOutputs,
                                          DifferentiableUnivariateScalarFunction squashingFunction,
                                          int randomSeed,
                                          double initializationRange)
Creates a new instance of ThreeLayerFeedforwardNeuralNetwork

Parameters:
numInputs - Input dimensionality, not including the bias term, must be greater than zero.
numHidden - Number of hidden units, not including the bias term, must be greater than zero.
numOutputs - Output dimensionality, must be greater than zero.
squashingFunction - Function to apply at the hidden layer.
randomSeed - Value to use as the random seed for the random-number generator.
initializationRange - Range of values to initialize the weights between, must be greater than or equal to zero.
Method Detail

clone

public ThreeLayerFeedforwardNeuralNetwork clone()
Description copied from class: AbstractCloneableSerializable
This makes public the clone method on the Object class and removes the exception that it throws. Its default behavior is to automatically create a clone of the exact type of object that the clone is called on and to copy all primitives but to keep all references, which means it is a shallow copy. Extensions of this class may want to override this method (but call super.clone() to implement a "smart copy". That is, to target the most common use case for creating a copy of the object. Because of the default behavior being a shallow copy, extending classes only need to handle fields that need to have a deeper copy (or those that need to be reset). Some of the methods in ObjectUtil may be helpful in implementing a custom clone method. Note: The contract of this method is that you must use super.clone() as the basis for your implementation.

Specified by:
clone in interface GradientDescendable
Specified by:
clone in interface Vectorizable
Specified by:
clone in interface VectorizableVectorFunction
Specified by:
clone in interface CloneableSerializable
Overrides:
clone in class AbstractRandomized
Returns:
A clone of this object.

computeParameterGradient

public Matrix computeParameterGradient(Vector input)
Description copied from interface: GradientDescendable
Computes the derivative of the function about the input with respect to the parameters of the function

Specified by:
computeParameterGradient in interface GradientDescendable
Specified by:
computeParameterGradient in interface ParameterGradientEvaluator<Vector,Matrix>
Parameters:
input - Point about which to differentiate w.r.t. the parameters
Returns:
Matrix of parameter gradients

convertToVector

public Vector convertToVector()
Description copied from interface: Vectorizable
Converts the object to a vector.

Specified by:
convertToVector in interface Vectorizable
Returns:
The Vector form of the object.

getNumParameters

public int getNumParameters()
Gets the total number of parameters in the neural net, including the bias weights.

Returns:
Total number of parameters in the neural net.

convertFromVector

public void convertFromVector(Vector parameters)
Description copied from interface: Vectorizable
Converts the object from a Vector of parameters.

Specified by:
convertFromVector in interface Vectorizable
Parameters:
parameters - The parameters to incorporate.

evaluate

public Vector evaluate(Vector input)
Description copied from interface: Evaluator
Evaluates the function on the given input and returns the output.

Specified by:
evaluate in interface Evaluator<Vector,Vector>
Parameters:
input - The input to evaluate.
Returns:
The output produced by evaluating the input.

evaluateHiddenLayerActivation

protected Vector evaluateHiddenLayerActivation(Vector input)
Computes the raw (unsquashed) activation at the hidden layer for the given input.

Parameters:
input - Input to compute the raw hidden activation of.
Returns:
Raw (unsquashed) activation at the hidden layer.

evaluateSquashedHiddenLayerActivation

protected Vector evaluateSquashedHiddenLayerActivation(Vector hiddenActivation)
Evaluates the squashed hidden-layer activation from its raw activation value. This is equivalent to apply an element-wise squashing function to the raw hidden activation values.

Parameters:
hiddenActivation - Raw (unsquashed) hidden activation values.
Returns:
Squashed hidden-layer activation.

evaluateOutputFromSquashedHiddenLayerActivation

protected Vector evaluateOutputFromSquashedHiddenLayerActivation(Vector squashedHiddenActivation)
Evaluates the output from the squashed hidden-layer activation.

Parameters:
squashedHiddenActivation - Squashed hidden-layer activation.
Returns:
Output of the neural net.

reinitializeWeights

public void reinitializeWeights()
Reinitializes the neural network parameters based on its current setup. It uses the object's internal random number generator to generate weights uniformly in the range of [-initializationRange, +initializationRange].


initializeWeights

public void initializeWeights(int inputDimensionality,
                              int hiddenDimensionality,
                              int outputDimensionality)
Initializes the neural net parameters for the given dimensions, not including the bias terms, using the object's random-number generator uniformly between the initialization range (and its negative value).

Parameters:
inputDimensionality - Number of the inputs, not including the bias term, must be greater than zero.
hiddenDimensionality - Number of the hidden units, not including the bias term, must be greater than zero.
outputDimensionality - Number of the outputs, must be greater than zero.

getOutputDimensionality

public int getOutputDimensionality()
Description copied from interface: VectorOutputEvaluator
Gets the expected dimensionality of the output vector of the evaluator, if it is known. If it is not known, -1 is returned.

Specified by:
getOutputDimensionality in interface VectorOutputEvaluator<Vector,Vector>
Returns:
The expected dimensionality of the output vector of the evaluator, or -1 if it is not known.

setOutputDimensionality

public void setOutputDimensionality(int outputDimensionality)
Sets the output dimensionality of the neural net by re-initializing the weights.

Parameters:
outputDimensionality - Desired output dimensionality, must be greater than zero.

getHiddenDimensionality

public int getHiddenDimensionality()
Gets the number of hidden units, not including the bias term.

Returns:
Number of hidden units, must be greater than zero.

setHiddenDimensionality

public void setHiddenDimensionality(int hiddenDimensionality)
Sets the number of hidden units, not including the bias term, by re-initializing the neural net's weights.

Parameters:
hiddenDimensionality - Number of hidden units, must be greater than zero.

getInputDimensionality

public int getInputDimensionality()
Description copied from interface: VectorInputEvaluator
Gets the expected dimensionality of the input vector to the evaluator, if it is known. If it is not known, -1 is returned.

Specified by:
getInputDimensionality in interface VectorInputEvaluator<Vector,Vector>
Returns:
The expected dimensionality of the input vector to the evaluator, or -1 if it is not known.

setInputDimensionality

public void setInputDimensionality(int inputDimensionality)
Sets the number of input units, not counting the bias term, by re-initializing the neural net's parameters.

Parameters:
inputDimensionality - Desired input dimensionality, must be greater than zero.

getSquashingFunction

public DifferentiableUnivariateScalarFunction getSquashingFunction()
Getter for squashingFunction

Returns:
Squashing function to apply at the hidden layer.

setSquashingFunction

public void setSquashingFunction(DifferentiableUnivariateScalarFunction squashingFunction)
Setter for squashingFunction

Parameters:
squashingFunction - Squashing function to apply at the hidden layer.

getInitializationRange

public double getInitializationRange()
Getter for initializationRange

Returns:
Range of values to initialize the weights between, must be greater than or equal to zero.

setInitializationRange

public void setInitializationRange(double initializationRange)
Setter for initializationRange

Parameters:
initializationRange - Range of values to initialize the weights between, must be greater than or equal to zero.