gov.sandia.cognition.learning.experiment
Class LearnerComparisonExperiment<InputDataType,FoldDataType,LearnedType,StatisticType,SummaryType>

java.lang.Object
  extended by gov.sandia.cognition.util.AbstractCloneableSerializable
      extended by gov.sandia.cognition.learning.experiment.AbstractLearningExperiment
          extended by gov.sandia.cognition.learning.experiment.AbstractValidationFoldExperiment<InputDataType,FoldDataType>
              extended by gov.sandia.cognition.learning.experiment.LearnerComparisonExperiment<InputDataType,FoldDataType,LearnedType,StatisticType,SummaryType>
Type Parameters:
InputDataType - The type of the data to perform the experiment with. This will be passed to the fold creator to create a number of folds on which to validate the performance of the learning algorithms.
FoldDataType - The type of data created by the fold creator that will go into the learning algorithms. Typically, this is the same as the InputDataType, but it does not need to be. It just needs to match the output of the fold creator and the input of the learning algorithms.
LearnedType - The type of the output produced by the learning algorithms whose performance will be evaluated on each fold of data.
StatisticType - The type of the statistic generated by the performance evaluator on each learned objects for each fold. It is created by passing the learned object plus the test data for the fold into the performance evaluator.
SummaryType - The type produced by the summarizer at the end of the experiment from a collection of the given statistics (one for each fold). This represents the performance result for the comparison of the performance of the learning algorithms for the whole experiment.
All Implemented Interfaces:
LearningExperiment, CloneableSerializable, Serializable, Cloneable
Direct Known Subclasses:
SupervisedLearnerComparisonExperiment

public class LearnerComparisonExperiment<InputDataType,FoldDataType,LearnedType,StatisticType,SummaryType>
extends AbstractValidationFoldExperiment<InputDataType,FoldDataType>
implements Serializable

The LearnerComparisonExperiment compares the performance of two machine learning algorithms to determine (using a statistical test) if the two algorithms have significantly different performance.

Since:
2.0
Author:
Justin Basilico
See Also:
Serialized Form

Nested Class Summary
static class LearnerComparisonExperiment.Result<SummaryType>
          Encapsulates the results of the comparison experiment.
 
Field Summary
protected  ConfidenceStatistic confidence
          The confidence statistic generated from the underlying performance statistics.
protected  Pair<BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>> learners
          The learners that the experiment is being performed on.
protected  PerformanceEvaluator<? super LearnedType,Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator
          The evaluator to use to compute the performance of the learned object on each fold.
protected  DefaultPair<ArrayList<StatisticType>,ArrayList<StatisticType>> statistics
          The performance evaluations made during the experiment.
protected  DefaultPair<SummaryType,SummaryType> summaries
          The summaries of performance.
protected  Summarizer<? super StatisticType,? extends SummaryType> summarizer
          The summarizer for summarizing the result of the performance evaluator from all the folds.
 
Fields inherited from class gov.sandia.cognition.learning.experiment.AbstractValidationFoldExperiment
foldCreator, numTrials
 
Fields inherited from class gov.sandia.cognition.learning.experiment.AbstractLearningExperiment
listeners
 
Constructor Summary
LearnerComparisonExperiment()
          Creates a new instance of LearnerComparisonExperiment.
LearnerComparisonExperiment(ValidationFoldCreator<InputDataType,FoldDataType> foldCreator, PerformanceEvaluator<? super LearnedType,Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator, NullHypothesisEvaluator<Collection<? extends StatisticType>> statisticalTest, Summarizer<? super StatisticType,? extends SummaryType> summarizer)
          Creates a new instance of LearnerComparisonExperiment.
 
Method Summary
 LearnerComparisonExperiment.Result<SummaryType> evaluate(BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType> learner1, BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType> learner2, Collection<? extends InputDataType> data)
          Evaluates the two batch learners using the given data on the same set of validation folds and returns the resulting information including the confidence statistic that the two are different along with the summary of their performance.
 LearnerComparisonExperiment.Result<SummaryType> evaluate(Pair<BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>> learners, Collection<? extends InputDataType> data)
          Evaluates the two batch learners using the given data on the same set of validation folds and returns the resulting information including the confidence statistic that the two are different along with the summary of their performance.
 ConfidenceStatistic getConfidence()
          Gets the confidence statistic that the two learners are different.
 Pair<BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>> getLearners()
          Gets the learners the experiment is being run on.
 PerformanceEvaluator<? super LearnedType,Collection<? extends FoldDataType>,? extends StatisticType> getPerformanceEvaluator()
          Gets the performance evaluator to apply to each fold.
 NullHypothesisEvaluator<Collection<? extends StatisticType>> getStatisticalTest()
          Gets the statistical test to use to determine if the two learners are significantly different.
 DefaultPair<ArrayList<StatisticType>,ArrayList<StatisticType>> getStatistics()
          Gets the performance evaluations for the trials of the experiment.
 DefaultPair<SummaryType,SummaryType> getSummaries()
          Gets the summaries of the experiment.
 Summarizer<? super StatisticType,? extends SummaryType> getSummarizer()
          Gets the summarizer of the performance evaluations.
protected  void runTrial(PartitionedDataset<FoldDataType> fold)
          Runs a single trial of the experiment on one fold of the data.
protected  void setConfidence(ConfidenceStatistic confidence)
          Sets the confidence statistic that the two learners are different.
protected  void setLearners(Pair<BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>> learners)
          Sets the learners the experiment is being run on.
 void setPerformanceEvaluator(PerformanceEvaluator<? super LearnedType,Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator)
          Sets the performance evaluator to apply to each fold.
 void setStatisticalTest(NullHypothesisEvaluator<Collection<? extends StatisticType>> statisticalTest)
          Sets the statistical test to use to determine if the two learners are significantly different.
protected  void setStatistics(DefaultPair<ArrayList<StatisticType>,ArrayList<StatisticType>> statistics)
          Sets the performance evaluations for the trials of the experiment.
protected  void setSummaries(DefaultPair<SummaryType,SummaryType> summaries)
          Sets the summaries of the experiment.
 void setSummarizer(Summarizer<? super StatisticType,? extends SummaryType> summarizer)
          Sets the summarizer of the performance evaluations.
 
Methods inherited from class gov.sandia.cognition.learning.experiment.AbstractValidationFoldExperiment
getFoldCreator, getNumTrials, runExperiment, setFoldCreator, setNumTrials
 
Methods inherited from class gov.sandia.cognition.learning.experiment.AbstractLearningExperiment
addListener, fireExperimentEnded, fireExperimentStarted, fireTrialEnded, fireTrialStarted, getListeners, removeListener, setListeners
 
Methods inherited from class gov.sandia.cognition.util.AbstractCloneableSerializable
clone
 
Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

performanceEvaluator

protected PerformanceEvaluator<? super LearnedType,Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator
The evaluator to use to compute the performance of the learned object on each fold.


summarizer

protected Summarizer<? super StatisticType,? extends SummaryType> summarizer
The summarizer for summarizing the result of the performance evaluator from all the folds.


learners

protected Pair<BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>> learners
The learners that the experiment is being performed on.


statistics

protected DefaultPair<ArrayList<StatisticType>,ArrayList<StatisticType>> statistics
The performance evaluations made during the experiment.


confidence

protected ConfidenceStatistic confidence
The confidence statistic generated from the underlying performance statistics.


summaries

protected DefaultPair<SummaryType,SummaryType> summaries
The summaries of performance.

Constructor Detail

LearnerComparisonExperiment

public LearnerComparisonExperiment()
Creates a new instance of LearnerComparisonExperiment.


LearnerComparisonExperiment

public LearnerComparisonExperiment(ValidationFoldCreator<InputDataType,FoldDataType> foldCreator,
                                   PerformanceEvaluator<? super LearnedType,Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator,
                                   NullHypothesisEvaluator<Collection<? extends StatisticType>> statisticalTest,
                                   Summarizer<? super StatisticType,? extends SummaryType> summarizer)
Creates a new instance of LearnerComparisonExperiment.

Parameters:
foldCreator - The object to use for creating the folds.
performanceEvaluator - The evaluator to use to compute the performance of the learned object on each fold.
statisticalTest - The statistical test to apply to the performance results of the two learners to determine if they are statistically different.
summarizer - The summarizer for summarizing the result of the performance evaluator from all the folds.
Method Detail

evaluate

public LearnerComparisonExperiment.Result<SummaryType> evaluate(Pair<BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>> learners,
                                                                Collection<? extends InputDataType> data)
Evaluates the two batch learners using the given data on the same set of validation folds and returns the resulting information including the confidence statistic that the two are different along with the summary of their performance.

Parameters:
learners - The two learners.
data - The data to use.
Returns:
The experimental results.

runTrial

protected void runTrial(PartitionedDataset<FoldDataType> fold)
Runs a single trial of the experiment on one fold of the data.

Specified by:
runTrial in class AbstractValidationFoldExperiment<InputDataType,FoldDataType>
Parameters:
fold - The fold to run the trial of the experiment on.

evaluate

public LearnerComparisonExperiment.Result<SummaryType> evaluate(BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType> learner1,
                                                                BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType> learner2,
                                                                Collection<? extends InputDataType> data)
Evaluates the two batch learners using the given data on the same set of validation folds and returns the resulting information including the confidence statistic that the two are different along with the summary of their performance.

Parameters:
learner1 - The first learner.
learner2 - The second learner.
data - The data to use.
Returns:
The experimental results.

getPerformanceEvaluator

public PerformanceEvaluator<? super LearnedType,Collection<? extends FoldDataType>,? extends StatisticType> getPerformanceEvaluator()
Gets the performance evaluator to apply to each fold.

Returns:
The performance evaluator to apply to each fold.

setPerformanceEvaluator

public void setPerformanceEvaluator(PerformanceEvaluator<? super LearnedType,Collection<? extends FoldDataType>,? extends StatisticType> performanceEvaluator)
Sets the performance evaluator to apply to each fold.

Parameters:
performanceEvaluator - The performance evaluator to apply to each fold.

getSummarizer

public Summarizer<? super StatisticType,? extends SummaryType> getSummarizer()
Gets the summarizer of the performance evaluations.

Returns:
The summarizer of the performance evaluations.

setSummarizer

public void setSummarizer(Summarizer<? super StatisticType,? extends SummaryType> summarizer)
Sets the summarizer of the performance evaluations.

Parameters:
summarizer - The summarizer of the performance evaluations.

getStatisticalTest

public NullHypothesisEvaluator<Collection<? extends StatisticType>> getStatisticalTest()
Gets the statistical test to use to determine if the two learners are significantly different.

Returns:
The statistical test.

setStatisticalTest

public void setStatisticalTest(NullHypothesisEvaluator<Collection<? extends StatisticType>> statisticalTest)
Sets the statistical test to use to determine if the two learners are significantly different.

Parameters:
statisticalTest - The statistical test.

getLearners

public Pair<BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>> getLearners()
Gets the learners the experiment is being run on.

Returns:
The learners.

setLearners

protected void setLearners(Pair<BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>,BatchLearner<? super Collection<? extends FoldDataType>,? extends LearnedType>> learners)
Sets the learners the experiment is being run on.

Parameters:
learners - The learners.

getStatistics

public DefaultPair<ArrayList<StatisticType>,ArrayList<StatisticType>> getStatistics()
Gets the performance evaluations for the trials of the experiment.

Returns:
The performance evaluations for the trials of the experiment.

setStatistics

protected void setStatistics(DefaultPair<ArrayList<StatisticType>,ArrayList<StatisticType>> statistics)
Sets the performance evaluations for the trials of the experiment.

Parameters:
statistics - The performance evaluations for the trials of the experiment.

getConfidence

public ConfidenceStatistic getConfidence()
Gets the confidence statistic that the two learners are different.

Returns:
The confidence statistic of the experiment.

setConfidence

protected void setConfidence(ConfidenceStatistic confidence)
Sets the confidence statistic that the two learners are different.

Parameters:
confidence - The confidence statistic of the experiment.

getSummaries

public DefaultPair<SummaryType,SummaryType> getSummaries()
Gets the summaries of the experiment.

Returns:
The summaries of the experiment.

setSummaries

protected void setSummaries(DefaultPair<SummaryType,SummaryType> summaries)
Sets the summaries of the experiment.

Parameters:
summaries - The summaries of the experiment.