Validation statistics requirements

From Intamap

This page gives the requirements for the validation measures we will apply to the test data as a simple list.

Some terms:

• training set: a portion of the full data set, used to directly training the model parameters.
• validation set: a portion of the full data set, withheld from training the model to estimate hyper-parameters in the model.
• test set: a portion of the full data set, withheld from training the model and hyper-parameters, used to assess the performance of the model on unseen data.
• hyper-parameters: not the Bayesian definition (these are all parameters) but some higher level parameters than cannot be easily estimated from the data, for example the number of hidden units in a neural network, the choice of covariance function in a GP (but not the estimation of parameters in the covariance function).

Validation measure requirements - things we want to compute to compare performance on the test data sets

• Accuracy measures on the mean:

1. Mean Error (bias)
2. Mean Square Error (variance)

• Accuracy measures on the marginal variance:

1. The thing people do comparing the kriging variance with the difference in the mean vs observation

• Accuracy measures on the distribution:

1. Malhalanobis distance
2. ROC curves for given thresholds
3. Area under the ROC curves

Issues we think it is important the validation plan should address (see also Test data requirements)

• Careful design of training, test and validation sets; their specification for fair comparison

Back to System Requirements.