Skip to contents

Validate a constructed GP, DGP, or linked (D)GP emulator

Source: R/validation.R
validate.Rd

This function validate a constructed GP, DGP, or linked (D)GP emulator via the Leave-One-Out (LOO) cross validation or Out-Of-Sample (OOS) validation.

Usage

validate(object, x_test, y_test, method, verb, force, cores, ...)

# S3 method for gp
validate(
  object,
  x_test = NULL,
  y_test = NULL,
  method = "mean_var",
  verb = TRUE,
  force = FALSE,
  cores = 1,
  ...
)

# S3 method for dgp
validate(
  object,
  x_test = NULL,
  y_test = NULL,
  method = "mean_var",
  verb = TRUE,
  force = FALSE,
  cores = 1,
  threading = FALSE,
  ...
)

# S3 method for lgp
validate(
  object,
  x_test = NULL,
  y_test = NULL,
  method = "mean_var",
  verb = TRUE,
  force = FALSE,
  cores = 1,
  threading = FALSE,
  ...
)

Arguments

object

can be one of the following:

  • the S3 class gp.

  • the S3 class dgp.

  • the S3 class lgp.

x_test

the OOS testing input data:

  • if x is an instance of the gp or dgp class, x_test is a matrix where each row is an input testing data point and each column is an input dimension.

  • if x is an instance of the lgp class, x_test can be a matrix or a list:

    • if x_test is a matrix, it is the global testing input data that feed into the emulators in the first layer of a system. The rows of x_test represent different input data points and the columns represent input dimensions across all emulators in the first layer of the system. In this case, it is assumed that the only global input to the system is the input to the emulators in the first layer and there is no global input to emulators in other layers.

    • if x_test is a list, it should have L (the number of layers in an emulator system) elements. The first element is a matrix that represents the global testing input data that feed into the emulators in the first layer of the system. The remaining L-1 elements are L-1 sub-lists, each of which contains a number (the same number of emulators in the corresponding layer) of matrices (rows being testing input data points and columns being input dimensions) that represent the global testing input data to the emulators in the corresponding layer. The matrices must be placed in the sub-lists based on how their corresponding emulators are placed in struc argument of lgp(). If there is no global input data to a certain emulator, set NULL in the corresponding sub-list of x_test.

x_test must be provided for the validation if x is an instance of the lgp. Defaults to NULL.

y_test

the OOS testing output data that correspond to x_test:

  • if x is an instance of the gp class, y_test is a matrix with only one column and each row being an testing output data point.

  • if x is an instance of the dgp class, y_test is a matrix with its rows being testing output data points and columns being output dimensions.

  • if x is an instance of the lgp class, y_test can be a single matrix or a list of matrices:

    • if y_test is a single matrix, then there is only one emulator in the final layer of the linked emulator system and y_test represents the emulator's output with rows being testing positions and columns being output dimensions.

    • if y_test is a list, then y_test should have M number (the same number of emulators in the final layer of the system) of matrices. Each matrix has its rows corresponding to testing positions and columns corresponding to output dimensions of the associated emulator in the final layer.

y_test must be provided for the validation if x is an instance of the lgp. Defaults to NULL.

method

the prediction approach in validations: mean-variance ("mean_var") or sampling ("sampling") approach. Defaults to "mean_var".

verb

a bool indicating if the trace information on validations will be printed during the function execution. Defaults to TRUE.

force

a bool indicating whether to force the LOO or OOS re-evaluation when loo or oos slot already exists in object. When force = FALSE, validate() will try to determine automatically if the LOO or OOS re-evaluation is needed. Set force to TRUE when LOO or OOS re-evaluation is required. Defaults to FALSE.

cores

the number of cores/workers to be used for the LOO or OOS validation. If set to NULL, the number of cores is set to (max physical cores available - 1). Defaults to 1.

...

N/A.

threading

a bool indicating whether to use the multi-threading to accelerate the LOO or OOS. Turning this option on could improve the speed of validations when the emulator is built with a moderately large number of training data points and the Matérn-2.5 kernel.

Value

  • If object is an instance of the gp class, an updated object is returned with an additional slot called loo (for LOO cross validation) or oos (for OOS validation) that contains:

    • two slots called x_train (or x_test) and y_train (or y_test) that contain the validation data points for LOO (or OOS).

    • a column matrix called mean, if method = "mean_var", or median, if method = "sampling", that contains the predictive means or medians of the GP emulator at validation positions.

    • three column matrices called std, lower, and upper that contain the predictive standard deviations and credible intervals of the GP emulator at validation positions. If method = "mean_var", the upper and lower bounds of a credible interval are two standard deviations above and below the predictive mean. If method = "sampling", the upper and lower bounds of a credible interval are 2.5th and 97.5th percentiles.

    • a numeric value called rmse that contains the root mean/median squared error of the GP emulator.

    • a numeric value called nrmse that contains the (min-max) normalized root mean/median squared error of the GP emulator. The min-max normalization is based on the maximum and minimum values of the validation outputs contained in y_train (or y_test).

    The rows of matrices (mean, median, std, lower, and upper) correspond to the validation positions.

  • If object is an instance of the dgp class, an updated object is returned with an additional slot called loo (for LOO cross validation) or oos (for OOS validation) that contains:

    • two slots called x_train (or x_test) and y_train (or y_test) that contain the validation data points for LOO (or OOS).

    • a matrix called mean, if method = "mean_var", or median, if method = "sampling", that contains the predictive means or medians of the DGP emulator at validation positions.

    • three matrices called std, lower, and upper that contain the predictive standard deviations and credible intervals of the DGP emulator at validation positions. If method = "mean_var", the upper and lower bounds of a credible interval are two standard deviations above and below the predictive mean. If method = "sampling", the upper and lower bounds of a credible interval are 2.5th and 97.5th percentiles.

    • a vector called rmse that contains the root mean/median squared errors of the DGP emulator across different output dimensions.

    • a vector called nrmse that contains the (min-max) normalized root mean/median squared errors of the DGP emulator across different output dimensions. The min-max normalization is based on the maximum and minimum values of the validation outputs contained in y_train (or y_test).

    The rows and columns of matrices (mean, median, std, lower, and upper) correspond to the validation positions and DGP emulator output dimensions, respectively.

  • If object is an instance of the lgp class, an updated object is returned with an additional slot called oos (for OOS validation) that contains:

    • two slots called x_test and y_test that contain the validation data points for OOS.

    • a list called mean, if method = "mean_var", or median, if method = "sampling", that contains the predictive means or medians of the linked (D)GP emulator at validation positions.

    • three lists called std, lower, and upper that contain the predictive standard deviations and credible intervals of the linked (D)GP emulator at validation positions. If method = "mean_var", the upper and lower bounds of a credible interval are two standard deviations above and below the predictive mean. If method = "sampling", the upper and lower bounds of a credible interval are 2.5th and 97.5th percentiles.

    • a list called rmse that contains the root mean/median squared errors of the linked (D)GP emulator.

    • a list called nrmse that contains the (min-max) normalized root mean/median squared errors of the linked (D)GP emulator. The min-max normalization is based on the maximum and minimum values of the validation outputs contained in y_test.

    Each element in mean, median, std, lower, upper, rmse, and nrmse corresponds to a (D)GP emulator in the final layer of the linked (D)GP emulator.

Details

See further examples and tutorials at https://mingdeyu.github.io/dgpsi-R/.

Note

  • When both x_test and y_test are NULL, the LOO cross validation will be implemented. Otherwise, OOS validation will be implemented. The LOO validation is only applicable to a GP or DGP emulator (i.e., x is an instance of the gp or dgp class). If a linked (D)GP emulator (i.e., x is an instance of the lgp class) is provided, x_test and y_test must also be provided for OOS validation.

  • Any R vector detected in x_test and y_test will be treated as a column vector and automatically converted into a single-column R matrix. Thus, if x_test or y_test is a single testing data point with multiple dimensions, it must be given as a matrix.

Examples

if (FALSE) {

# See gp(), dgp(), or lgp() for an example.
}