Grid Search

Given a set of possible hyperparameter values, the function trains models with all the possible combinations of hyperparameters.

Usage

gridSearch(
  model,
  hypers,
  metric,
  test = NULL,
  env = NULL,
  save_models = TRUE,
  interactive = TRUE,
  progress = TRUE
)

Arguments

model: SDMmodel or SDMmodelCV object.
hypers: named list containing the values of the hyperparameters that should be tuned, see details.
metric: character. The metric used to evaluate the models, possible values are: "auc", "tss" and "aicc".
test: SWD object. Testing dataset used to evaluate the model, not used with aicc and SDMmodelCV objects.
env: rast containing the environmental variables, used only with "aicc".
save_models: logical. If FALSE the models are not saved and the output contains only a data frame with the metric values for each hyperparameter combination. Set it to FALSE when there are many combinations to avoid R crashing for memory overload.
interactive: logical. If FALSE the interactive chart is not created.
progress: logical. If TRUE shows a progress bar.

Value

SDMtune object.

Details

To know which hyperparameters can be tuned you can use the output of the function getTunableArgs. Hyperparameters not included in the hypers argument take the value that they have in the passed model.

An interactive chart showing in real-time the steps performed by the algorithm is displayed in the Viewer pane.

Author

Sergio Vignali

Examples

# Acquire environmental variables
files <- list.files(path = file.path(system.file(package = "dismo"), "ex"),
                    pattern = "grd",
                    full.names = TRUE)

predictors <- terra::rast(files)

# Prepare presence and background locations
p_coords <- virtualSp$presence
bg_coords <- virtualSp$background

# Create SWD object
data <- prepareSWD(species = "Virtual species",
                   p = p_coords,
                   a = bg_coords,
                   env = predictors,
                   categorical = "biome")
#> ℹ Extracting predictor information for presence locations
#> ✔ Extracting predictor information for presence locations [35ms]
#> 
#> ℹ Extracting predictor information for absence/background locations
#> ✔ Extracting predictor information for absence/background locations [63ms]
#> 

# Split presence locations in training (80%) and testing (20%) datasets
datasets <- trainValTest(data,
                         test = 0.2,
                         only_presence = TRUE)
train <- datasets[[1]]
test <- datasets[[2]]

# Train a model
model <- train(method = "Maxnet",
               data = train,
               fc = "l")

# Define the hyperparameters to test
h <- list(reg = 1:2,
          fc = c("lqp", "lqph"))

# Run the function using the AUC as metric
output <- gridSearch(model,
                     hypers = h,
                     metric = "auc",
                     test = test)
#> Grid Search  ■■■■■■■■■■■■■                     40% | ETA:  5s - 00:00:3.4
#> Grid Search  ■■■■■■■■■■■■■■■■■■■■■■■■■         80% | ETA:  2s - 00:00:6.3
#> Grid Search  ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■  100% | ETA:  0s - 00:00:6.3
#> 
output@results
#>     fc reg train_AUC  test_AUC   diff_AUC
#> 1  lqp   1 0.8647428 0.8508125 0.01393031
#> 2 lqph   1 0.8746716 0.8490475 0.02562406
#> 3  lqp   2 0.8603672 0.8453250 0.01504219
#> 4 lqph   2 0.8674672 0.8482200 0.01924719
output@models
#> [[1]]
#> 
#> ── Object of class: <SDMmodel> ──
#> 
#> Method: Maxnet
#> 
#> ── Hyperparameters 
#> • fc: "lqp"
#> • reg: 1
#> 
#> ── Info 
#> • Species: Virtual species
#> • Presence locations: 320
#> • Absence locations: 5000
#> 
#> ── Variables 
#> • Continuous: "bio1", "bio12", "bio16", "bio17", "bio5", "bio6", "bio7", and
#> "bio8"
#> • Categorical: "biome"
#> 
#> [[2]]
#> 
#> ── Object of class: <SDMmodel> ──
#> 
#> Method: Maxnet
#> 
#> ── Hyperparameters 
#> • fc: "lqph"
#> • reg: 1
#> 
#> ── Info 
#> • Species: Virtual species
#> • Presence locations: 320
#> • Absence locations: 5000
#> 
#> ── Variables 
#> • Continuous: "bio1", "bio12", "bio16", "bio17", "bio5", "bio6", "bio7", and
#> "bio8"
#> • Categorical: "biome"
#> 
#> [[3]]
#> 
#> ── Object of class: <SDMmodel> ──
#> 
#> Method: Maxnet
#> 
#> ── Hyperparameters 
#> • fc: "lqp"
#> • reg: 2
#> 
#> ── Info 
#> • Species: Virtual species
#> • Presence locations: 320
#> • Absence locations: 5000
#> 
#> ── Variables 
#> • Continuous: "bio1", "bio12", "bio16", "bio17", "bio5", "bio6", "bio7", and
#> "bio8"
#> • Categorical: "biome"
#> 
#> [[4]]
#> 
#> ── Object of class: <SDMmodel> ──
#> 
#> Method: Maxnet
#> 
#> ── Hyperparameters 
#> • fc: "lqph"
#> • reg: 2
#> 
#> ── Info 
#> • Species: Virtual species
#> • Presence locations: 320
#> • Absence locations: 5000
#> 
#> ── Variables 
#> • Continuous: "bio1", "bio12", "bio16", "bio17", "bio5", "bio6", "bio7", and
#> "bio8"
#> • Categorical: "biome"
#> 

# Order results by highest test AUC
output@results[order(-output@results$test_AUC), ]
#>     fc reg train_AUC  test_AUC   diff_AUC
#> 1  lqp   1 0.8647428 0.8508125 0.01393031
#> 2 lqph   1 0.8746716 0.8490475 0.02562406
#> 4 lqph   2 0.8674672 0.8482200 0.01924719
#> 3  lqp   2 0.8603672 0.8453250 0.01504219

# Run the function using the AICc as metric and without saving the trained
# models, helpful when numerous hyperparameters are tested to avoid memory
# problems
output <- gridSearch(model,
                     hypers = h,
                     metric = "aicc",
                     env = predictors,
                     save_models = FALSE)
#> Grid Search  ■■■■■■■■■■■■■                     40% | ETA:  6s - 00:00:04
#> Grid Search  ■■■■■■■■■■■■■■■■■■■               60% | ETA:  3s - 00:00:4.6
#> Grid Search  ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■  100% | ETA:  0s - 00:00:07
#> 
output@results
#>     fc reg     AICc delta_AICc
#> 1  lqp   1 5264.961   20.57583
#> 2 lqph   1 5272.979   28.59386
#> 3  lqp   2 5279.299   34.91364
#> 4 lqph   2 5244.385    0.00000