Intro

In the previous articles you have learned the core functions of SDMtune and how to perform data-driven variable selection. In this article you will learn how to tune the model hyperparameters.

Training, validation and testing split

When you tune the model hyperparameters you iteratively adjust the hyperparameters while monitoring the changes in the evaluation metric computed using the testing dataset. In this process, the information contained in the testing dataset leaks in the model and therefore, at the end of the process, the testing dataset doesn’t represent anymore an independent set to evaluate the model (Müller and Guido 2016, @Chollet2018). A better strategy, than splitting the observation locations in training and testing, would be to split them into training, validation and testing datasets. The training dataset is then used to train the model, the validation datasets to drive the hyperparameter tuning and the testing dataset to evaluate the tuned model. The function trainValTest() allows to split the data in three folds containing the provided percentage of data. For illustration purpose let’s split the presence locations in training (60%), validation (20%) and testing (20%) datasets:

library(zeallot)  # For unpacking assignment
c(train, val, test) %<-% trainValTest(data, val = 0.2, test = 0.2, only_presence = TRUE, seed = 61516)
cat("# Training  : ", nrow(train@data))
#> # Training  :  5240
cat("# Validation: ", nrow(val@data))
#> # Validation:  5080
cat("# Testing   : ", nrow(test@data))
#> # Testing   :  5080

We now train a Maxnet model with default settings and using the training dataset:

model <- train("Maxnet", data = train)

Another approach would be to split the data in two folds: training and testing, use the cross validation strategy with the training dataset to tune the model hyperparameters, and evaluate the tuned model with the unseen held apart testing dataset. For execution time reason we demonstrate the first approach but you are free to try out the second one.

Check the effect of varying one hyperparameter

To see the effect of varying one hyperparameter on the model performance we can use the function gridSearch(). The function iterates through a set of predefined hyperparameter values, train the model and displays in real-time the evaluation metric in the RStudio viewer pane (hover over the points to get a tooltip with extra information). Let’s see how the AUC changes varying the regularization multiplier. First we have to define the values for the hyperparameter that we want to test. For that we create a named list that we will use as an argument for the function gridSearch():

# Define the values for bg
h <- list(reg = seq(0.2, 1, 0.1))
# Call the gridSearch function
exp_1 <- gridSearch(model, hypers = h, metric = "auc", test = val)

As you noticed we used the validation dataset as test argument. The output of the function is an object of class SDMtune(). Let’s print it:

exp_1
#> Object of class:  SDMtune 
#> 
#> Models configuration:
#> --------------------
#> fc: lqph
#> reg: 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1

When you print the output, the text contains the models configuration that have been used during the execution of the function. In our case, only the regularization multiplier reg has multiple values. You can plot the SDMtune() object:

plot(exp_1, title = "Experiment 1")

and you can also recreate the interactive chart using:

plot(exp_1, title = "Experiment 1", interactive = TRUE)

The SDMtune() object stores the results in the slot @results:

exp_1@results
#>     fc reg train_AUC  test_AUC   diff_AUC
#> 1 lqph 0.2 0.9008033 0.8235925 0.07721083
#> 2 lqph 0.3 0.8988542 0.8268050 0.07204917
#> 3 lqph 0.4 0.8972125 0.8300300 0.06718250
#> 4 lqph 0.5 0.8953675 0.8325025 0.06286500
#> 5 lqph 0.6 0.8936800 0.8350225 0.05865750
#> 6 lqph 0.7 0.8918142 0.8370500 0.05476417
#> 7 lqph 0.8 0.8897492 0.8388800 0.05086917
#> 8 lqph 0.9 0.8883683 0.8389875 0.04938083
#> 9 lqph 1.0 0.8875433 0.8395025 0.04804083

You can order them with:

exp_1@results[order(-exp_1@results$test_AUC), ]
#>     fc reg train_AUC  test_AUC   diff_AUC
#> 9 lqph 1.0 0.8875433 0.8395025 0.04804083
#> 8 lqph 0.9 0.8883683 0.8389875 0.04938083
#> 7 lqph 0.8 0.8897492 0.8388800 0.05086917
#> 6 lqph 0.7 0.8918142 0.8370500 0.05476417
#> 5 lqph 0.6 0.8936800 0.8350225 0.05865750
#> 4 lqph 0.5 0.8953675 0.8325025 0.06286500
#> 3 lqph 0.4 0.8972125 0.8300300 0.06718250
#> 2 lqph 0.3 0.8988542 0.8268050 0.07204917
#> 1 lqph 0.2 0.9008033 0.8235925 0.07721083

Try yourself

Try to see how TSS changes varying the regularization multiplier from 1 to 4 (highlight to see the solution):

# Define the values for reg
h <- list(reg = 1:4)
# Call the gridSearch function
exp_2 <- gridSearch(model, hypers = h, metric = "tss", test = val)

and how AUC changes varying the feature combinations using the following values: l, lq, lh, lqp, lqph and lqpht (highlight to see the solution):

# Define the values for fc
h <- list(fc = c("l", "lq", "lh", "lqp", "lqph", "lqpht"))
# Call the gridSearch function
exp_3 <- gridSearch(model, hypers = h, metric = "auc", test = val)

Train a Maxent model and see how the AUC changes varying the number of iterations from 300 to 1100 with increments of 200 (highlight to see the solution):

maxent_model <- train("Maxent", data = data)
# Define the values for fc
h <- list("iter" = seq(300, 1100, 200))
# Call the gridSearch function
exp_4 <- gridSearch(maxent_model, hypers = h, metric = "auc", test = val)

To see which hyperparameters can be tuned in a given model use the function getTunableArgs(). For example:

getTunableArgs(model)
#> [1] "fc"  "reg"

Tune hyperparameters

To tune the model hyperparameters you should run all the possible combinations of hyperparameters. Here is an example using combinations of regularization multiplier and feature classes:

h <- list(reg = seq(0.2, 2, 0.2), fc = c("l", "lq", "lh", "lqp", "lqph", "lqpht"))
exp_5 <- gridSearch(model, hypers = h, metric = "auc", test = val)

This code takes already quite long as it has to train 60 models. Imagine if you want to check more values for the regularization multiplier and maybe add the number of iterations (in the case of a Maxent model). The number of models to be trained increases exponentially and consequently the execution time. In the next two paragraphs we will present two possible alternative to the gridSearch() function.

Optimize Model

The previous function doesn’t learn anything from the trained models, it just selects n random combinations of hyperparameters. The function optimizeModel() uses a genetic algorithm to find an optimum or near optimum solution. Check the function documentation to understand how it works, here we provide the code to execute it:

exp_7 <- optimizeModel(model, hypers = h, metric = "auc", test = val, pop = 15, gen = 2, seed = 798)

Evaluate final model

Let’s say we want to use the best tuned model found by the randomSearch() function. Before evaluating the model using the testing dataset, we can merge the training and the validation datasets together to increase the number of locations and train a new model with the merged observations and the tuned configuration. At this point we may have removed variables using the varSel() or reduceVar() function. If this is the case, we cannot merge directly the initial datasets which contain all the environmental variables. We can extract the train dataset with the selected variables from the output of the experiment and merge it with the validation dataset using the function mergeSWD():

index <- which.max(exp_6@results$test_AUC)  # Index of the best model in the experiment
new_train <- exp_6@models[[index]]@data  # New train dataset containing only the selected variables
merged_data <- mergeSWD(new_train, val, only_presence = TRUE) # Merge only presence data

The val dataset contains all the initial environmental variables but the mergeSWD() function will merge only those that are present in both datasets (in case you have performed variable selection).

Then we get the model configuration from the experiment 6:

head(exp_6@results)
#>     fc reg train_AUC  test_AUC   diff_AUC
#> 1   lp 2.2 0.8691417 0.8482400 0.02090167
#> 2   lp 0.8 0.8734875 0.8472100 0.02627750
#> 3 lqph 3.2 0.8713750 0.8435500 0.02782500
#> 4 lqph 2.4 0.8764117 0.8435025 0.03290917
#> 5   lq 3.8 0.8597967 0.8419325 0.01786417
#> 6   lh 3.2 0.8698442 0.8414150 0.02842917

The best model is at row 1 and was trained using lp feature class combination and 2.2 as regularization multiplier:

final_model <- train("Maxnet", data = merged_data, fc = exp_6@results[index, 1], reg = exp_6@results[index, 2])

Now we can evaluate the final model using the held apart testing dataset:

auc(final_model, test = test)
#> [1] 0.8325913

Hyperparameters tuning with cross validation

Another approach would be to split the data in two folds: training and testing, use the cross validation strategy with the training dataset to tune the model hyperparameters, and evaluate the tuned model with the unseen held apart testing dataset.

# Create the folds from the training dataset
folds <- randomFolds(train, k = 4, only_presence = TRUE, seed = 25)
# Train the model
cv_model <- train("Maxent", data = train, folds = folds)

All the previous examples can be applied to the cross validation, here an example with randomSearch (note that in this case the testing dataset is not provided as is taken from the folds stored in the SDMmodelCV):

h <- list(reg = seq(0.2, 5, 0.2), fc = c("l", "lq", "lh", "lp", "lqp", "lqph"))
exp_8 <- randomSearch(cv_model, hypers = h, metric = "auc", pop = 10, seed = 65466)

The function randomSearch orders the models according to the best value of the metric for the testing dataset. In this case we can train a model using the best hyperparameters configuration and without cross validation with:

final_model <- train("Maxent", data = exp_8@models[[1]]@data, fc = exp_8@results[1, 1], reg = exp_8@results[1, 2])
auc(final_model, test = test)

Conclusion

In this article you have learned:

  • the training/validation/testing evaluation strategy;
  • how to explore the effect of changing one model hyperparameter at time;
  • how to tune the model hyperparameters using three different functions;
  • how to merge two SWD() objects;
  • how to evaluate a final model using the held apart testing dataset.

References

Chollet, François, and J. J. Allaire. 2018. Deep learning with R. 1st ed. Manning Publications Co.

Müller, Andreas C., and Sarah Guido. 2016. Introduction to machine learning with Python : a guide for data scientists.