Catch Errors

Error catching is an important thing to consider when creating Monte Carlo simulations. Sometimes, iterative algorithms will ‘fail to converge’ or crash for other reasons (e.g., sparse data). However, SimDesign makes this process much easier because the internal functions are wrapped in a try block already, therefore the simulation will not terminate unexpectedly. Below we demonstrate what happens when errors are thrown and caught, and how this information is tracked in the returned object.

Define the functions

As usual, define the functions of interest.

library(SimDesign)
# SimFunctions(comments=FALSE)

Design <- data.frame(N = c(10,20,30))

Generate <- function(condition, fixed_objects = NULL) {
    ret <- with(condition, rnorm(N))
    ret
}

Analyse <- function(condition, dat, fixed_objects = NULL) {
    whc <- sample(c(0,1,2,3), 1, prob = c(.7, .20, .05, .05))
    if(whc == 0){
       ret <- mean(dat)
    } else if(whc == 1){
        ret <- t.test() # missing arguments
    } else if(whc == 2){
        ret <- t.test('invalid') # invalid arguments
    } else if(whc == 3){
        # throw error manually 
        stop('Manual error thrown') 
    }
    # manual warnings
    if(sample(c(TRUE, FALSE), 1, prob = c(.1, .9)))
        warning('This warning happens rarely')
    if(sample(c(TRUE, FALSE), 1, prob = c(.5, .5)))
        warning('This warning happens much more often')
    ret
}

Summarise <- function(condition, results, fixed_objects = NULL) {
    ret <- c(bias = bias(results, 0))
    ret
}

The above simulation is just an example of how errors are tracked in SimDesign, as well as how to throw a manual error in case the data should be re-drawn (e.g., when a model converges but fails to do so before some number of predefined iterations).

Run the simulation

result <- runSimulation(Design, replications = 100, 
                       generate=Generate, analyse=Analyse, summarise=Summarise)

## 
Design row: 1/3;   Started: Sun Oct 29 19:05:21 2017;   Total elapsed time: 0.00s 
## 
Design row: 2/3;   Started: Sun Oct 29 19:05:22 2017;   Total elapsed time: 0.06s 
## 
Design row: 3/3;   Started: Sun Oct 29 19:05:22 2017;   Total elapsed time: 0.10s

print(result)

##    N     bias ERROR: .Error : Manual error thrown\n
## 1 10 -0.03091                                     6
## 2 20 -0.01525                                     9
## 3 30 -0.01007                                     4
##   ERROR: .Error in t.test.default("invalid") : not enough 'x' observations\n
## 1                                                                          7
## 2                                                                         10
## 3                                                                          4
##   ERROR: .Error in t.test.default() : argument "x" is missing, with no default\n
## 1                                                                             32
## 2                                                                             29
## 3                                                                             25
##   WARNING: .This warning happens much more often
## 1                                             49
## 2                                             44
## 3                                             51
##   WARNING: .This warning happens rarely REPLICATIONS SIM_TIME
## 1                                    10          100    0.06s
## 2                                    12          100    0.05s
## 3                                    14          100    0.04s
##                  COMPLETED
## 1 Sun Oct 29 19:05:22 2017
## 2 Sun Oct 29 19:05:22 2017
## 3 Sun Oct 29 19:05:22 2017

What you’ll immediately notice from this output object is that the name of the error/warning thrown, and the function from which the error was thrown, are included as additional columns in the output with the prefix ERROR:. Furthermore, the frequency in which the error occurred are also included for each design condition (here the t.test.default() error, where no inputs were supplied, occurred more often than the manually thrown error as well as the invalid-input error). This behaviour is also tracked for WARNING messages as well in case there are clues as to why estimation models are having difficulty (or for other reasons whereby the warnings may be more serious).

Finally, SimDesign has a safety built-in feature controlled by with max_errors argument to avoid getting stuck in infinite loops. By default if more than 50 errors are consecutively returned then the simulation will be halted and the final error message will be returned. This safety feature is built-in because too many consecutive stop() calls generally indicates a major problem in the simulation code.

What to do

If errors occur too often then these design conditions should either be extracted out of the simulation or further inspected to determine if they can be fixed (e.g., providing better starting values, increasing convergence criteria/number of iterations, etc). The use of the debugging features may be useful to track down issues here as well. For example, wrap the problematic objects in a try() call and add the line if(is(object, 'try-error')) browser() to jump into the location/replication where the object unexpectedly witnessed an error, and attempt to discern why the error occurred and how to fix it.

Catching errors

Phil Chalmers

2017-10-29

Catch Errors

Define the functions

Run the simulation

What to do