[TEST] Statistical unit tests about rmo HOT 2 CLOSED

On another thought, we might want to keep the p-value threshold as high as possible. To do this, we have to limit the cases on which a statistical unit test is run. Hence, these should be only run on very few tests or they should be skipped on most without going into a special debug mode.

from rmo.

Number of Parametrizations

In theory, all algorithms should be tested for their statistical quality. However, here, some parametrizations mights be different enough such that the algorithms should be tested on both of them. A Lévy frailty model with Exponential jumps might be very different from one with Pareto jumps — even one with small Exponential jumps might be very different from one with large Exponential jumps.

A simple summary of all suitable parametrizations (not taking into account specific values) gives:

• ESM: 9+
• Arnold: 9+
• Ex. Arnold: 9+
• Cuadras-Augé: 1
• LFM: (2 x 3)+
• Total: ~35+

If, in the future, the package is opened to other random number generators, this number could double or tripple.

Options for the threshold of p

1. Choose the threshold very small, e.g. 0.01% to make sure that even with 100 tests, the probability of a false positive is still as small as 1%.
2. Limit the number of total statistical tests to a small number, e.g. 10. Then we could choose the threshold as 0.1% to have the probability of a false positive of 1%.

Conclusion

Choosing the threshold is difficult and requires compromises. I think, the best option is use option 2, but choose the parametrizations and RNG engines carefully. More tests could be provided with a default skip to have better analysis capabilities.

from rmo.