### Abstract

attractive to analyze these outputs through sequential statistical methods such as sequential probability ratio tests (SPRTs). We investigate several SPRTs for choosing between two hypothesized values for the mean output (response). One SPRT is published in Wald (1945), and allows general distribution types. For a normal (Gaussian) distribution this SPRT assumes a known variance, but in our modified SPRT we estimate the variance. Another SPRT is published in Hall (1962), and assumes a normal distribution with an unknown variance estimated from a pilot sample. We also investigate a modification, replacing this pilot-sample estimator by a fully sequential estimator. We present a sequence of Monte

Carlo experiments for quantifying the performance of these SPRTs. In experiment #1 the simulation outputs are normal. This experiment suggests that Wald (1945)’s SPRT with estimated variance gives significantly high error rates. Hall (1962)’s original and modified SPRTs are conservative; i.e., the actual error rates are much smaller than the prespecified (nominal) rates. The most efficient SPRT is our modified Hall (1962) SPRT. In experiment #2 we examine the robustness of the various SPRTs in case of nonnormal output. If we know that the output has a specific nonnormal distribution such as the exponential distribution, then we may also apply Wald (1945)’s original SPRT. Throughout our investigation we pay special attention to the design and analysis of these experiments.

Original language | English |
---|---|

Place of Publication | Tilburg |

Publisher | CentER, Center for Economic Research |

Number of pages | 39 |

Volume | 2017-001 |

Publication status | Published - 10 Jan 2017 |

### Publication series

Name | CentER Discussion Paper |
---|---|

Volume | 2017-001 |

### Fingerprint

### Keywords

- sequential test
- Wald
- Hall
- robustness
- lognormal
- gamma distribution
- Monte Carlo

### Cite this

*Sequential Probability Ration Tests: Conservative and Robust*. (CentER Discussion Paper; Vol. 2017-001). Tilburg: CentER, Center for Economic Research.

}

**Sequential Probability Ration Tests : Conservative and Robust.** / Kleijnen, J.P.C.; Shi, Wen.

Research output: Working paper › Discussion paper › Other research output

TY - UNPB

T1 - Sequential Probability Ration Tests

T2 - Conservative and Robust

AU - Kleijnen, J.P.C.

AU - Shi, Wen

PY - 2017/1/10

Y1 - 2017/1/10

N2 - In practice, most computers generate simulation outputs sequentially, so it isattractive to analyze these outputs through sequential statistical methods such as sequential probability ratio tests (SPRTs). We investigate several SPRTs for choosing between two hypothesized values for the mean output (response). One SPRT is published in Wald (1945), and allows general distribution types. For a normal (Gaussian) distribution this SPRT assumes a known variance, but in our modified SPRT we estimate the variance. Another SPRT is published in Hall (1962), and assumes a normal distribution with an unknown variance estimated from a pilot sample. We also investigate a modification, replacing this pilot-sample estimator by a fully sequential estimator. We present a sequence of MonteCarlo experiments for quantifying the performance of these SPRTs. In experiment #1 the simulation outputs are normal. This experiment suggests that Wald (1945)’s SPRT with estimated variance gives significantly high error rates. Hall (1962)’s original and modified SPRTs are conservative; i.e., the actual error rates are much smaller than the prespecified (nominal) rates. The most efficient SPRT is our modified Hall (1962) SPRT. In experiment #2 we examine the robustness of the various SPRTs in case of nonnormal output. If we know that the output has a specific nonnormal distribution such as the exponential distribution, then we may also apply Wald (1945)’s original SPRT. Throughout our investigation we pay special attention to the design and analysis of these experiments.

AB - In practice, most computers generate simulation outputs sequentially, so it isattractive to analyze these outputs through sequential statistical methods such as sequential probability ratio tests (SPRTs). We investigate several SPRTs for choosing between two hypothesized values for the mean output (response). One SPRT is published in Wald (1945), and allows general distribution types. For a normal (Gaussian) distribution this SPRT assumes a known variance, but in our modified SPRT we estimate the variance. Another SPRT is published in Hall (1962), and assumes a normal distribution with an unknown variance estimated from a pilot sample. We also investigate a modification, replacing this pilot-sample estimator by a fully sequential estimator. We present a sequence of MonteCarlo experiments for quantifying the performance of these SPRTs. In experiment #1 the simulation outputs are normal. This experiment suggests that Wald (1945)’s SPRT with estimated variance gives significantly high error rates. Hall (1962)’s original and modified SPRTs are conservative; i.e., the actual error rates are much smaller than the prespecified (nominal) rates. The most efficient SPRT is our modified Hall (1962) SPRT. In experiment #2 we examine the robustness of the various SPRTs in case of nonnormal output. If we know that the output has a specific nonnormal distribution such as the exponential distribution, then we may also apply Wald (1945)’s original SPRT. Throughout our investigation we pay special attention to the design and analysis of these experiments.

KW - sequential test

KW - Wald

KW - Hall

KW - robustness

KW - lognormal

KW - gamma distribution

KW - Monte Carlo

M3 - Discussion paper

VL - 2017-001

T3 - CentER Discussion Paper

BT - Sequential Probability Ration Tests

PB - CentER, Center for Economic Research

CY - Tilburg

ER -