Sequential Probability Ration Tests: Conservative and Robust

Research output: Working paperDiscussion paperOther research output

458 Downloads (Pure)

Abstract

In practice, most computers generate simulation outputs sequentially, so it is
attractive to analyze these outputs through sequential statistical methods such as sequential probability ratio tests (SPRTs). We investigate several SPRTs for choosing between two hypothesized values for the mean output (response). One SPRT is published in Wald (1945), and allows general distribution types. For a normal (Gaussian) distribution this SPRT assumes a known variance, but in our modified SPRT we estimate the variance. Another SPRT is published in Hall (1962), and assumes a normal distribution with an unknown variance estimated from a pilot sample. We also investigate a modification, replacing this pilot-sample estimator by a fully sequential estimator. We present a sequence of Monte
Carlo experiments for quantifying the performance of these SPRTs. In experiment #1 the simulation outputs are normal. This experiment suggests that Wald (1945)’s SPRT with estimated variance gives significantly high error rates. Hall (1962)’s original and modified SPRTs are conservative; i.e., the actual error rates are much smaller than the prespecified (nominal) rates. The most efficient SPRT is our modified Hall (1962) SPRT. In experiment #2 we examine the robustness of the various SPRTs in case of nonnormal output. If we know that the output has a specific nonnormal distribution such as the exponential distribution, then we may also apply Wald (1945)’s original SPRT. Throughout our investigation we pay special attention to the design and analysis of these experiments.
Original languageEnglish
Place of PublicationTilburg
PublisherCentER, Center for Economic Research
Number of pages39
Volume2017-001
Publication statusPublished - 10 Jan 2017

Publication series

NameCentER Discussion Paper
Volume2017-001

Fingerprint

Sequential Probability Ratio Test
Output
Gaussian distribution
Experiment
Error Rate
Estimator
Non-normal Distribution
Sequential Methods
Exponential distribution
Statistical method

Keywords

  • sequential test
  • Wald
  • Hall
  • robustness
  • lognormal
  • gamma distribution
  • Monte Carlo

Cite this

Kleijnen, J. P. C., & Shi, W. (2017). Sequential Probability Ration Tests: Conservative and Robust. (CentER Discussion Paper; Vol. 2017-001). Tilburg: CentER, Center for Economic Research.
Kleijnen, J.P.C. ; Shi, Wen. / Sequential Probability Ration Tests : Conservative and Robust. Tilburg : CentER, Center for Economic Research, 2017. (CentER Discussion Paper).
@techreport{5f24e30d79314be496f068f8898e6667,
title = "Sequential Probability Ration Tests: Conservative and Robust",
abstract = "In practice, most computers generate simulation outputs sequentially, so it isattractive to analyze these outputs through sequential statistical methods such as sequential probability ratio tests (SPRTs). We investigate several SPRTs for choosing between two hypothesized values for the mean output (response). One SPRT is published in Wald (1945), and allows general distribution types. For a normal (Gaussian) distribution this SPRT assumes a known variance, but in our modified SPRT we estimate the variance. Another SPRT is published in Hall (1962), and assumes a normal distribution with an unknown variance estimated from a pilot sample. We also investigate a modification, replacing this pilot-sample estimator by a fully sequential estimator. We present a sequence of MonteCarlo experiments for quantifying the performance of these SPRTs. In experiment #1 the simulation outputs are normal. This experiment suggests that Wald (1945)’s SPRT with estimated variance gives significantly high error rates. Hall (1962)’s original and modified SPRTs are conservative; i.e., the actual error rates are much smaller than the prespecified (nominal) rates. The most efficient SPRT is our modified Hall (1962) SPRT. In experiment #2 we examine the robustness of the various SPRTs in case of nonnormal output. If we know that the output has a specific nonnormal distribution such as the exponential distribution, then we may also apply Wald (1945)’s original SPRT. Throughout our investigation we pay special attention to the design and analysis of these experiments.",
keywords = "sequential test, Wald, Hall, robustness, lognormal, gamma distribution, Monte Carlo",
author = "J.P.C. Kleijnen and Wen Shi",
year = "2017",
month = "1",
day = "10",
language = "English",
volume = "2017-001",
series = "CentER Discussion Paper",
publisher = "CentER, Center for Economic Research",
type = "WorkingPaper",
institution = "CentER, Center for Economic Research",

}

Kleijnen, JPC & Shi, W 2017 'Sequential Probability Ration Tests: Conservative and Robust' CentER Discussion Paper, vol. 2017-001, CentER, Center for Economic Research, Tilburg.

Sequential Probability Ration Tests : Conservative and Robust. / Kleijnen, J.P.C.; Shi, Wen.

Tilburg : CentER, Center for Economic Research, 2017. (CentER Discussion Paper; Vol. 2017-001).

Research output: Working paperDiscussion paperOther research output

TY - UNPB

T1 - Sequential Probability Ration Tests

T2 - Conservative and Robust

AU - Kleijnen, J.P.C.

AU - Shi, Wen

PY - 2017/1/10

Y1 - 2017/1/10

N2 - In practice, most computers generate simulation outputs sequentially, so it isattractive to analyze these outputs through sequential statistical methods such as sequential probability ratio tests (SPRTs). We investigate several SPRTs for choosing between two hypothesized values for the mean output (response). One SPRT is published in Wald (1945), and allows general distribution types. For a normal (Gaussian) distribution this SPRT assumes a known variance, but in our modified SPRT we estimate the variance. Another SPRT is published in Hall (1962), and assumes a normal distribution with an unknown variance estimated from a pilot sample. We also investigate a modification, replacing this pilot-sample estimator by a fully sequential estimator. We present a sequence of MonteCarlo experiments for quantifying the performance of these SPRTs. In experiment #1 the simulation outputs are normal. This experiment suggests that Wald (1945)’s SPRT with estimated variance gives significantly high error rates. Hall (1962)’s original and modified SPRTs are conservative; i.e., the actual error rates are much smaller than the prespecified (nominal) rates. The most efficient SPRT is our modified Hall (1962) SPRT. In experiment #2 we examine the robustness of the various SPRTs in case of nonnormal output. If we know that the output has a specific nonnormal distribution such as the exponential distribution, then we may also apply Wald (1945)’s original SPRT. Throughout our investigation we pay special attention to the design and analysis of these experiments.

AB - In practice, most computers generate simulation outputs sequentially, so it isattractive to analyze these outputs through sequential statistical methods such as sequential probability ratio tests (SPRTs). We investigate several SPRTs for choosing between two hypothesized values for the mean output (response). One SPRT is published in Wald (1945), and allows general distribution types. For a normal (Gaussian) distribution this SPRT assumes a known variance, but in our modified SPRT we estimate the variance. Another SPRT is published in Hall (1962), and assumes a normal distribution with an unknown variance estimated from a pilot sample. We also investigate a modification, replacing this pilot-sample estimator by a fully sequential estimator. We present a sequence of MonteCarlo experiments for quantifying the performance of these SPRTs. In experiment #1 the simulation outputs are normal. This experiment suggests that Wald (1945)’s SPRT with estimated variance gives significantly high error rates. Hall (1962)’s original and modified SPRTs are conservative; i.e., the actual error rates are much smaller than the prespecified (nominal) rates. The most efficient SPRT is our modified Hall (1962) SPRT. In experiment #2 we examine the robustness of the various SPRTs in case of nonnormal output. If we know that the output has a specific nonnormal distribution such as the exponential distribution, then we may also apply Wald (1945)’s original SPRT. Throughout our investigation we pay special attention to the design and analysis of these experiments.

KW - sequential test

KW - Wald

KW - Hall

KW - robustness

KW - lognormal

KW - gamma distribution

KW - Monte Carlo

M3 - Discussion paper

VL - 2017-001

T3 - CentER Discussion Paper

BT - Sequential Probability Ration Tests

PB - CentER, Center for Economic Research

CY - Tilburg

ER -

Kleijnen JPC, Shi W. Sequential Probability Ration Tests: Conservative and Robust. Tilburg: CentER, Center for Economic Research. 2017 Jan 10. (CentER Discussion Paper).