Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models

Research output: Working paperDiscussion paperOther research output

318 Downloads (Pure)

Abstract

This chapter gives a survey on the use of statistical designs for what-if analysis in simula- tion, including sensitivity analysis, optimization, and validation/verification. Sensitivity analysis is divided into two phases. The first phase is a pilot stage, which consists of screening or searching for the important factors among (say) hundreds of potentially important factors. A novel screening technique is presented, namely sequential bifurcation. The second phase uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as a metamodel or a response surface. Regression analysis gives better results when the simu- lation experiment is well designed, using either classical statistical designs (such as frac- tional factorials) or optimal designs (such as pioneered by Fedorov, Kiefer, and Wolfo- witz). To optimize the simulated system, the analysts may apply Response Surface Metho- dology (RSM); RSM combines regression analysis, statistical designs, and steepest-ascent hill-climbing. To validate a simulation model, again regression analysis and statistical designs may be applied. Several numerical examples and case-studies illustrate how statisti- cal techniques can reduce the ad hoc character of simulation; that is, these statistical techniques can make simulation studies give more general results, in less time. Appendix 1 summarizes confidence intervals for expected values, proportions, and quantiles, in termi- nating and steady-state simulations. Appendix 2 gives details on four variance reduction techniques, namely common pseudorandom numbers, antithetic numbers, control variates or regression sampling, and importance sampling. Appendix 3 describes jackknifing, which may give robust confidence intervals.
Original languageEnglish
Place of PublicationTilburg
PublisherOperations research
Number of pages61
Volume1997-52
Publication statusPublished - 1997

Publication series

NameCentER Discussion Paper
Volume1997-52

Fingerprint

Experimental design
Sensitivity Analysis
Simulation Model
Regression Analysis
Optimization
Response Surface
Screening
Confidence interval
Simulation
Control Variates
Pseudorandom numbers
Hill Climbing
Variance Reduction
Ascent
Verification and Validation
Methodology
Importance Sampling
Model Analysis
Metamodel
Quantile

Keywords

  • least squares
  • distribution-free
  • non-parametric
  • stopping rule
  • run-length
  • Von Neumann
  • median
  • seed
  • likelihood ratio

Cite this

Kleijnen, J. P. C. (1997). Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models. (CentER Discussion Paper; Vol. 1997-52). Tilburg: Operations research.
@techreport{c0e2bc10e5504cf2b6496452dfea1ffe,
title = "Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models",
abstract = "This chapter gives a survey on the use of statistical designs for what-if analysis in simula- tion, including sensitivity analysis, optimization, and validation/verification. Sensitivity analysis is divided into two phases. The first phase is a pilot stage, which consists of screening or searching for the important factors among (say) hundreds of potentially important factors. A novel screening technique is presented, namely sequential bifurcation. The second phase uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as a metamodel or a response surface. Regression analysis gives better results when the simu- lation experiment is well designed, using either classical statistical designs (such as frac- tional factorials) or optimal designs (such as pioneered by Fedorov, Kiefer, and Wolfo- witz). To optimize the simulated system, the analysts may apply Response Surface Metho- dology (RSM); RSM combines regression analysis, statistical designs, and steepest-ascent hill-climbing. To validate a simulation model, again regression analysis and statistical designs may be applied. Several numerical examples and case-studies illustrate how statisti- cal techniques can reduce the ad hoc character of simulation; that is, these statistical techniques can make simulation studies give more general results, in less time. Appendix 1 summarizes confidence intervals for expected values, proportions, and quantiles, in termi- nating and steady-state simulations. Appendix 2 gives details on four variance reduction techniques, namely common pseudorandom numbers, antithetic numbers, control variates or regression sampling, and importance sampling. Appendix 3 describes jackknifing, which may give robust confidence intervals.",
keywords = "least squares, distribution-free, non-parametric, stopping rule, run-length, Von Neumann, median, seed, likelihood ratio",
author = "J.P.C. Kleijnen",
note = "Pagination: 61",
year = "1997",
language = "English",
volume = "1997-52",
series = "CentER Discussion Paper",
publisher = "Operations research",
type = "WorkingPaper",
institution = "Operations research",

}

Kleijnen, JPC 1997 'Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models' CentER Discussion Paper, vol. 1997-52, Operations research, Tilburg.

Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models. / Kleijnen, J.P.C.

Tilburg : Operations research, 1997. (CentER Discussion Paper; Vol. 1997-52).

Research output: Working paperDiscussion paperOther research output

TY - UNPB

T1 - Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models

AU - Kleijnen, J.P.C.

N1 - Pagination: 61

PY - 1997

Y1 - 1997

N2 - This chapter gives a survey on the use of statistical designs for what-if analysis in simula- tion, including sensitivity analysis, optimization, and validation/verification. Sensitivity analysis is divided into two phases. The first phase is a pilot stage, which consists of screening or searching for the important factors among (say) hundreds of potentially important factors. A novel screening technique is presented, namely sequential bifurcation. The second phase uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as a metamodel or a response surface. Regression analysis gives better results when the simu- lation experiment is well designed, using either classical statistical designs (such as frac- tional factorials) or optimal designs (such as pioneered by Fedorov, Kiefer, and Wolfo- witz). To optimize the simulated system, the analysts may apply Response Surface Metho- dology (RSM); RSM combines regression analysis, statistical designs, and steepest-ascent hill-climbing. To validate a simulation model, again regression analysis and statistical designs may be applied. Several numerical examples and case-studies illustrate how statisti- cal techniques can reduce the ad hoc character of simulation; that is, these statistical techniques can make simulation studies give more general results, in less time. Appendix 1 summarizes confidence intervals for expected values, proportions, and quantiles, in termi- nating and steady-state simulations. Appendix 2 gives details on four variance reduction techniques, namely common pseudorandom numbers, antithetic numbers, control variates or regression sampling, and importance sampling. Appendix 3 describes jackknifing, which may give robust confidence intervals.

AB - This chapter gives a survey on the use of statistical designs for what-if analysis in simula- tion, including sensitivity analysis, optimization, and validation/verification. Sensitivity analysis is divided into two phases. The first phase is a pilot stage, which consists of screening or searching for the important factors among (say) hundreds of potentially important factors. A novel screening technique is presented, namely sequential bifurcation. The second phase uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as a metamodel or a response surface. Regression analysis gives better results when the simu- lation experiment is well designed, using either classical statistical designs (such as frac- tional factorials) or optimal designs (such as pioneered by Fedorov, Kiefer, and Wolfo- witz). To optimize the simulated system, the analysts may apply Response Surface Metho- dology (RSM); RSM combines regression analysis, statistical designs, and steepest-ascent hill-climbing. To validate a simulation model, again regression analysis and statistical designs may be applied. Several numerical examples and case-studies illustrate how statisti- cal techniques can reduce the ad hoc character of simulation; that is, these statistical techniques can make simulation studies give more general results, in less time. Appendix 1 summarizes confidence intervals for expected values, proportions, and quantiles, in termi- nating and steady-state simulations. Appendix 2 gives details on four variance reduction techniques, namely common pseudorandom numbers, antithetic numbers, control variates or regression sampling, and importance sampling. Appendix 3 describes jackknifing, which may give robust confidence intervals.

KW - least squares

KW - distribution-free

KW - non-parametric

KW - stopping rule

KW - run-length

KW - Von Neumann

KW - median

KW - seed

KW - likelihood ratio

M3 - Discussion paper

VL - 1997-52

T3 - CentER Discussion Paper

BT - Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models

PB - Operations research

CY - Tilburg

ER -