Shrinkage priors for Bayesian penalized regression

Sara Van Erp, Daniel L. Oberski, Joris Mulder

Research output: Contribution to journalReview articleScientificpeer-review

Abstract

In linear regression problems with many predictors, penalized regression techniques are often used to guard against overfitting and to select variables relevant for predicting an outcome variable. Recently, Bayesian penalization is becoming increasingly popular in which the prior distribution performs a function similar to that of the penalty term in classical penalization. Specifically, the so-called shrinkage priors in Bayesian penalization aim to shrink small effects to zero while maintaining true large effects. Compared to classical penalization techniques, Bayesian penalization techniques perform similarly or sometimes even better, and they offer additional advantages such as readily available uncertainty estimates, automatic estimation of the penalty parameter, and more flexibility in terms of penalties that can be considered. However, many different shrinkage priors exist and the available, often quite technical, literature primarily focuses on presenting one shrinkage prior and often provides comparisons with only one or two other shrinkage priors. This can make it difficult for researchers to navigate through the many prior options and choose a shrinkage prior for the problem at hand. Therefore, the aim of this paper is to provide a comprehensive overview of the literature on Bayesian penalization. We provide a theoretical and conceptual comparison of nine different shrinkage priors and parametrize the priors, if possible, in terms of scale mixture of normal distributions to facilitate comparisons. We illustrate different characteristics and behaviors of the shrinkage priors and compare their performance in terms of prediction and variable selection in a simulation study. Additionally, we provide two empirical examples to illustrate the application of Bayesian penalization. Finally, an R package bayesreg is available online (https://github.com/sara-vanerp/bayesreg) which allows researchers to perform Bayesian penalized regression with novel shrinkage priors in an easy manner.
Original languageEnglish
Pages (from-to)31-50
JournalJournal of Mathematical Psychology
Volume89
DOIs
Publication statusPublished - 2019

Fingerprint

Penalized Regression
Shrinkage
Penalization
Normal Distribution
Linear Models
Penalty
Mixture of Normal Distributions
Scale Mixture
Overfitting
Normal distribution
Variable Selection
Prior distribution
Linear regression
Predictors
Choose
Flexibility
Simulation Study
Uncertainty
Prediction

Keywords

  • ADAPTIVE LASSO
  • Bayesian
  • Empirical Bayes
  • FREQUENTIST
  • HORSESHOE
  • INFORMATION
  • MODELS
  • Penalization
  • REGULARIZATION
  • Regression
  • Shrinkage priors
  • VARIABLE-SELECTION

Cite this

@article{f2d954c38100413eaf4074c2176d6e4d,
title = "Shrinkage priors for Bayesian penalized regression",
abstract = "In linear regression problems with many predictors, penalized regression techniques are often used to guard against overfitting and to select variables relevant for predicting an outcome variable. Recently, Bayesian penalization is becoming increasingly popular in which the prior distribution performs a function similar to that of the penalty term in classical penalization. Specifically, the so-called shrinkage priors in Bayesian penalization aim to shrink small effects to zero while maintaining true large effects. Compared to classical penalization techniques, Bayesian penalization techniques perform similarly or sometimes even better, and they offer additional advantages such as readily available uncertainty estimates, automatic estimation of the penalty parameter, and more flexibility in terms of penalties that can be considered. However, many different shrinkage priors exist and the available, often quite technical, literature primarily focuses on presenting one shrinkage prior and often provides comparisons with only one or two other shrinkage priors. This can make it difficult for researchers to navigate through the many prior options and choose a shrinkage prior for the problem at hand. Therefore, the aim of this paper is to provide a comprehensive overview of the literature on Bayesian penalization. We provide a theoretical and conceptual comparison of nine different shrinkage priors and parametrize the priors, if possible, in terms of scale mixture of normal distributions to facilitate comparisons. We illustrate different characteristics and behaviors of the shrinkage priors and compare their performance in terms of prediction and variable selection in a simulation study. Additionally, we provide two empirical examples to illustrate the application of Bayesian penalization. Finally, an R package bayesreg is available online (https://github.com/sara-vanerp/bayesreg) which allows researchers to perform Bayesian penalized regression with novel shrinkage priors in an easy manner.",
keywords = "ADAPTIVE LASSO, Bayesian, Empirical Bayes, FREQUENTIST, HORSESHOE, INFORMATION, MODELS, Penalization, REGULARIZATION, Regression, Shrinkage priors, VARIABLE-SELECTION",
author = "{Van Erp}, Sara and Oberski, {Daniel L.} and Joris Mulder",
year = "2019",
doi = "10.1016/j.jmp.2018.12.004",
language = "English",
volume = "89",
pages = "31--50",
journal = "Journal of Mathematical Psychology",
issn = "0022-2496",
publisher = "ACADEMIC PRESS INC ELSEVIER SCIENCE",

}

Shrinkage priors for Bayesian penalized regression. / Van Erp, Sara; Oberski, Daniel L.; Mulder, Joris.

In: Journal of Mathematical Psychology, Vol. 89, 2019, p. 31-50.

Research output: Contribution to journalReview articleScientificpeer-review

TY - JOUR

T1 - Shrinkage priors for Bayesian penalized regression

AU - Van Erp, Sara

AU - Oberski, Daniel L.

AU - Mulder, Joris

PY - 2019

Y1 - 2019

N2 - In linear regression problems with many predictors, penalized regression techniques are often used to guard against overfitting and to select variables relevant for predicting an outcome variable. Recently, Bayesian penalization is becoming increasingly popular in which the prior distribution performs a function similar to that of the penalty term in classical penalization. Specifically, the so-called shrinkage priors in Bayesian penalization aim to shrink small effects to zero while maintaining true large effects. Compared to classical penalization techniques, Bayesian penalization techniques perform similarly or sometimes even better, and they offer additional advantages such as readily available uncertainty estimates, automatic estimation of the penalty parameter, and more flexibility in terms of penalties that can be considered. However, many different shrinkage priors exist and the available, often quite technical, literature primarily focuses on presenting one shrinkage prior and often provides comparisons with only one or two other shrinkage priors. This can make it difficult for researchers to navigate through the many prior options and choose a shrinkage prior for the problem at hand. Therefore, the aim of this paper is to provide a comprehensive overview of the literature on Bayesian penalization. We provide a theoretical and conceptual comparison of nine different shrinkage priors and parametrize the priors, if possible, in terms of scale mixture of normal distributions to facilitate comparisons. We illustrate different characteristics and behaviors of the shrinkage priors and compare their performance in terms of prediction and variable selection in a simulation study. Additionally, we provide two empirical examples to illustrate the application of Bayesian penalization. Finally, an R package bayesreg is available online (https://github.com/sara-vanerp/bayesreg) which allows researchers to perform Bayesian penalized regression with novel shrinkage priors in an easy manner.

AB - In linear regression problems with many predictors, penalized regression techniques are often used to guard against overfitting and to select variables relevant for predicting an outcome variable. Recently, Bayesian penalization is becoming increasingly popular in which the prior distribution performs a function similar to that of the penalty term in classical penalization. Specifically, the so-called shrinkage priors in Bayesian penalization aim to shrink small effects to zero while maintaining true large effects. Compared to classical penalization techniques, Bayesian penalization techniques perform similarly or sometimes even better, and they offer additional advantages such as readily available uncertainty estimates, automatic estimation of the penalty parameter, and more flexibility in terms of penalties that can be considered. However, many different shrinkage priors exist and the available, often quite technical, literature primarily focuses on presenting one shrinkage prior and often provides comparisons with only one or two other shrinkage priors. This can make it difficult for researchers to navigate through the many prior options and choose a shrinkage prior for the problem at hand. Therefore, the aim of this paper is to provide a comprehensive overview of the literature on Bayesian penalization. We provide a theoretical and conceptual comparison of nine different shrinkage priors and parametrize the priors, if possible, in terms of scale mixture of normal distributions to facilitate comparisons. We illustrate different characteristics and behaviors of the shrinkage priors and compare their performance in terms of prediction and variable selection in a simulation study. Additionally, we provide two empirical examples to illustrate the application of Bayesian penalization. Finally, an R package bayesreg is available online (https://github.com/sara-vanerp/bayesreg) which allows researchers to perform Bayesian penalized regression with novel shrinkage priors in an easy manner.

KW - ADAPTIVE LASSO

KW - Bayesian

KW - Empirical Bayes

KW - FREQUENTIST

KW - HORSESHOE

KW - INFORMATION

KW - MODELS

KW - Penalization

KW - REGULARIZATION

KW - Regression

KW - Shrinkage priors

KW - VARIABLE-SELECTION

U2 - 10.1016/j.jmp.2018.12.004

DO - 10.1016/j.jmp.2018.12.004

M3 - Review article

VL - 89

SP - 31

EP - 50

JO - Journal of Mathematical Psychology

JF - Journal of Mathematical Psychology

SN - 0022-2496

ER -