Workflow techniques for the robust use of Bayes factors

Inferences about hypotheses are ubiquitous in the cognitive sciences. Bayes factors provide one general wayto compare different hypotheses by their compatibility with the observed data. Those quantifications can thenalso be used to choose between hypotheses. While Bayes factors provide an immediate approach to hypothesistesting, they are highly sensitive to details of the data/model assumptions and it’s unclear whether the detailsof the computational implementation (such as bridge sampling) are unbiased for complex analyses. Here, westudy how Bayes factors misbehave under different conditions. This includes a study of errors in the estimationof Bayes factors; thefirst-ever use of simulation-based calibration to test the accuracy and bias of Bayes factorestimates using bridge sampling; a study of the stability of Bayes factors against different MCMC draws andsampling variation in the data; and a look at the variability of decisions based on Bayes factors using a utilityfunction. We outline a Bayes factor workflow that researchers can use to study whether Bayes factors are ro-bust for their individual analysis. Reproducible code is available fromhttps://osf.io/y354c/.