Predicting undesired treatment outcome in mental healthcare: Machine learning study

K. van Mens*, J. Lokkerbol, R. T. J. M. Janssen, R. P. J. de Lange, B. Tiemens

*Corresponding author for this work

Research output: Contribution to journalArticleScientific

53 Downloads (Pure)


It remains a challenge to predict which treatment will work for which patient in mental healthcare.

In this study we compare machine algorithms to predict during treatment which patients will not benefit from brief mental health treatment and present trade-offs that must be considered before an algorithm can be used in clinical practice.

Using an anonymized dataset containing routine outcome monitoring data from a mental healthcare organization in the Netherlands (n = 2,655), we applied three machine learning algorithms to predict treatment outcome. The algorithms were internally validated with cross-validation on a training sample (n = 1,860) and externally validated on an unseen test sample (n = 795).

The performance of the three algorithms did not significantly differ on the test set. With a default classification cut-off at 0.5 predicted probability, the extreme gradient boosting algorithm showed the highest positive predictive value (ppv) of 0.71(0.61 – 0.77) with a sensitivity of 0.35 (0.29 – 0.41) and area under the curve of 0.78. A trade-off can be made between ppv and sensitivity by choosing different cut-off probabilities. With a cut-off at 0.63, the ppv increased to 0.87 and the sensitivity dropped to 0.17. With a cut-off of at 0.38, the ppv decreased to 0.61 and the sensitivity increased to 0.57.

Machine learning can be used to predict treatment outcomes based on routine monitoring data.This allows practitioners to choose their own trade-off between being selective and more certain versus inclusive and less certain.
Original languageEnglish
Number of pages24
JournalJMIR Medical Informatics
Publication statusPublished - 2019


Dive into the research topics of 'Predicting undesired treatment outcome in mental healthcare: Machine learning study'. Together they form a unique fingerprint.

Cite this