False alarm? A comprehensive reanalysis of "Evidence that psychopathology symptom networks have limited replicability" by Forbes, Wright, Markon, and Krueger (2017)

Denny Borsboom, Eiko Fried, Sacha Epskamp, Lourens Waldorp, Claudia Van Borkulo, Han van der Maas, A.O.J. Cramer

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Forbes, Wright, Markon, and Krueger (2017) stated that "psychopathology networks have limited replicability" (p. 1011) and that "popular network analysis methods produce unreliable results" (p. 1011). These conclusions are based on an assessment of the replicability of four different network models for symptoms of major depression and generalized anxiety across two samples; in addition, Forbes et al. analyzed the stability of the network models within the samples using split-halves. Our reanalysis of the same data with the same methods led to results directly opposed to theirs: All network models replicated very well across the two data sets and across the split-halves. We trace the differences between Forbes et al.'s results and our own to the fact that they did not appear to accurately implement all network models and used debatable metrics to assess replicability. In particular, they deviated from existing estimation routines for relative importance networks, did not acknowledge the fact that the skip structure used in the interviews strongly distorted correlations between symptoms, and incorrectly assumed that network structures and metrics should be the same not only across the different samples but also across the different network models used. In addition to a comprehensive reanalysis of the data, we end with a discussion of best practices concerning future research into the replicability of psychometric networks.
Original languageEnglish
Pages (from-to)989-999
JournalJournal of Abnormal Psychology
Volume126
Issue number7
DOIs
Publication statusPublished - 2017

Fingerprint

Practice Guidelines
Interviews
Depression

Cite this

Borsboom, Denny ; Fried, Eiko ; Epskamp, Sacha ; Waldorp, Lourens ; Van Borkulo, Claudia ; van der Maas, Han ; Cramer, A.O.J. / False alarm? A comprehensive reanalysis of "Evidence that psychopathology symptom networks have limited replicability" by Forbes, Wright, Markon, and Krueger (2017). In: Journal of Abnormal Psychology. 2017 ; Vol. 126, No. 7. pp. 989-999.
@article{5d98474b5a5241ae87e28902220e82a7,
title = "False alarm? A comprehensive reanalysis of {"}Evidence that psychopathology symptom networks have limited replicability{"} by Forbes, Wright, Markon, and Krueger (2017)",
abstract = "Forbes, Wright, Markon, and Krueger (2017) stated that {"}psychopathology networks have limited replicability{"} (p. 1011) and that {"}popular network analysis methods produce unreliable results{"} (p. 1011). These conclusions are based on an assessment of the replicability of four different network models for symptoms of major depression and generalized anxiety across two samples; in addition, Forbes et al. analyzed the stability of the network models within the samples using split-halves. Our reanalysis of the same data with the same methods led to results directly opposed to theirs: All network models replicated very well across the two data sets and across the split-halves. We trace the differences between Forbes et al.'s results and our own to the fact that they did not appear to accurately implement all network models and used debatable metrics to assess replicability. In particular, they deviated from existing estimation routines for relative importance networks, did not acknowledge the fact that the skip structure used in the interviews strongly distorted correlations between symptoms, and incorrectly assumed that network structures and metrics should be the same not only across the different samples but also across the different network models used. In addition to a comprehensive reanalysis of the data, we end with a discussion of best practices concerning future research into the replicability of psychometric networks.",
author = "Denny Borsboom and Eiko Fried and Sacha Epskamp and Lourens Waldorp and {Van Borkulo}, Claudia and {van der Maas}, Han and A.O.J. Cramer",
year = "2017",
doi = "10.1037/abn0000306",
language = "English",
volume = "126",
pages = "989--999",
journal = "Journal of Abnormal Psychology",
issn = "0021-843X",
publisher = "American Psychological Association",
number = "7",

}

False alarm? A comprehensive reanalysis of "Evidence that psychopathology symptom networks have limited replicability" by Forbes, Wright, Markon, and Krueger (2017). / Borsboom, Denny; Fried, Eiko; Epskamp, Sacha; Waldorp, Lourens; Van Borkulo, Claudia; van der Maas, Han; Cramer, A.O.J.

In: Journal of Abnormal Psychology, Vol. 126, No. 7, 2017, p. 989-999.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - False alarm? A comprehensive reanalysis of "Evidence that psychopathology symptom networks have limited replicability" by Forbes, Wright, Markon, and Krueger (2017)

AU - Borsboom, Denny

AU - Fried, Eiko

AU - Epskamp, Sacha

AU - Waldorp, Lourens

AU - Van Borkulo, Claudia

AU - van der Maas, Han

AU - Cramer, A.O.J.

PY - 2017

Y1 - 2017

N2 - Forbes, Wright, Markon, and Krueger (2017) stated that "psychopathology networks have limited replicability" (p. 1011) and that "popular network analysis methods produce unreliable results" (p. 1011). These conclusions are based on an assessment of the replicability of four different network models for symptoms of major depression and generalized anxiety across two samples; in addition, Forbes et al. analyzed the stability of the network models within the samples using split-halves. Our reanalysis of the same data with the same methods led to results directly opposed to theirs: All network models replicated very well across the two data sets and across the split-halves. We trace the differences between Forbes et al.'s results and our own to the fact that they did not appear to accurately implement all network models and used debatable metrics to assess replicability. In particular, they deviated from existing estimation routines for relative importance networks, did not acknowledge the fact that the skip structure used in the interviews strongly distorted correlations between symptoms, and incorrectly assumed that network structures and metrics should be the same not only across the different samples but also across the different network models used. In addition to a comprehensive reanalysis of the data, we end with a discussion of best practices concerning future research into the replicability of psychometric networks.

AB - Forbes, Wright, Markon, and Krueger (2017) stated that "psychopathology networks have limited replicability" (p. 1011) and that "popular network analysis methods produce unreliable results" (p. 1011). These conclusions are based on an assessment of the replicability of four different network models for symptoms of major depression and generalized anxiety across two samples; in addition, Forbes et al. analyzed the stability of the network models within the samples using split-halves. Our reanalysis of the same data with the same methods led to results directly opposed to theirs: All network models replicated very well across the two data sets and across the split-halves. We trace the differences between Forbes et al.'s results and our own to the fact that they did not appear to accurately implement all network models and used debatable metrics to assess replicability. In particular, they deviated from existing estimation routines for relative importance networks, did not acknowledge the fact that the skip structure used in the interviews strongly distorted correlations between symptoms, and incorrectly assumed that network structures and metrics should be the same not only across the different samples but also across the different network models used. In addition to a comprehensive reanalysis of the data, we end with a discussion of best practices concerning future research into the replicability of psychometric networks.

U2 - 10.1037/abn0000306

DO - 10.1037/abn0000306

M3 - Article

VL - 126

SP - 989

EP - 999

JO - Journal of Abnormal Psychology

JF - Journal of Abnormal Psychology

SN - 0021-843X

IS - 7

ER -