Using confidence intervals for assessing reliability of real tests

P.R. Oosterwijk; L.A. van der Ark; K. Sijtsma

doi:10.1177/1073191117737375

Using confidence intervals for assessing reliability of real tests

P.R. Oosterwijk, L.A. van der Ark^*, K. Sijtsma

^*Corresponding author for this work

Department of Methodology and Statistics

Research output: Contribution to journal › Article › Scientific › peer-review

14 Citations (Scopus)

93 Downloads (Pure)

Abstract

Test authors report sample reliability values but rarely consider the sampling error and related confidence intervals. This study investigated the truth of this conjecture for 116 tests with 1,024 reliability estimates (105 pertaining to test batteries and 919 to tests measuring a single attribute) obtained from an online database. Based on 90% confidence intervals, approximately 20% of the initial quality assessments had to be downgraded. For 95% confidence intervals, the percentage was approximately 23%. The results demonstrated that reported reliability values cannot be trusted without considering their estimation precision.
Keywords: confidence intervals for reliability; precision of reported reliability; quality assessment of reliability; test-score reliability

Original language	English
Pages (from-to)	1207-1216
Journal	Assessment
Volume	26
Issue number	7
DOIs	https://doi.org/10.1177/1073191117737375
Publication status	Published - 2019

Keywords

COEFFICIENT ALPHA
HISTORY
LOWER BOUNDS
SCORE
confidence intervals for reliability
precision of reported reliability
quality assessment of reliability
test-score reliability

Access to Document

10.1177/1073191117737375

MTO Sijtsma using confidence intervals Assessment 2019Final published version, 282 KB

Cite this

@article{f1589ed1344e4ba2b94e582bf60854d2,

title = "Using confidence intervals for assessing reliability of real tests",

abstract = "Test authors report sample reliability values but rarely consider the sampling error and related confidence intervals. This study investigated the truth of this conjecture for 116 tests with 1,024 reliability estimates (105 pertaining to test batteries and 919 to tests measuring a single attribute) obtained from an online database. Based on 90% confidence intervals, approximately 20% of the initial quality assessments had to be downgraded. For 95% confidence intervals, the percentage was approximately 23%. The results demonstrated that reported reliability values cannot be trusted without considering their estimation precision.Keywords: confidence intervals for reliability; precision of reported reliability; quality assessment of reliability; test-score reliability",

keywords = "COEFFICIENT ALPHA, HISTORY, LOWER BOUNDS, SCORE, confidence intervals for reliability, precision of reported reliability, quality assessment of reliability, test-score reliability",

author = "P.R. Oosterwijk and {van der Ark}, L.A. and K. Sijtsma",

year = "2019",

doi = "10.1177/1073191117737375",

language = "English",

volume = "26",

pages = "1207--1216",

journal = "Assessment",

issn = "1073-1911",

publisher = "Sage Publications, Inc.",

number = "7",

}

TY - JOUR

T1 - Using confidence intervals for assessing reliability of real tests

AU - Oosterwijk, P.R.

AU - van der Ark, L.A.

AU - Sijtsma, K.

PY - 2019

Y1 - 2019

N2 - Test authors report sample reliability values but rarely consider the sampling error and related confidence intervals. This study investigated the truth of this conjecture for 116 tests with 1,024 reliability estimates (105 pertaining to test batteries and 919 to tests measuring a single attribute) obtained from an online database. Based on 90% confidence intervals, approximately 20% of the initial quality assessments had to be downgraded. For 95% confidence intervals, the percentage was approximately 23%. The results demonstrated that reported reliability values cannot be trusted without considering their estimation precision.Keywords: confidence intervals for reliability; precision of reported reliability; quality assessment of reliability; test-score reliability

AB - Test authors report sample reliability values but rarely consider the sampling error and related confidence intervals. This study investigated the truth of this conjecture for 116 tests with 1,024 reliability estimates (105 pertaining to test batteries and 919 to tests measuring a single attribute) obtained from an online database. Based on 90% confidence intervals, approximately 20% of the initial quality assessments had to be downgraded. For 95% confidence intervals, the percentage was approximately 23%. The results demonstrated that reported reliability values cannot be trusted without considering their estimation precision.Keywords: confidence intervals for reliability; precision of reported reliability; quality assessment of reliability; test-score reliability

KW - COEFFICIENT ALPHA

KW - HISTORY

KW - LOWER BOUNDS

KW - SCORE

KW - confidence intervals for reliability

KW - precision of reported reliability

KW - quality assessment of reliability

KW - test-score reliability

U2 - 10.1177/1073191117737375

DO - 10.1177/1073191117737375

M3 - Article

SN - 1073-1911

VL - 26

SP - 1207

EP - 1216

JO - Assessment

JF - Assessment

IS - 7

ER -

Using confidence intervals for assessing reliability of real tests

Abstract

Keywords

Access to Document

Fingerprint

Cite this