How useful are corpus-based methods for extrapolating psycholinguistic variables?

Pawel Mandera*, Emmanuel Keuleers, Marc Brysbaert

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

28 Citations (Scopus)

Abstract

Subjective ratings for age of acquisition, concreteness, affective valence, and many other variables are an important element of psycholinguistic research. However, even for well-studied languages, ratings usually cover just a small part of the vocabulary. A possible solution involves using corpora to build a semantic similarity space and to apply machine learning techniques to extrapolate existing ratings to previously unrated words. We conduct a systematic comparison of two extrapolation techniques: k-nearest neighbours, and random forest, in combination with semantic spaces built using latent semantic analysis, topic model, a hyperspace analogue to language (HAL)-like model, and a skip-gram model. A variant of the k-nearest neighbours method used with skip-gram word vectors gives the most accurate predictions but the random forest method has an advantage of being able to easily incorporate additional predictors. We evaluate the usefulness of the methods by exploring how much of the human performance in a lexical decision task can be explained by extrapolated ratings for age of acquisition and how precisely we can assign words to discrete categories based on extrapolated ratings. We find that at least some of the extrapolation methods may introduce artefacts to the data and produce results that could lead to different conclusions that would be reached based on the human ratings. From a practical point of view, the usefulness of ratings extrapolated with the described methods may be limited.

Original languageEnglish
Pages (from-to)1623-1642
Number of pages20
JournalThe Quarterly Journal of Experimental Psychology
Volume68
Issue number8
DOIs
Publication statusPublished - 3 Aug 2015
Externally publishedYes

Keywords

  • Semantic models
  • Human ratings
  • Machine learning
  • LATENT SEMANTIC ANALYSIS
  • AGE-OF-ACQUISITION
  • LEXICAL COOCCURRENCE
  • ENGLISH WORDS
  • NORMS
  • CONCRETENESS
  • PROJECT
  • RATINGS
  • MODELS
  • LEMMAS

Cite this