The combined use of virtual reality and EEG to study language processing in naturalistic environments

Johanne Tromp*, David Peeters, Antje S. Meyer, Peter Hagoort

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

When we comprehend language, we often do this in rich settings where we can use many cues to understand what someone is saying. However, it has traditionally been difficult to design experiments with rich three-dimensional contexts that resemble our everyday environments, while maintaining control over the linguistic and nonlinguistic information that is available. Here we test the validity of combining electroencephalography (EEG) and virtual reality (VR) to overcome this problem. We recorded electrophysiological brain activity during language processing in a well-controlled three-dimensional virtual audiovisual environment. Participants were immersed in a virtual restaurant while wearing EEG equipment. In the restaurant, participants encountered virtual restaurant guests. Each guest was seated at a separate table with an object on it (e.g., a plate with salmon). The restaurant guest would then produce a sentence (e.g., "I just ordered this salmon."). The noun in the spoken sentence could either match ("salmon") or mismatch ("pasta") the object on the table, creating a situation in which the auditory information was either appropriate or inappropriate in the visual context. We observed a reliable N400 effect as a consequence of the mismatch. This finding validates the combined use of VR and EEG as a tool to study the neurophysiological mechanisms of everyday language comprehension in rich, ecologically valid settings.

Original languageEnglish
Pages (from-to)862-869
Number of pages8
JournalBehavior Research Methods
Volume50
Issue number2
DOIs
Publication statusPublished - Apr 2018
Externally publishedYes

Keywords

  • Language comprehension
  • Language processing
  • EEG
  • Virtual reality
  • N400
  • SEMANTIC INTEGRATION
  • WORLD EVENTS
  • SPEECH
  • COMPREHENSION
  • ERP
  • MODEL

Cite this

Tromp, Johanne ; Peeters, David ; Meyer, Antje S. ; Hagoort, Peter. / The combined use of virtual reality and EEG to study language processing in naturalistic environments. In: Behavior Research Methods. 2018 ; Vol. 50, No. 2. pp. 862-869.
@article{5ad2857160ee435f8a3706d24cf2b45a,
title = "The combined use of virtual reality and EEG to study language processing in naturalistic environments",
abstract = "When we comprehend language, we often do this in rich settings where we can use many cues to understand what someone is saying. However, it has traditionally been difficult to design experiments with rich three-dimensional contexts that resemble our everyday environments, while maintaining control over the linguistic and nonlinguistic information that is available. Here we test the validity of combining electroencephalography (EEG) and virtual reality (VR) to overcome this problem. We recorded electrophysiological brain activity during language processing in a well-controlled three-dimensional virtual audiovisual environment. Participants were immersed in a virtual restaurant while wearing EEG equipment. In the restaurant, participants encountered virtual restaurant guests. Each guest was seated at a separate table with an object on it (e.g., a plate with salmon). The restaurant guest would then produce a sentence (e.g., {"}I just ordered this salmon.{"}). The noun in the spoken sentence could either match ({"}salmon{"}) or mismatch ({"}pasta{"}) the object on the table, creating a situation in which the auditory information was either appropriate or inappropriate in the visual context. We observed a reliable N400 effect as a consequence of the mismatch. This finding validates the combined use of VR and EEG as a tool to study the neurophysiological mechanisms of everyday language comprehension in rich, ecologically valid settings.",
keywords = "Language comprehension, Language processing, EEG, Virtual reality, N400, SEMANTIC INTEGRATION, WORLD EVENTS, SPEECH, COMPREHENSION, ERP, MODEL",
author = "Johanne Tromp and David Peeters and Meyer, {Antje S.} and Peter Hagoort",
year = "2018",
month = "4",
doi = "10.3758/s13428-017-0911-9",
language = "English",
volume = "50",
pages = "862--869",
journal = "Behavior Research Methods",
issn = "1554-351X",
publisher = "Springer",
number = "2",

}

The combined use of virtual reality and EEG to study language processing in naturalistic environments. / Tromp, Johanne; Peeters, David; Meyer, Antje S.; Hagoort, Peter.

In: Behavior Research Methods, Vol. 50, No. 2, 04.2018, p. 862-869.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - The combined use of virtual reality and EEG to study language processing in naturalistic environments

AU - Tromp, Johanne

AU - Peeters, David

AU - Meyer, Antje S.

AU - Hagoort, Peter

PY - 2018/4

Y1 - 2018/4

N2 - When we comprehend language, we often do this in rich settings where we can use many cues to understand what someone is saying. However, it has traditionally been difficult to design experiments with rich three-dimensional contexts that resemble our everyday environments, while maintaining control over the linguistic and nonlinguistic information that is available. Here we test the validity of combining electroencephalography (EEG) and virtual reality (VR) to overcome this problem. We recorded electrophysiological brain activity during language processing in a well-controlled three-dimensional virtual audiovisual environment. Participants were immersed in a virtual restaurant while wearing EEG equipment. In the restaurant, participants encountered virtual restaurant guests. Each guest was seated at a separate table with an object on it (e.g., a plate with salmon). The restaurant guest would then produce a sentence (e.g., "I just ordered this salmon."). The noun in the spoken sentence could either match ("salmon") or mismatch ("pasta") the object on the table, creating a situation in which the auditory information was either appropriate or inappropriate in the visual context. We observed a reliable N400 effect as a consequence of the mismatch. This finding validates the combined use of VR and EEG as a tool to study the neurophysiological mechanisms of everyday language comprehension in rich, ecologically valid settings.

AB - When we comprehend language, we often do this in rich settings where we can use many cues to understand what someone is saying. However, it has traditionally been difficult to design experiments with rich three-dimensional contexts that resemble our everyday environments, while maintaining control over the linguistic and nonlinguistic information that is available. Here we test the validity of combining electroencephalography (EEG) and virtual reality (VR) to overcome this problem. We recorded electrophysiological brain activity during language processing in a well-controlled three-dimensional virtual audiovisual environment. Participants were immersed in a virtual restaurant while wearing EEG equipment. In the restaurant, participants encountered virtual restaurant guests. Each guest was seated at a separate table with an object on it (e.g., a plate with salmon). The restaurant guest would then produce a sentence (e.g., "I just ordered this salmon."). The noun in the spoken sentence could either match ("salmon") or mismatch ("pasta") the object on the table, creating a situation in which the auditory information was either appropriate or inappropriate in the visual context. We observed a reliable N400 effect as a consequence of the mismatch. This finding validates the combined use of VR and EEG as a tool to study the neurophysiological mechanisms of everyday language comprehension in rich, ecologically valid settings.

KW - Language comprehension

KW - Language processing

KW - EEG

KW - Virtual reality

KW - N400

KW - SEMANTIC INTEGRATION

KW - WORLD EVENTS

KW - SPEECH

KW - COMPREHENSION

KW - ERP

KW - MODEL

U2 - 10.3758/s13428-017-0911-9

DO - 10.3758/s13428-017-0911-9

M3 - Article

VL - 50

SP - 862

EP - 869

JO - Behavior Research Methods

JF - Behavior Research Methods

SN - 1554-351X

IS - 2

ER -