Linking language to the visual world: Neural correlates of comprehending verbal reference to objects through pointing and visual cues

David Peeters*, Tineke M. Snijders, Peter Hagoort, Asli Ozyurek

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

In everyday communication speakers often refer in speech and/or gesture to objects in their immediate environment, thereby shifting their addressee's attention to an intended referent. The neurobiological infrastructure involved in the comprehension of such basic multimodal communicative acts remains unclear. In an event-related fMRI study, we presented participants with pictures of a speaker and two objects while they concurrently listened to her speech. In each picture, one of the objects was singled out, either through the speaker's index-finger pointing gesture or through a visual cue that made the object perceptually more salient in the absence of gesture. A mismatch (compared to a match) between speech and the object singled out by the speaker's pointing gesture led to enhanced activation in left IFG and bilateral pMTG, showing the importance of these areas in conceptual matching between speech and referent. Moreover, a match (compared to a mismatch) between speech and the object made salient through a visual cue led to enhanced activation in the mentalizing system, arguably reflecting an attempt to converge on a jointly attended referent in the absence of pointing. These findings shed new light on the neurobiological underpinnings of the core communicative process of comprehending a speaker's multimodal referential act and stress the power of pointing as an important natural device to link speech to objects.

Original languageEnglish
Pages (from-to)21-29
Number of pages9
JournalNeuropsychologia
Volume95
DOIs
Publication statusPublished - 27 Jan 2017
Externally publishedYes

Keywords

  • Multimodal communication
  • Gesture
  • Pointing
  • Conceptual matching
  • Referential communication
  • Pragmatics
  • Mentalizing
  • SUPERIOR TEMPORAL SULCUS
  • CO-SPEECH GESTURES
  • SEMANTIC INFORMATION
  • BRAIN-REGIONS
  • INTEGRATION
  • GAZE
  • FMRI
  • COMMUNICATION
  • METAANALYSIS
  • ACTIVATIONS

Cite this

@article{b16312fba55f43aaadb8d137bf6035cd,
title = "Linking language to the visual world: Neural correlates of comprehending verbal reference to objects through pointing and visual cues",
abstract = "In everyday communication speakers often refer in speech and/or gesture to objects in their immediate environment, thereby shifting their addressee's attention to an intended referent. The neurobiological infrastructure involved in the comprehension of such basic multimodal communicative acts remains unclear. In an event-related fMRI study, we presented participants with pictures of a speaker and two objects while they concurrently listened to her speech. In each picture, one of the objects was singled out, either through the speaker's index-finger pointing gesture or through a visual cue that made the object perceptually more salient in the absence of gesture. A mismatch (compared to a match) between speech and the object singled out by the speaker's pointing gesture led to enhanced activation in left IFG and bilateral pMTG, showing the importance of these areas in conceptual matching between speech and referent. Moreover, a match (compared to a mismatch) between speech and the object made salient through a visual cue led to enhanced activation in the mentalizing system, arguably reflecting an attempt to converge on a jointly attended referent in the absence of pointing. These findings shed new light on the neurobiological underpinnings of the core communicative process of comprehending a speaker's multimodal referential act and stress the power of pointing as an important natural device to link speech to objects.",
keywords = "Multimodal communication, Gesture, Pointing, Conceptual matching, Referential communication, Pragmatics, Mentalizing, SUPERIOR TEMPORAL SULCUS, CO-SPEECH GESTURES, SEMANTIC INFORMATION, BRAIN-REGIONS, INTEGRATION, GAZE, FMRI, COMMUNICATION, METAANALYSIS, ACTIVATIONS",
author = "David Peeters and Snijders, {Tineke M.} and Peter Hagoort and Asli Ozyurek",
year = "2017",
month = "1",
day = "27",
doi = "10.1016/j.neuropsychologia.2016.12.004",
language = "English",
volume = "95",
pages = "21--29",
journal = "Neuropsychologia",
issn = "0028-3932",
publisher = "PERGAMON-ELSEVIER SCIENCE LTD",

}

Linking language to the visual world : Neural correlates of comprehending verbal reference to objects through pointing and visual cues. / Peeters, David; Snijders, Tineke M.; Hagoort, Peter; Ozyurek, Asli.

In: Neuropsychologia, Vol. 95, 27.01.2017, p. 21-29.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Linking language to the visual world

T2 - Neural correlates of comprehending verbal reference to objects through pointing and visual cues

AU - Peeters, David

AU - Snijders, Tineke M.

AU - Hagoort, Peter

AU - Ozyurek, Asli

PY - 2017/1/27

Y1 - 2017/1/27

N2 - In everyday communication speakers often refer in speech and/or gesture to objects in their immediate environment, thereby shifting their addressee's attention to an intended referent. The neurobiological infrastructure involved in the comprehension of such basic multimodal communicative acts remains unclear. In an event-related fMRI study, we presented participants with pictures of a speaker and two objects while they concurrently listened to her speech. In each picture, one of the objects was singled out, either through the speaker's index-finger pointing gesture or through a visual cue that made the object perceptually more salient in the absence of gesture. A mismatch (compared to a match) between speech and the object singled out by the speaker's pointing gesture led to enhanced activation in left IFG and bilateral pMTG, showing the importance of these areas in conceptual matching between speech and referent. Moreover, a match (compared to a mismatch) between speech and the object made salient through a visual cue led to enhanced activation in the mentalizing system, arguably reflecting an attempt to converge on a jointly attended referent in the absence of pointing. These findings shed new light on the neurobiological underpinnings of the core communicative process of comprehending a speaker's multimodal referential act and stress the power of pointing as an important natural device to link speech to objects.

AB - In everyday communication speakers often refer in speech and/or gesture to objects in their immediate environment, thereby shifting their addressee's attention to an intended referent. The neurobiological infrastructure involved in the comprehension of such basic multimodal communicative acts remains unclear. In an event-related fMRI study, we presented participants with pictures of a speaker and two objects while they concurrently listened to her speech. In each picture, one of the objects was singled out, either through the speaker's index-finger pointing gesture or through a visual cue that made the object perceptually more salient in the absence of gesture. A mismatch (compared to a match) between speech and the object singled out by the speaker's pointing gesture led to enhanced activation in left IFG and bilateral pMTG, showing the importance of these areas in conceptual matching between speech and referent. Moreover, a match (compared to a mismatch) between speech and the object made salient through a visual cue led to enhanced activation in the mentalizing system, arguably reflecting an attempt to converge on a jointly attended referent in the absence of pointing. These findings shed new light on the neurobiological underpinnings of the core communicative process of comprehending a speaker's multimodal referential act and stress the power of pointing as an important natural device to link speech to objects.

KW - Multimodal communication

KW - Gesture

KW - Pointing

KW - Conceptual matching

KW - Referential communication

KW - Pragmatics

KW - Mentalizing

KW - SUPERIOR TEMPORAL SULCUS

KW - CO-SPEECH GESTURES

KW - SEMANTIC INFORMATION

KW - BRAIN-REGIONS

KW - INTEGRATION

KW - GAZE

KW - FMRI

KW - COMMUNICATION

KW - METAANALYSIS

KW - ACTIVATIONS

U2 - 10.1016/j.neuropsychologia.2016.12.004

DO - 10.1016/j.neuropsychologia.2016.12.004

M3 - Article

VL - 95

SP - 21

EP - 29

JO - Neuropsychologia

JF - Neuropsychologia

SN - 0028-3932

ER -