The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension

Neil Cohn, Ray Jackendoff, Phillip J Holcomb, Gina R Kuperberg

Research output: Contribution to journalArticleScientificpeer-review

38 Citations (Scopus)

Abstract

Constituent structure has long been established as a central feature of human language. Analogous to how syntax organizes words in sentences, a narrative grammar organizes sequential images into hierarchic constituents. Here we show that the brain draws upon this constituent structure to comprehend wordless visual narratives. We recorded neural responses as participants viewed sequences of visual images (comics strips) in which blank images either disrupted individual narrative constituents or fell at natural constituent boundaries. A disruption of either the first or the second narrative constituent produced a left-lateralized anterior negativity effect between 500 and 700ms. Disruption of the second constituent also elicited a posteriorly-distributed positivity (P600) effect. These neural responses are similar to those associated with structural violations in language and music. These findings provide evidence that comprehenders use a narrative structure to comprehend visual sequences and that the brain engages similar neurocognitive mechanisms to build structure across multiple domains.

Original languageEnglish
Pages (from-to)63-70
Number of pages8
JournalNeuropsychologia
Volume64
DOIs
Publication statusPublished - Nov 2014
Externally publishedYes

Keywords

  • grammar
  • prediction
  • narrative
  • visual language
  • comics
  • discourse

Fingerprint Dive into the research topics of 'The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension'. Together they form a unique fingerprint.

  • Cite this