(Pea)nuts and bolts of visual narrative

structure and meaning in sequential image comprehension

Neil Cohn, Martin Paczynski, Ray Jackendoff, Phillip J Holcomb, Gina R Kuperberg

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.

Original languageEnglish
Pages (from-to)1-38
Number of pages38
JournalCognitive Psychology
Volume65
Issue number1
DOIs
Publication statusPublished - Aug 2012
Externally publishedYes

Fingerprint

Nuts (fasteners)
Peas
Bolts
comprehension
Semantics
semantics
narrative
Processing
experiment
Cognitive systems
Enterprise resource planning
Experiments
syntax
Language

Keywords

  • narrative
  • comics
  • visual language
  • discoruse
  • grammar
  • Event related potentials
  • cognitive neuroscience
  • Linguistics
  • Visual Perception

Cite this

Cohn, Neil ; Paczynski, Martin ; Jackendoff, Ray ; Holcomb, Phillip J ; Kuperberg, Gina R. / (Pea)nuts and bolts of visual narrative : structure and meaning in sequential image comprehension. In: Cognitive Psychology. 2012 ; Vol. 65, No. 1. pp. 1-38.
@article{9a77eb6d31e64ed084a982df9d07f5cf,
title = "(Pea)nuts and bolts of visual narrative: structure and meaning in sequential image comprehension",
abstract = "Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.",
keywords = "narrative, comics, visual language , discoruse, grammar, Event related potentials, cognitive neuroscience, Linguistics, Visual Perception",
author = "Neil Cohn and Martin Paczynski and Ray Jackendoff and Holcomb, {Phillip J} and Kuperberg, {Gina R}",
note = "Copyright {\circledC} 2012 Elsevier Inc. All rights reserved.",
year = "2012",
month = "8",
doi = "10.1016/j.cogpsych.2012.01.003",
language = "English",
volume = "65",
pages = "1--38",
journal = "Cognitive Psychology",
issn = "0010-0285",
publisher = "ACADEMIC PRESS INC ELSEVIER SCIENCE",
number = "1",

}

(Pea)nuts and bolts of visual narrative : structure and meaning in sequential image comprehension. / Cohn, Neil; Paczynski, Martin; Jackendoff, Ray; Holcomb, Phillip J; Kuperberg, Gina R.

In: Cognitive Psychology, Vol. 65, No. 1, 08.2012, p. 1-38.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - (Pea)nuts and bolts of visual narrative

T2 - structure and meaning in sequential image comprehension

AU - Cohn, Neil

AU - Paczynski, Martin

AU - Jackendoff, Ray

AU - Holcomb, Phillip J

AU - Kuperberg, Gina R

N1 - Copyright © 2012 Elsevier Inc. All rights reserved.

PY - 2012/8

Y1 - 2012/8

N2 - Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.

AB - Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.

KW - narrative

KW - comics

KW - visual language

KW - discoruse

KW - grammar

KW - Event related potentials

KW - cognitive neuroscience

KW - Linguistics

KW - Visual Perception

U2 - 10.1016/j.cogpsych.2012.01.003

DO - 10.1016/j.cogpsych.2012.01.003

M3 - Article

VL - 65

SP - 1

EP - 38

JO - Cognitive Psychology

JF - Cognitive Psychology

SN - 0010-0285

IS - 1

ER -