Sequences in harmony: Cognitive interactions between musical and visual narrative structure

Morgan T. Patrick, Neil Cohn, John Mertus, Sheila E. Blumstein

Research output: Contribution to journalArticleScientificpeer-review

Abstract

From film and television to graphic storytelling, tonal music can accompany visual narratives in a variety of contexts. The apprehension of both musical and narrative sequences involves temporal categories in ordered patterning, which raises an interesting question: Do musical progressions and visual narratives rely on shared sequence processing mechanisms? If this is the case, then cues from music and sequential static images, when presented simultaneously, should interact during audiovisual online processing. We tested this question by measuring reaction times to target picture panels appearing in visual narrative (comic strip) sequences, which were presented panel by panel and synchronized with musical chord progressions. Image sequences were either coherent narratives or incoherent (random) panels, and they were aligned with musical accompaniment consisting of coherent tonal chord progressions or non-tonal (unrelated) chords. Reaction times were faster for target images in coherent sequences than incoherent sequences, and even faster for coherent images with tonal accompaniment than non-tonal chords. This indicated an interaction between sequential structures across domains. We take these results as evidence for a shared, domain-general sequence processing mechanism operating across music and visual narrative.
Original languageEnglish
Article number103981
Number of pages9
JournalActa Psychologica
Volume238
Early online dateJul 2023
DOIs
Publication statusPublished - Aug 2023

Keywords

  • Domain-generality
  • Sequence processing
  • Tonality
  • Visual narrative

Fingerprint

Dive into the research topics of 'Sequences in harmony: Cognitive interactions between musical and visual narrative structure'. Together they form a unique fingerprint.

Cite this