Using Deep Learning to Detect Facial Markers of Complex Decision Making

Gianluca Guglielmo, Irene Font Peradejordi, Michał Klincewicz

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Abstract

In this paper, we report on an experiment with The Walking
Dead (TWD), which is a narrative-driven adventure game where players have to survive in a post-apocalyptic world filled with zombies. We
used OpenFace software to extract action unit (AU) intensities of facial expressions characteristic of decision-making processes and then we
implemented a simple convolution neural network (CNN) to see which
AUs are predictive of decision-making. More specifically, this study aims
to identify the facial regions that are predictive of decision-making. Our
results provide evidence that the pre-decision variations in action units
17 (chin raiser), 23 (lip tightener), and 25 (parting of lips) are predictive of decision-making processes. Furthermore, when combined, their
predictive power increased up to 0.81 accuracy on the test set; we offer
speculations about why it is that these particular three AUs were found
to be connected to decision-making. Our results also suggest that machine learning methods in combination with video games may be used
to accurately and automatically identify complex decision-making processes using AU intensity alone. Finally, our study offers a new method
to test specific hypotheses about the relationships between higher-order
cognitive processes and behavior, which relies on both narrative video
games and easily accessible software, like OpenFace.
Original languageEnglish
Title of host publicationAdvances in Computer Games 2021
PublisherSpringer
Pages1-10
Number of pages10
Edition2021
Publication statusAccepted/In press - 23 Nov 2021

Keywords

  • Video Games
  • Decision-Making
  • Facial Expression
  • Facial Action Coding System (FACS)
  • Machine Learning
  • Deep Learning

Fingerprint

Dive into the research topics of 'Using Deep Learning to Detect Facial Markers of Complex Decision Making'. Together they form a unique fingerprint.

Cite this