Using Deep Learning to Detect Facial Markers of Complex Decision Making

Gianluca Guglielmo, Irene Font Peradejordi, Michał Klincewicz

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review


In this paper, we report on an experiment with The Walking
Dead (TWD), which is a narrative-driven adventure game where players have to survive in a post-apocalyptic world filled with zombies. We
used OpenFace software to extract action unit (AU) intensities of facial expressions characteristic of decision-making processes and then we
implemented a simple convolution neural network (CNN) to see which
AUs are predictive of decision-making. More specifically, this study aims
to identify the facial regions that are predictive of decision-making. Our
results provide evidence that the pre-decision variations in action units
17 (chin raiser), 23 (lip tightener), and 25 (parting of lips) are predictive of decision-making processes. Furthermore, when combined, their
predictive power increased up to 0.81 accuracy on the test set; we offer
speculations about why it is that these particular three AUs were found
to be connected to decision-making. Our results also suggest that machine learning methods in combination with video games may be used
to accurately and automatically identify complex decision-making processes using AU intensity alone. Finally, our study offers a new method
to test specific hypotheses about the relationships between higher-order
cognitive processes and behavior, which relies on both narrative video
games and easily accessible software, like OpenFace.
Original languageEnglish
Title of host publicationAdvances in Computer Games 2021
Number of pages10
Publication statusAccepted/In press - 23 Nov 2021


  • Video Games
  • Decision-Making
  • Facial Expression
  • Facial Action Coding System (FACS)
  • Machine Learning
  • Deep Learning


Dive into the research topics of 'Using Deep Learning to Detect Facial Markers of Complex Decision Making'. Together they form a unique fingerprint.

Cite this