Using Deep Learning to Detect Facial Markers of Complex Decision Making

In this paper, we report on an experiment with The Walking
Dead (TWD), which is a narrative-driven adventure game where players have to survive in a post-apocalyptic world filled with zombies. We
used OpenFace software to extract action unit (AU) intensities of facial expressions characteristic of decision-making processes and then we
implemented a simple convolution neural network (CNN) to see which
AUs are predictive of decision-making. More specifically, this study aims
to identify the facial regions that are predictive of decision-making. Our
results provide evidence that the pre-decision variations in action units
17 (chin raiser), 23 (lip tightener), and 25 (parting of lips) are predictive of decision-making processes. Furthermore, when combined, their
predictive power increased up to 0.81 accuracy on the test set; we offer
speculations about why it is that these particular three AUs were found
to be connected to decision-making. Our results also suggest that machine learning methods in combination with video games may be used
to accurately and automatically identify complex decision-making processes using AU intensity alone. Finally, our study offers a new method
to test specific hypotheses about the relationships between higher-order
cognitive processes and behavior, which relies on both narrative video
games and easily accessible software, like OpenFace.