Recent studies have investigated the neural correlates of how we perceive emotions of individuals or a group of individuals using images of individual bodily expressions. However, it is still largely unknown how we perceive the emotion of a dynamic crowd. This fMRI study used realistic videos of a large group of people expressing fearful, happy or neutral emotions. Furthermore, the emotions were expressed by either unrelated individuals in the group or by an interacting group. It was hypothesized that the dynamics between the people in a crowd is a more salient signal than merely the emotion of the crowd. Second, it was expected that the group interaction is of special importance in a fearful or “panic” situation, as opposed to a happy or neutral situation. Using a fast-event related design, it was revealed that observing interactive individuals, more so than independently expressive individuals, activated networks related to the perception, execution and integration of action and emotion. Most importantly, the interactive or panicked crowds, as opposed to the individually fearful crowds, triggered more anticipatory and action preparation activity, whereas the brain was less sensitive to the dynamics of individuals in a happy or neutral crowd. This is the first study to assess the effect of the dynamics between people and the collectively displayed emotion as an important aspect of emotional crowd perception.