Evolving chatbots, shifted perceptions: A near-replication of Liebrecht and Van der Weegen (2019)

We performed a replication study of Liebrecht and Van der Weegen on the effects of conversational human voice (CHV) and brand familiarity in chatbot conversations on social presence and brand attitude. Additionally the effect of CHV on perceptions of warmth and competence was examined. The first goal was to determine if the findings of the original study still hold, given rapid developments in chatbot technology. The study was partially replicated, with generally smaller effect sizes; employment of CHV in chatbot messages resulted in increased social presence and consequently increased brand attitude. Moreover, no significant effect of brand familiarity was found, and no significant total effect of competence. Also, no significant total effect on warmth was found. Our second goal was to explore factors researchers should consider when replicating chatbot studies and enhancing replicability of new research. Based on our experiences, we provide a checklist for future chatbot research.