Varying image description tasks: spoken versus written descriptions

Automatic image description systems are commonly trained and evaluated on written image descriptions. At the same time, these systems are often used to provide spoken descriptions (e.g., for visually impaired users) through apps like TapTapSee or Seeing AI. This is not a problem, as long as spoken and written descriptions are very similar. However, linguistic research suggests that spoken language often differs from written language. These differences are not regular and vary from context to context. Therefore, this paper investigates whether there are differences between written and spoken image descriptions, even if they are elicited through similar tasks. We compared descriptions produced in two languages (English and Dutch) and found substantial differences between spoken and written descriptions in both languages. Future research should examine if users prefer the spoken over the written style and, if so, aim to emulate spoken descriptions.