Context-based Language Generation in Multi-modal Human-Computer Interaction

This project deals with language generation as a part of the development of multimodal human-computer dialogue systems based on a combination of natural language and direct manipulation. Such a system should be able to generate utterances using linguistic and visual modalities. Especially two questions are considered here: 1) in what way is the generation of utterances directed by the cognitive-semantic context and the multimodal environment and 2) which aspects determine the use of a linguistic versus, or in combination with the visual modality.