Measure only what is measurable: Towards conversation requirements for evaluating task-oriented dialogue systems

Emiel Van Miltenburg; Anouck Braggaar; Emmelyn Croes; Florian Kunneman; Christine Liebrecht; Gabriella Martijn

Measure only what is measurable: Towards conversation requirements for evaluating task-oriented dialogue systems

Emiel Van Miltenburg
, Anouck Braggaar
, Emmelyn Croes
, Florian Kunneman
, Christine Liebrecht
, Gabriella Martijn

Communication and Cognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

5 Downloads (Pure)

Abstract

Chatbots for customer service have been widely studied in many different fields, ranging from Natural Language Processing (NLP) to Communication Science. These fields have developed different evaluation practices to assess chatbot performance (e.g., fluency, task success) and to measure the impact of chatbot usage on the user's perception of the organisation controlling the chatbot (e.g., brand attitude) as well as their willingness to enter a business transaction or to continue to use the chatbot in the future (i.e., purchase intention, reuse intention). While NLP researchers have developed many automatic measures of success, other fields mainly use questionnaires to compare different chatbots. This paper explores the extent to which we can bridge the gap between the two, and proposes a research agenda to further explore this question.

Original language	English
Title of host publication	Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²)
Editors	Kaustubh Dhole, Miruna Clinciu
Place of Publication	Vienna, Austria and virtual meeting
Publisher	Association for Computational Linguistics
Pages	231-238
Number of pages	8
ISBN (Print)	979-8-89176-261-9
Publication status	Published - 1 Jul 2025
Event	4th Workshop on Generation, Evaluation, and Metrics - The Austria Center Vienna (hybrid), Vienna, Austria Duration: 31 Jul 2025 → 1 Aug 2025 https://gem-benchmark.com/workshop

Conference

Conference	4th Workshop on Generation, Evaluation, and Metrics
Abbreviated title	GEM 2025
Country/Territory	Austria
City	Vienna
Period	31/07/25 → 1/08/25
Internet address	https://gem-benchmark.com/workshop

Keywords

natural language generation
evaluation metrics
prompt engineering
benchmarking
model assessment

Access to Document

Measure only what is measurable - Miltenburg et al.,Final published version, 114 KBLicence: CC BY

https://aclanthology.org/2025.gem-1.18/Licence: CC BY

Smooth Operator. Development and effects of personalized conversational AI.
Liebrecht, C. (Principal Investigator), van Hooijdonk, C. M. J. (CoPI), Krahmer, E. (CoPI), van Miltenburg, E. (CoPI), Kunneman, F. (CoPI), Hoeken, H. (CoPI) & te Molder, H. (CoPI)
31/03/21 → 31/03/25
Project: Research project

Cite this

Van Miltenburg, E., Braggaar, A., Croes, E., Kunneman, F., Liebrecht, C., & Martijn, G. (2025). Measure only what is measurable: Towards conversation requirements for evaluating task-oriented dialogue systems. In K. Dhole, & M. Clinciu (Eds.), Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²) (pp. 231-238). Association for Computational Linguistics. https://aclanthology.org/2025.gem-1.18/

Van Miltenburg, Emiel ; Braggaar, Anouck ; Croes, Emmelyn et al. / Measure only what is measurable : Towards conversation requirements for evaluating task-oriented dialogue systems. Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²). editor / Kaustubh Dhole ; Miruna Clinciu. Vienna, Austria and virtual meeting : Association for Computational Linguistics, 2025. pp. 231-238

@inproceedings{cfa9a956f20e48599de45c9ddcc60fd4,

title = "Measure only what is measurable: Towards conversation requirements for evaluating task-oriented dialogue systems",

abstract = "Chatbots for customer service have been widely studied in many different fields, ranging from Natural Language Processing (NLP) to Communication Science. These fields have developed different evaluation practices to assess chatbot performance (e.g., fluency, task success) and to measure the impact of chatbot usage on the user's perception of the organisation controlling the chatbot (e.g., brand attitude) as well as their willingness to enter a business transaction or to continue to use the chatbot in the future (i.e., purchase intention, reuse intention). While NLP researchers have developed many automatic measures of success, other fields mainly use questionnaires to compare different chatbots. This paper explores the extent to which we can bridge the gap between the two, and proposes a research agenda to further explore this question.",

keywords = "natural language generation, evaluation metrics, prompt engineering, benchmarking, model assessment",

author = "\{Van Miltenburg\}, Emiel and Anouck Braggaar and Emmelyn Croes and Florian Kunneman and Christine Liebrecht and Gabriella Martijn",

year = "2025",

month = jul,

day = "1",

language = "English",

isbn = "979-8-89176-261-9",

pages = "231--238",

editor = "Kaustubh Dhole and Miruna Clinciu",

booktitle = "Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²)",

publisher = "Association for Computational Linguistics",

note = "4th Workshop on Generation, Evaluation, and Metrics , GEM 2025 ; Conference date: 31-07-2025 Through 01-08-2025",

url = "https://gem-benchmark.com/workshop",

}

Van Miltenburg, E , Braggaar, A , Croes, E, Kunneman, F, Liebrecht, C & Martijn, G 2025, Measure only what is measurable: Towards conversation requirements for evaluating task-oriented dialogue systems. in K Dhole & M Clinciu (eds), Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²). Association for Computational Linguistics, Vienna, Austria and virtual meeting, pp. 231-238, 4th Workshop on Generation, Evaluation, and Metrics , Vienna, Austria, 31/07/25. <https://aclanthology.org/2025.gem-1.18/>

Measure only what is measurable: Towards conversation requirements for evaluating task-oriented dialogue systems. / Van Miltenburg, Emiel ; Braggaar, Anouck ; Croes, Emmelyn et al.
Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²). ed. / Kaustubh Dhole; Miruna Clinciu. Vienna, Austria and virtual meeting: Association for Computational Linguistics, 2025. p. 231-238.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Measure only what is measurable

T2 - 4th Workshop on Generation, Evaluation, and Metrics

AU - Van Miltenburg, Emiel

AU - Braggaar, Anouck

AU - Croes, Emmelyn

AU - Kunneman, Florian

AU - Liebrecht, Christine

AU - Martijn, Gabriella

PY - 2025/7/1

Y1 - 2025/7/1

N2 - Chatbots for customer service have been widely studied in many different fields, ranging from Natural Language Processing (NLP) to Communication Science. These fields have developed different evaluation practices to assess chatbot performance (e.g., fluency, task success) and to measure the impact of chatbot usage on the user's perception of the organisation controlling the chatbot (e.g., brand attitude) as well as their willingness to enter a business transaction or to continue to use the chatbot in the future (i.e., purchase intention, reuse intention). While NLP researchers have developed many automatic measures of success, other fields mainly use questionnaires to compare different chatbots. This paper explores the extent to which we can bridge the gap between the two, and proposes a research agenda to further explore this question.

AB - Chatbots for customer service have been widely studied in many different fields, ranging from Natural Language Processing (NLP) to Communication Science. These fields have developed different evaluation practices to assess chatbot performance (e.g., fluency, task success) and to measure the impact of chatbot usage on the user's perception of the organisation controlling the chatbot (e.g., brand attitude) as well as their willingness to enter a business transaction or to continue to use the chatbot in the future (i.e., purchase intention, reuse intention). While NLP researchers have developed many automatic measures of success, other fields mainly use questionnaires to compare different chatbots. This paper explores the extent to which we can bridge the gap between the two, and proposes a research agenda to further explore this question.

KW - natural language generation

KW - evaluation metrics

KW - prompt engineering

KW - benchmarking

KW - model assessment

M3 - Conference contribution

SN - 979-8-89176-261-9

SP - 231

EP - 238

BT - Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²)

A2 - Dhole, Kaustubh

A2 - Clinciu, Miruna

PB - Association for Computational Linguistics

CY - Vienna, Austria and virtual meeting

Y2 - 31 July 2025 through 1 August 2025

ER -

Van Miltenburg E , Braggaar A , Croes E, Kunneman F, Liebrecht C, Martijn G. Measure only what is measurable: Towards conversation requirements for evaluating task-oriented dialogue systems. In Dhole K, Clinciu M, editors, Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²). Vienna, Austria and virtual meeting: Association for Computational Linguistics. 2025. p. 231-238

Measure only what is measurable: Towards conversation requirements for evaluating task-oriented dialogue systems

Abstract

Conference

Keywords

Access to Document

Fingerprint

Projects

Smooth Operator. Development and effects of personalized conversational AI.

Cite this