What question characteristics are related to comprehension problems in political attitude questions? And what type of answering behaviour do people expose when they do not understand the question? We investigated these issues in the context of Voting Advice Applications (VAAs). These online tools provide users with a voting advice based on their answers to a set of about 30 political attitude questions. VAAs have become a central source of political information (see Garzia & Marschall, 2012) and research shows that the VAA voting advice affects the content of the vote cast (e.g., Andreadis & Wall, 2014). Therefore, it is of utmost importance to investigate to what extent VAA users understand the questions that lead to the voting advice and how they respond in case of comprehension difficulties. Study 1 consists of cognitive interviews with 60 users, each one filling out 30 VAA statements prior to 2014 municipal elections in the Dutch Municipality Utrecht. The verbalizations of these respondents were recorded and categorized for several types of comprehension problems by two independent coders (Kappa/Kappa max between 0.58 and 0.98). Results show that VAA users encounter a comprehension problem for – on average - about 1 in 5 questions. About two-thirds of these are related to the semantic meaning of the question, covering difficulties with political jargon (e.g., 'dog tax'), or geographical terms (e.g., a specific street in Utrecht). One-third of the comprehension problems are related to the pragmatic comprehension-about the question. In these cases, the respondent does understand the literal meaning of all concepts in the question, but lacks contextual knowledge for providing a well-considered answer. Such pragmatic comprehension problems are often triggered by vague quantifying term in the question (e.g., taxes on housing should be raised), which make the users realize they lack knowledge about the current state of affairs ('How high is that tax now?'). In case of comprehension problems, VAA users often assume a certain question meaning, and hardly ever proceed by looking for information on the web. Nevertheless, a large majority of the respondents provides a substantive answer (often the middle option). In Study 2, we investigated whether the question characteristics leading to comprehension difficulties in Study 1, lead to more neutral and no opinion-answers when statistically analyzed across a larger set of questions in a larger set of VAAs. We performed statistical analyses of all answers provided by 357,858 VAA respondents who used one of the 34 different municipal VAAs during the Dutch municipality elections in 2014. Results in Study 2 confirm that political jargon, geographical locations and vague quantifying terms are related to more neutral and/or no opinion-answers. Interestingly, there seems to be a relation between the type of comprehension problem and the type of answer provided: semantic meaning problems often result in no opinion answers, whereas pragmatic problems are related to neutral responses.
|Publication status||Published - 2017|
|Event||Conference of the European Survey Research Association - Lisbon, Portugal|
Duration: 17 Jul 2017 → 21 Jul 2017
Conference number: 7
|Conference||Conference of the European Survey Research Association|
|Period||17/07/17 → 21/07/17|
Kamoen, N., & Holleman, B. (2017). I don't get it: Response difficulties in answering political attitude statements in Voting Advice Applications. Abstract from Conference of the European Survey Research Association, Lisbon, Portugal.