I don't get it

Response difficulties in answering political attitude statements in Voting Advice Applications

Naomi Kamoen, Bregje Holleman

Research output: Contribution to conferenceAbstractOther research output

Abstract

What question characteristics are related to comprehension problems in political attitude questions? And what type of answering behaviour do people expose when they do not understand the question? We investigated these issues in the context of Voting Advice Applications (VAAs). These online tools provide users with a voting advice based on their answers to a set of about 30 political attitude questions. VAAs have become a central source of political information (see Garzia & Marschall, 2012) and research shows that the VAA voting advice affects the content of the vote cast (e.g., Andreadis & Wall, 2014). Therefore, it is of utmost importance to investigate to what extent VAA users understand the questions that lead to the voting advice and how they respond in case of comprehension difficulties. Study 1 consists of cognitive interviews with 60 users, each one filling out 30 VAA statements prior to 2014 municipal elections in the Dutch Municipality Utrecht. The verbalizations of these respondents were recorded and categorized for several types of comprehension problems by two independent coders (Kappa/Kappa max between 0.58 and 0.98). Results show that VAA users encounter a comprehension problem for – on average - about 1 in 5 questions. About two-thirds of these are related to the semantic meaning of the question, covering difficulties with political jargon (e.g., 'dog tax'), or geographical terms (e.g., a specific street in Utrecht). One-third of the comprehension problems are related to the pragmatic comprehension-about the question. In these cases, the respondent does understand the literal meaning of all concepts in the question, but lacks contextual knowledge for providing a well-considered answer. Such pragmatic comprehension problems are often triggered by vague quantifying term in the question (e.g., taxes on housing should be raised), which make the users realize they lack knowledge about the current state of affairs ('How high is that tax now?'). In case of comprehension problems, VAA users often assume a certain question meaning, and hardly ever proceed by looking for information on the web. Nevertheless, a large majority of the respondents provides a substantive answer (often the middle option). In Study 2, we investigated whether the question characteristics leading to comprehension difficulties in Study 1, lead to more neutral and no opinion-answers when statistically analyzed across a larger set of questions in a larger set of VAAs. We performed statistical analyses of all answers provided by 357,858 VAA respondents who used one of the 34 different municipal VAAs during the Dutch municipality elections in 2014. Results in Study 2 confirm that political jargon, geographical locations and vague quantifying terms are related to more neutral and/or no opinion-answers. Interestingly, there seems to be a relation between the type of comprehension problem and the type of answer provided: semantic meaning problems often result in no opinion answers, whereas pragmatic problems are related to neutral responses.
Original languageEnglish
Publication statusPublished - 2017
EventConference of the European Survey Research Association - Lisbon, Portugal
Duration: 17 Jul 201721 Jul 2017
Conference number: 7
https://www.europeansurveyresearch.org/

Conference

ConferenceConference of the European Survey Research Association
Abbreviated titleESRA
CountryPortugal
CityLisbon
Period17/07/1721/07/17
Internet address

Fingerprint

political attitude
voting
comprehension
pragmatics
taxes
municipality
election
semantics
lack
voter
housing

Cite this

Kamoen, N., & Holleman, B. (2017). I don't get it: Response difficulties in answering political attitude statements in Voting Advice Applications. Abstract from Conference of the European Survey Research Association, Lisbon, Portugal.
Kamoen, Naomi ; Holleman, Bregje. / I don't get it : Response difficulties in answering political attitude statements in Voting Advice Applications. Abstract from Conference of the European Survey Research Association, Lisbon, Portugal.
@conference{bd209af5d6b44c16b66ceadc71485180,
title = "I don't get it: Response difficulties in answering political attitude statements in Voting Advice Applications",
abstract = "What question characteristics are related to comprehension problems in political attitude questions? And what type of answering behaviour do people expose when they do not understand the question? We investigated these issues in the context of Voting Advice Applications (VAAs). These online tools provide users with a voting advice based on their answers to a set of about 30 political attitude questions. VAAs have become a central source of political information (see Garzia & Marschall, 2012) and research shows that the VAA voting advice affects the content of the vote cast (e.g., Andreadis & Wall, 2014). Therefore, it is of utmost importance to investigate to what extent VAA users understand the questions that lead to the voting advice and how they respond in case of comprehension difficulties. Study 1 consists of cognitive interviews with 60 users, each one filling out 30 VAA statements prior to 2014 municipal elections in the Dutch Municipality Utrecht. The verbalizations of these respondents were recorded and categorized for several types of comprehension problems by two independent coders (Kappa/Kappa max between 0.58 and 0.98). Results show that VAA users encounter a comprehension problem for – on average - about 1 in 5 questions. About two-thirds of these are related to the semantic meaning of the question, covering difficulties with political jargon (e.g., 'dog tax'), or geographical terms (e.g., a specific street in Utrecht). One-third of the comprehension problems are related to the pragmatic comprehension-about the question. In these cases, the respondent does understand the literal meaning of all concepts in the question, but lacks contextual knowledge for providing a well-considered answer. Such pragmatic comprehension problems are often triggered by vague quantifying term in the question (e.g., taxes on housing should be raised), which make the users realize they lack knowledge about the current state of affairs ('How high is that tax now?'). In case of comprehension problems, VAA users often assume a certain question meaning, and hardly ever proceed by looking for information on the web. Nevertheless, a large majority of the respondents provides a substantive answer (often the middle option). In Study 2, we investigated whether the question characteristics leading to comprehension difficulties in Study 1, lead to more neutral and no opinion-answers when statistically analyzed across a larger set of questions in a larger set of VAAs. We performed statistical analyses of all answers provided by 357,858 VAA respondents who used one of the 34 different municipal VAAs during the Dutch municipality elections in 2014. Results in Study 2 confirm that political jargon, geographical locations and vague quantifying terms are related to more neutral and/or no opinion-answers. Interestingly, there seems to be a relation between the type of comprehension problem and the type of answer provided: semantic meaning problems often result in no opinion answers, whereas pragmatic problems are related to neutral responses.",
author = "Naomi Kamoen and Bregje Holleman",
year = "2017",
language = "English",
note = "Conference of the European Survey Research Association, ESRA ; Conference date: 17-07-2017 Through 21-07-2017",
url = "https://www.europeansurveyresearch.org/",

}

Kamoen, N & Holleman, B 2017, 'I don't get it: Response difficulties in answering political attitude statements in Voting Advice Applications' Conference of the European Survey Research Association, Lisbon, Portugal, 17/07/17 - 21/07/17, .

I don't get it : Response difficulties in answering political attitude statements in Voting Advice Applications. / Kamoen, Naomi; Holleman, Bregje.

2017. Abstract from Conference of the European Survey Research Association, Lisbon, Portugal.

Research output: Contribution to conferenceAbstractOther research output

TY - CONF

T1 - I don't get it

T2 - Response difficulties in answering political attitude statements in Voting Advice Applications

AU - Kamoen, Naomi

AU - Holleman, Bregje

PY - 2017

Y1 - 2017

N2 - What question characteristics are related to comprehension problems in political attitude questions? And what type of answering behaviour do people expose when they do not understand the question? We investigated these issues in the context of Voting Advice Applications (VAAs). These online tools provide users with a voting advice based on their answers to a set of about 30 political attitude questions. VAAs have become a central source of political information (see Garzia & Marschall, 2012) and research shows that the VAA voting advice affects the content of the vote cast (e.g., Andreadis & Wall, 2014). Therefore, it is of utmost importance to investigate to what extent VAA users understand the questions that lead to the voting advice and how they respond in case of comprehension difficulties. Study 1 consists of cognitive interviews with 60 users, each one filling out 30 VAA statements prior to 2014 municipal elections in the Dutch Municipality Utrecht. The verbalizations of these respondents were recorded and categorized for several types of comprehension problems by two independent coders (Kappa/Kappa max between 0.58 and 0.98). Results show that VAA users encounter a comprehension problem for – on average - about 1 in 5 questions. About two-thirds of these are related to the semantic meaning of the question, covering difficulties with political jargon (e.g., 'dog tax'), or geographical terms (e.g., a specific street in Utrecht). One-third of the comprehension problems are related to the pragmatic comprehension-about the question. In these cases, the respondent does understand the literal meaning of all concepts in the question, but lacks contextual knowledge for providing a well-considered answer. Such pragmatic comprehension problems are often triggered by vague quantifying term in the question (e.g., taxes on housing should be raised), which make the users realize they lack knowledge about the current state of affairs ('How high is that tax now?'). In case of comprehension problems, VAA users often assume a certain question meaning, and hardly ever proceed by looking for information on the web. Nevertheless, a large majority of the respondents provides a substantive answer (often the middle option). In Study 2, we investigated whether the question characteristics leading to comprehension difficulties in Study 1, lead to more neutral and no opinion-answers when statistically analyzed across a larger set of questions in a larger set of VAAs. We performed statistical analyses of all answers provided by 357,858 VAA respondents who used one of the 34 different municipal VAAs during the Dutch municipality elections in 2014. Results in Study 2 confirm that political jargon, geographical locations and vague quantifying terms are related to more neutral and/or no opinion-answers. Interestingly, there seems to be a relation between the type of comprehension problem and the type of answer provided: semantic meaning problems often result in no opinion answers, whereas pragmatic problems are related to neutral responses.

AB - What question characteristics are related to comprehension problems in political attitude questions? And what type of answering behaviour do people expose when they do not understand the question? We investigated these issues in the context of Voting Advice Applications (VAAs). These online tools provide users with a voting advice based on their answers to a set of about 30 political attitude questions. VAAs have become a central source of political information (see Garzia & Marschall, 2012) and research shows that the VAA voting advice affects the content of the vote cast (e.g., Andreadis & Wall, 2014). Therefore, it is of utmost importance to investigate to what extent VAA users understand the questions that lead to the voting advice and how they respond in case of comprehension difficulties. Study 1 consists of cognitive interviews with 60 users, each one filling out 30 VAA statements prior to 2014 municipal elections in the Dutch Municipality Utrecht. The verbalizations of these respondents were recorded and categorized for several types of comprehension problems by two independent coders (Kappa/Kappa max between 0.58 and 0.98). Results show that VAA users encounter a comprehension problem for – on average - about 1 in 5 questions. About two-thirds of these are related to the semantic meaning of the question, covering difficulties with political jargon (e.g., 'dog tax'), or geographical terms (e.g., a specific street in Utrecht). One-third of the comprehension problems are related to the pragmatic comprehension-about the question. In these cases, the respondent does understand the literal meaning of all concepts in the question, but lacks contextual knowledge for providing a well-considered answer. Such pragmatic comprehension problems are often triggered by vague quantifying term in the question (e.g., taxes on housing should be raised), which make the users realize they lack knowledge about the current state of affairs ('How high is that tax now?'). In case of comprehension problems, VAA users often assume a certain question meaning, and hardly ever proceed by looking for information on the web. Nevertheless, a large majority of the respondents provides a substantive answer (often the middle option). In Study 2, we investigated whether the question characteristics leading to comprehension difficulties in Study 1, lead to more neutral and no opinion-answers when statistically analyzed across a larger set of questions in a larger set of VAAs. We performed statistical analyses of all answers provided by 357,858 VAA respondents who used one of the 34 different municipal VAAs during the Dutch municipality elections in 2014. Results in Study 2 confirm that political jargon, geographical locations and vague quantifying terms are related to more neutral and/or no opinion-answers. Interestingly, there seems to be a relation between the type of comprehension problem and the type of answer provided: semantic meaning problems often result in no opinion answers, whereas pragmatic problems are related to neutral responses.

M3 - Abstract

ER -

Kamoen N, Holleman B. I don't get it: Response difficulties in answering political attitude statements in Voting Advice Applications. 2017. Abstract from Conference of the European Survey Research Association, Lisbon, Portugal.