Using Eye-Tracking to Understand Decision-Making in Conjoint Experiments

Abstract

Conjoint experiments are popular, but there is a paucity of research on respondents’ underlying decision-making processes. We leverage eye-tracking methodology and a series of conjoint experiments, administered to university students and local community members, to examine how respondents process information in conjoint surveys. There are two main findings. First, attribute importance measures inferred from the stated choice data are correlated with attribute importance measures based on eye movement. This validation test supports the interpretation of common conjoint metrics, such as average marginal component effects (AMCEs), as measures of attribute importance. Second, when we experimentally increase the number of attributes and profiles in the conjoint table, respondents view a larger absolute number of cells but a smaller fraction of the total cells displayed. Moving from two to three profiles, respondents search more within-profile, rather than within-attribute, to build summary evaluations. However, respondents’ stated choices remain fairly stable regardless of the number of attributes and profiles in the conjoint table. Together, these patterns speak to the robustness of conjoint experiments and are consistent with a bounded rationality mechanism. Respondents adapt to complexity by selectively incorporating relevant new information to focus on important attributes, while ignoring less relevant information to reduce cognitive processing costs.

Subjects, Trials, and Levels: Statistical Power in Conjoint Experiments

“…The median number of attributes is 7 (p 25 = 5 and p 75 = 8) and seems to follow what is recommended by recent literature on satisficing in conjoint experiments (Bansak et al 2018;Jenke et al 2020). The median of the number of levels for the biggest attribute is 5 (p 25 = 4 and p 75 = 9).…”

“…Based on previous research (see , Jenke et al 2020), we assume that respondents form the probability of selecting a profile in two complementary ways: horizontally and vertically. Respondents make within-attribute horizontal comparisons-i.e.…”

Why Austerity? The Mass Politics of a Contested Policy

“…17 This represents a departure from common conjoint experimental designs, which generally draw from a much smaller number of levels for each attribute, and allows us to collect information on a much more fine-grained space of austerity packages than would be feasible with a standard conjoint design. To achieve data coverage over as much of the space as possible, we asked each of our respondents to evaluate 10 pairs of conjoint profiles (20 packages in total), 18 thereby taking advantage of recent research showing that response quality in conjoint experiments does not deteriorate after this number of tasks (Bansak et al 2018;Jenke et al 2021). In addition, for the portion of our analyses in which we predict levels of support for specific austerity packages, we pair our high-dimensional conjoint design with an analogous flexible prediction method, described below.…”

NIMBYism as Place-Protective Action: The Politics of Housebuilding

“…Figure 3 shows an example conjoint task; the full list of possible attribute levels is shown later. These numbers of attributes and tasks have been shown to elicit reasonable responses and are comfortably below the level at which satisficing from respondents begins to occur (Bansak et al 2018(Bansak et al , 2019Jenke et al 2020). The order of the attributes was randomised for every task, and all attribute levels had an equal probability of appearance.…”

“…I first do this using Average Marginal Component Effects (AMCEs). Here an AMCE measures the causal effect of a particular attribute level, relative to a baseline, on the probability of one housing development being chosen over another, averaging over many different housing developments in the experiment(Bansak et al 2020). 12 It is well-suited to answering a question such as "does building detached houses instead of 1-2 bedroom flats affect the probability of choosing a development as much as increasing the distance of a development from respondents' homes from 1/2 mile away to 2 miles away?…”