Test–Retest Reliability of Choice Experiments in Environmental Valuation

Abstract

The paper presents the results of the first test-retest study on choice experiments in environmental valuation. In a survey concerning landscape externalities of onshore wind power in central Germany, respondents answered the same five choice sets at two different points in time. Each choice set comprised three alternatives described by five attributes, and the time interval between the test and the retest was eleven months. The analysis takes place at three different levels, investigating choice consistency at the choice task level and repeatability of the latent construct utility at the level of parametric models as well as at the level of willingness-to-pay estimates. At the choice task level we observed 59 % identical choices. The parametric analysis shows that the test and retest estimates are not equal, even when we control for scale, that is, differences in the error variance. However, comparing the marginal willingness-to-pay estimates among test and retest reveals only a statistically significant difference for one of the attributes. Overall, this indicates a moderate test-retest reliability taking into account that consistency at the choice task level overlooks the stochastic nature of the process underlying discrete choice experiments.

Willingness to pay for threatened and endangered marine species: a review of the literature and prospects for policy use

“…Fewer examinations of temporal stability of CE preferences and values have been undertaken, and none have examined long time periods. However, the existing studies tend to support stability of WTP values over short term periods of up to a year (Bliem et al, 2012;Liebe et al, 2012).…”

“…If people's preferences and values for protecting TER marine species change over time, then using older value information in a benefits transfer will lead to biased results (i.e., increase the transfer error). In general, the empirical literature assessing the temporal stability of WTP estimates from SP studies, generally through test-retest samples or two independent samples engaged at different time periods, suggests that time periods up to about five years yield temporally stable preferences and values (e.g., Carson et al, 1997;McConnell et al, 1998;Brouwer and Bateman, 2005;Skourtos et al, 2010;Liebe et al, 2012) 29 . If one applies this rule of thumb to the literature examined here based on publication year, only eight studies (Lew et al, 2010;Ojea and Loureiro, 2010;Wallmo and Lew, 2011, 2012Boxall et al, 2012;Kontogianni et al, 2012;Stithou and Scarpa, 2012) comprise the set of viable studies that are recent enough to have preferences and values that are likely unchanged, but with several due to "expire" shortly.…”

Preference Stability and Choice Consistency in Discrete Choice Experiments

“…Within the DCE literature, there areonly a few examples of the same respondents completing identical sets at different points in time 1 . Bryan et al (2000), Ryan et al (2006), San Miguel et al (2002 and Skjoldborg et al (2009) investigate the issue of intertemporal choice consistency within health DCE studies, while Liebe et al (2012), Schaafsma et al (2014) and Czajkowski et al (2014) do so in environmental contexts. The analytical focus of these papers is typically the level of choice consistency over time [in terms of raw counts or via measures of consistency, typically Cohen's statistic of inter-rater reliability (Cohen 1960)] and/or the stability of parameter estimates from choice models estimated on the pooled data.…”

Preference and WTP stability for public forest management

“…This indicated that there was no difference between preferences in these 30 samples. Liebe, Meyerhoff and Hartje (2012) used an Error Component model to compare preference 31 and WTP estimates in two samples collected 11 months apart. Choices over on-shore wind power 32 options were reasonably consistent over the interval, but WTP estimates differed significantly for around 33 half of the attribute values.…”