Proxy documents as a source of measurement error in the Comparative Manifestos Project

Abstract

This paper considers the issue of document type diversity in the Comparative Manifestos Project (CMP). For many years the CMP has been collecting and coding a variety of documents, such as speeches, pamphlets, newspaper articles and leaflets, as manifesto proxies. By using previously unexplored archival material to perform controlled comparisons between different types of documents, this paper argues that the coding of such documents introduced considerable measurement error to party position estimates. Statistical analyses indicate that this measurement error is systematic rather than random as it is often manifested as centrist bias in parties' left-right position estimates. Consequently, the paper argues that random error correction methods cannot always correct for error attributed to the coding of proxy documents. The paper concludes with some recommendations for thirdparty users of the CMP data and documents and a plea to the CMP research team.

Estimating parties’ policy positions through voting advice applications: Some methodological considerations

“…In turn, both steps can introduce unreliability in the coding of parties' positions. It is well known that many parties do not publish manifestos (Gemenis, 2012), whereas some parties do not have very clearly stated positions on all issues that are included in VAAs. Therefore, party position estimates in VAAs are partially dependent on how 'deep' researchers are willing to search in order to find some evidence, which can be used for coding.…”

“…This implies that coding reliability is also constrained by the type of documents used (cf. Gemenis, 2012). When lacking credible documents that explicitly refer to the statements used by VAAs, coders are in a much worse position in estimating the positions of parties.…”

“…However, in the absence of documents at the top, expert coders are left to decide themselves about how much further in the hierarchy of documents to go. Moreover, interviews and statements made during parliamentary debate appear at the bottom of the hierarchy, but it is still questionable whether such evidence can be considered as representative under the party as a unitary actor assumption (Gemenis, 2012). Equally controversial is using evidence from legislative voting in order to gauge parties' positions, as it is well established that voting patterns often reflect party discipline and government-opposition dynamics, which implies that roll-calls cannot always be considered as parties' ideal point estimates (Spirling and McLean, 2007).…”

Varieties of Radicalism: Examining the Diversity of Radical Left Parties and Voters in Western Europe

“…Among the most frequently highlighted are issues of data validity, reliability (Pennings and Keman 2002, Benoit and Laver 2007, Mikhaylov et al 2012, and of comparability across parties, countries and types of texts (Gabel and Huber 2000;Benoit et al 2012;Gemenis 2012). For the specific case of RLPs, an additional problem is that the way the traditional left-right scale is extracted with the CMP data relies on dimensional analyses that are very dependent on the distribution of parties used to identify the items that form the left-right scale.…”

Private Debt and the Anglo-Liberal Growth Model

“…Uncertainty in the estimates generated by a lack of reliability in the coding of the (quasi-) sentences, or by the use of proxy documents in place of manifestos, is not taken into account in the data (Benoit, Laver, and Mikhaylov 2009;Gemenis 2012). However, to the extent that the errors are random for the issue areas I am interested in, the addition of uncertainty around each (party salience) estimate will serve to increase variation in each country, and thus in each regime.…”