Conducting Small-Scale Multimethod Questionnaire Evaluation

Abstract

When pretesting survey questionnaires, there are benefits to using a multimethod approach. However, using multiple methods can be cost prohibitive. In 2021, the National Agricultural Statistics Service (NASS) received feedback from stakeholders regarding concerns about double counting of grain stocks in the Agricultural Survey and the Grain Stocks Report. Data from these two surveys are used to produce the Grain Stocks publication, which is a principal Federal Economic Indicator publication. To address stakeholder feedback, NASS evaluated these two multimode surveys, in a short timeframe, with limited resources. Despite these limitations, NASS utilized expert review, cognitive testing, usability testing and behavior coding to evaluate these two survey questionnaires. This paper demonstrates that multimethod pretesting can be done effectively on a low-cost, small-scale basis.