Findings derived from self-reported, structured survey questionnaires are commonly used in evaluation and applied research to inform policy-making and program decisions. Although there are a variety of issues related to the quality of survey evidence (e.g., sampling precision), the validity of response processes-how respondents process thoughts and perceptions when answering questionnaires-is crucial. We assess the extent to which cognitive interviews, used to test survey questions as part of the process of questionnaire design and refinement, can strengthen the quality of survey evidence for small-and medium-scale evaluations. To illustrate how cognitive interviews can reveal respondents' response processes and improved questionnaire interpretations, we present excerpts from two types of evaluations (e.g., needs assessment) in two domains (e.g., education). The paper concludes with a brief summary about how the use of cognitive interviews can be augmented in survey development, refinement, and adaptation to improve survey questionnaire interpretations in evaluations with limited resources.

No abstract

The authors examine student ratings within a new framework that emphasizes six distinct aspects of validity : content, substantive, structural, generalizability, external, and consequential. They It is fitting that as the new millennium arrives, a major revision of the Standards for Educational and Psychological Testing (American Psychological Association, 1985) has been completed and is the subject of much debate and dialogue. It reflects some changes in how the assessment community thinks about the concept of validity. The literature responsible for this shift began with Messick' s benchmark chapter on validity in the third edition of Educational Measurement (1989) and continued with Shepard' s article in the Review of Educational Research (1993) and articles on the consequences of assessment by Lane, Parke, and Stone, 1998;Linn, 1998;Reckase, 1998;Yen, 1998;Cronbach, 1989; and Moss, 1992, 1996. How validation efforts should be conducted and how assessment results are used is shifting as a consequence of this body of work.What are the implications of this work for student ratings of instruction? We take up this fundamental question in this chapter. We begin this effort by defining the concepts that are the basis for a new unified concept of validity. We briefly summarize the past validity research on student ratings using the traditional validity framework. Based on the concepts presented, we then do a critical analysis of student ratings research using the new validation framework. Synthesizing the results of the analysis, we propose an initial agenda for future student ratings' validation efforts.

No abstract

In evaluation and applied social research, focus groups may be used to gather different kinds of evidence (e.g., opinion, tacit knowledge). In this article, we argue that making focus group design choices explicitly in relation to the type of evidence required would enhance the empirical value and rigor associated with focus group utilization. We offer a descriptive framework to highlight contrasting design characteristics and the type of evidence they generate. We present examples of focus groups from education and healthcare evaluations to illustrate the relationship between focus group evidence, design, and how focus groups are conducted. To enhance the credibility of focus group evidence and maximize potential learning from this popular qualitative data collection method, we offer a set of questions to guide evaluators reflection and decision making about focus group design and implementation.