Assessing the Validity of a Language Arts Instruction Log through Triangulation

Abstract

In this study we attempted to illuminate why measures of instruction sometimes fail to meet discrete tests of validity. We used a triangulation strategy-multiple methods, data sources, and researchers-to investigate teachers' and observers' reports on a daily language arts log. Data came from a pilot study of the log conducted in 8 urban public elementary schools. Statistical results increased our confidence in the log's ability to measure: a) instruction at grosser levels of detail, b) instructional activities that occurred more frequently, and c) word analysis instruction. Some qualitative evidence gave us greater confidence in the instrument-for example, when teachers differed from observers because they possessed background knowledge not available to observers. Other qualitative evidence illustrated dilemmas inherent in measuring instruction. Overall, we believe triangulation strategies provided a more holistic understanding of the validity of teachers' reports of instruction than past validity studies. Mullens & Graylor 1999;Mullens & Kasprzyk, 1996Porter, Kirst, Osthoff, Smithson, & Schneider, 1993). In these studies, teachers' self-reports were compared to data from classroom observations, instructional "artifacts" such as textbooks, or teachers' reports on other instruments, and the differences examined.Studies of the validity of instruction measures have typically followed a common analytical sequence-a measure is compared to discrete criteria, and if the criteria is not met, the validity of the measure is called into question. The criteria employed have varied from study to study. In the earlier process-product research, the criteria was interobserver agreement, with greater agreement among third-party observers taken to indicate greater validity. Burstein et. al. (1995) 4 used multiple criteria to examine the correspondence between teachers' questionnaire responses, their responses to an instruction log, and curricular topics in textbook chapters teachers claimed to have covered during the questionnaire's reporting period. Again, teachers' responses to the questionnaire were considered more valid when they agreed with the other two sources of data.In our opinion, basing judgments about the validity of instructional measures solely on their agreement with criteria has fostered a faulty impression that such judgments are straightforward either/or propositions. The logic seems to be that, as long as the criteria have been met, one can be confident that the measure is valid. A major outcome of this either/or focus has been that studies in this area have primarily documented the degree to which measures of instruction do or do not agree with the criteria. Beyond this kind of evidence, however, validity studies have yielded little information about why measures of instruction fail to meet criteria. In the case of measures based on self-reports, little is known about how teachers formulate self-reports of their instruction or about the factors that lead teachers to provide self-reports that dive...

Measuring the Learning-Centered Leadership Expertise of School Principals

“…Next we discuss the conceptual definitions of expertise in general and present the specific domains of leadership expertise we attempt to measure and third, we present the results of a study that implemented the two measures of leadership expertise: principal surveys and open-ended scenarios. We follow the work of Camburn and Barnes (2004) and rely on triangulation as a way to validate measures, supporting the view that "multiple methods enhance the validity of research findings by overcoming the weaknesses or bias in each method" (p. 51). At the onset we acknowledge that our work is preliminary both conceptually and empirically.…”

Gender differences in participatory leadership: An examination of principals’ time spent working with others

“…However, in contrast to previous studies, which have mainly used annual surveys, interviews, and observations, we use EOD logs to study actual daily time allocation patterns of male and female principals. The EOD log is a unique method of studying school leadership practice using self-reported calendar data in which survey participants recount how they spent their time working across different activity domains and with different sets of people (for more detailed descriptions of the instrument, see Barnes, Camburn, Sanders, & Sebastian, 2010;Camburn & Barnes, 2004;Camburn et al, 2010;Rowan, Camburn, & Correnti, 2008). Studies have examined the validity of the log instrument in obtaining principal practice data, by comparing it to other types of data such as observations, surveys, and interviews Spillane & Zuberi, 2009).…”

Creating an Indicator of K--12 Classroom Coverage of Science, Technology, Engineering, and Math (STEM) Content and Practices

“…Logs also impose added costs related to monitoring of responses and the need for financial incentives to promote high response rates. Prior research on the technical quality of instructional logs provided some support for using instructional logs to collect information about classroom instruction but also identified issues that threaten the validity of log results, such as differences in teachers' understanding of the log questions, the lack of occurrence of certain instructional activities covered by log questions, and inconsistent technical properties when teachers use rating scales with different levels to answer the same log questions (Ball et al, 1999;Brandon and Taum, 2005;Camburn and Barnes, 2004;Rowan, Camburn, and Correnti, 2004;Rowan, Harrison, and Hayes, 2004;Le et al, 2004). These concerns also apply to surveys more generally.…”