No abstract

Institutions of higher education are expected to address accountability and transparency of student learning and development efforts. The Student Opinion Scale assesses examinee motivation during testing and asks students to report level of effort invested and perceived importance of taks. This article evaluates validity evidence and supports widespread instrument use.

Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other four-year institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100% of the QR and SR items were successfully mapped to each of the partner institutions’ learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) results from the partner institutions for the learning objectives and major subtests are strikingly similar across very different student populations. We interpret lower reliabilities from one institution to be the result of test administration and student motivation factors, the latter being a serious threat to the health and vigor of any assessment program. Validity study results at the partner institutions add to the evidence of construct validity of the QR and SR instruments. While our studies focus on QR and SR instruments, the methods will apply to other instruments and other institutions as they attempt to answer important questions about student learning outcomes

No abstract

To investigate the cognitive validity of scientific and quantitative reasoning items, “think-alouds” (verbal solutions) were elicited for a general education instrument. Several items were not aligned in terms of anticipated versus actual content, and the instrument's accuracy is questioned. We discuss study weaknesses and merits of this framework and analysis.