Contradictory results have been reported regarding the accuracy of various methods used to assess student learning in higher education. The current study examined student learning outcomes across a multi-section and multi-instructor psychology research course with both indirect and direct assessments in a sample of 67 undergraduate students. The indirect method measured student perceived knowledge and abilities on course topics, while the direct method measured actual knowledge where students answered test questions or solved problems reflecting course content. Both measures independently demonstrated increases from pretest to posttest; however the indirect measure did not correlate with final course grades. Results also showed respondents scoring lower on the direct measure were overconfident (as measured by indirect score) in their perceived knowledge and ability, the Dunning-Kruger Effect. Based on our findings, we concluded that the indirect method was not an accurate measure of student learning, but may have benefits as an instructional tool.

Substantial progress has been made toward applying technology enhanced conversation‐based assessments (CBAs) to measure the English‐language proficiency of English learners (ELs). CBAs are conversation‐based systems that use conversations among computer‐animated agents and a test taker. We expanded the design and capability of prior conversation‐based instructional and assessment systems and developed a CBA designed to measure the English language skills and the mathematics knowledge of middle school ELs. The prototype CBA simulates an authentic and engaging mathematics classroom where the test taker interacts with two virtual agents to solve math problems. We embedded feedback and supports that are triggered by how the CBA interprets students' written responses. In this study, we administered the CBA to middle school ELs (N = 82) residing in the United States. We examined the extent to which the CBA system was able to consistently interpret the students' responses (722 responses for the 82 students). The study findings helped us to understand the factors that affect the accuracy of the CBA system's interpretations and shed light on how to improve CBA systems that incorporate scaffolding.

No abstract

No abstract