Multi-method Longitudinal Assessment of Transferrable Intellectual Learning Outcomes

Abstract

Natalie Simper coordinates a Queen's research project investigating the development and measurement of general learning outcomes. Natalie comes from an Australian Senior-Secondary/ Post-Secondary teaching background, with experience at the State-wide level in curriculum development, large-scale assessment, and evaluation and assessment of outcomes based education. Multi-method longitudinal assessment of transferable intellectual learning outcomes AbstractThis paper describes the first phase of a four-year longitudinal study of transferable intellectual learning outcomes at a research-intensive university in Canada. These skills, including critical thinking, problem solving, communication and lifelong learning, are the subject of discussion in higher education generally (e.g. the American Association of Colleges and Universities (AAC&U) Essential Learning Outcomes) and engineering in the form of accreditation requirements. The project is using multiple methods to assess the development of these skills in engineering, humanities, physical science, and social science sectors. The first year of the study which involved a double cross-sectional assessment of first and fourth year undergraduate students using standardized tests, meta-rubrics used to score course artefacts, and a learning orientation instrument. Interim results are showing significant development of transferable intellectual skills from first to fourth year, with significant differences between instruments in cost, alignment, and participation rates.

Evaluation of competency methods in engineering education: a systematic review

“…(Eichelman, Clark, and Bodnar 2015) and (Galván-Sánchez et al 2017) used rubrics to measure student performances in demonstrating written and oral communication, respectively. Also, (Frank et al 2015) has objectively measured students' performance on written communication using two valid methods (the VALUE rubric the CLA+). (Wilkins, Bernstein, and Bekki 2015), on the other hand, validated a test that measures not only student self-perceived knowledge in communication skills (such as active listening, assertive self-expression, and receiving and responding to feedback), and their confidence to use these skills, but also their ability to apply these communication skills.…”

“…Use orally appropriate, relevant and compelling content that shows mastery of the subject (Saunders et al 2003;Wasserman, Schmidt, and Jendrucko 2003 Interact with and adapt the oral communication to different audiences (Wasserman, Schmidt, and Jendrucko 2003;Bousaba et al 2014;Dohaney et al 2015;Ortiz et al 2015) Questions and answers (n = 4) Understand and answer questions (Saunders et al 2003;Amante et al 2010;Keshavarz and Vaziri 2012;Ryan 2012) Team uniformity (n = 2) Show homogeneous transition between the presenters (visual and oral presentation) (Ryan 2012;Galván-Sánchez et al 2017) Written communication (n = 24) Language (n = 13) Use the appropriate vocabulary, syntax and grammar (Saunders et al 2003;Wasserman, Schmidt, and Jendrucko 2003;Flateby and Fehr 2008;Keshavarz and Vaziri 2012;Jones and Abdallah 2013;Mourtos 2014;Eichelman, Clark, and Bodnar 2015;Frank et al 2015;García et al 2015;Backer 2016;Barr 2016;de-Juan et al 2016;Fries et al 2017) Report organization (n = 12) Develop a clear, logic, structured and organised report (Saunders et al 2003;Wasserman, Schmidt, and Jendrucko 2003;Flateby and Fehr 2008;Keshavarz and Vaziri 2012;Jones and Abdallah 2013;Mourtos 2014;Eichelman, Clark, and Bodnar 2015;Frank et al 2015;Barr 2016;de-Juan et al 2016;…”

“…References Content development (n = 10) Write appropriate, relevant and compelling content that shows mastery of the subject (Saunders et al 2003;Wasserman, Schmidt, and Jendrucko 2003;Flateby and Fehr 2008;Keshavarz and Vaziri 2012;Jones and Abdallah 2013;Mourtos 2014;Eichelman, Clark, and Bodnar 2015;Frank et al 2015;Barr 2016;Fries et al 2017) Source citation (n = 8) Cite sources accurately (Saunders et al 2003; Barnes, Dyrenfurth, and Newton 2012;Jones and Abdallah 2013;Eichelman, Clark, and Bodnar 2015;Frank et al 2015;García et al 2015;Barr 2016; Ramírez-Echeverry, Olarte Dussán, and García-Carillo 2016) Context and purpose (n = 6) Perform a report according to the context and purpose of the situation (Wasserman, Schmidt, and Jendrucko 2003;Flateby and Fehr 2008;Eichelman, Clark, and Bodnar 2015;Frank et al 2015;Barr 2016; Ramírez-Echeverry, Olarte Dussán, and García-Carillo 2016) Audience (n = 4)…”

“…Adapt the written communication according to the audience (Wasserman, Schmidt, and Jendrucko 2003;Flateby and Fehr 2008;Frank et al 2015; Ramírez-Echeverry, Olarte Dussán, and García-Carillo 2016) Graphical/tables support (n = 3)…”

How We Know They're Learning: Comparing Approaches to Longitudinal Assessment of Transferable Learning Outcomes

“…The frameworks and selection of tools described in this section are summarized from a previous publication that also provides more details about the motivation and context for this study 12 . The study focuses on use of authentic assessments, in which students are asked to demonstrate competencies on contextualized tasks that emulate real-world situations.…”