A comparative study exploring the impact of assessment criteria on eliciting graphical capability

Abstract

Much of formal educational provision correctly focuses on developing critical numeracy and literacy skills. Contemporary living in a digital image culture supports education now developing what Fish 1 describes as our visualizing instinct. At second level the national graphics curriculum in ________ has moved from Technical Drawing (TD) to Design and Communication Graphics (DCG). The traditional vocational focus is now replaced with learning graphics through design driven activities, which encourage outcomes of innovation and creativity. While students still develop subject specific content knowledge, drafting skills and communication skills, the approach to learning is concerned with the design process as a vehicle for transferable learning. Capturing the process of learning and not the product can be a challenge for conventional assessment methods. This paper explores the impact of determinist and non-determinist approaches to assessment on the nature and outcomes of the learning activity. The method employed an offset cohort analysis type study to explore the performance and output of two homogenous groups of initial teacher education students (stratified sample n=40). Group one were given defined assessment criteria, while the second group were expected to define the criteria based on their definition of graphical capability. The paper highlights the variance between groups in selecting, applying and executing appropriate graphical principles and medium, while solving an identical design brief. The paper concludes by highlighting the importance of understanding the impact of assessment criteria on student performance. Students, who constructed not only their own meaning, but also the rationale for their meaning, demonstrated a higher level of graphical capability.

Using Adaptive Comparative Judgment to Holistically Assess Creativity of Design Solutions: A Comparison of First-Year Students and Educators' Judgments

“…This is promising in terms of measuring constructs like creativity and design in engineering education, which manifest and are defined in various ways [10], [22], in a holistic manner. Although the reliability was found to be moderately high, it is lower than previous studies [3], [5], [7], [11], [16], [20]; however, these studies were assessing work based on less subjective criteria. The reliability of the rank in the current study could have been increased if all ACJ participants had completed their sessions.…”

“…The reliability of both sessions was moderately high (SSRfaculty = 0.65 ± 0.02, SSRstudents = 0.71 ± 0.02); however, this value was notably lower than prior studies [3], [5], [7], [11], [16], [20]. Two factors are important to consider in this regard.…”

“…Judges of ACJ sessions have previously included students, professionals and instructors/ faculty [3], [5]- [7], [16]. The number of judgements that must be made in an ACJ session is typically specified as a function of the number of items of work to be assessed [13], with this number typically being divided by the total number of judges to determine the number of judgements asked of each individual judge.…”

“…Previous studies that have used ACJ to assess different elements of design have reported SSR values of ~0.9 [3], [5], [7], [11], [16], [20]. Additionally, as this assessment approach relies on judgements being made by various individuals to set out the standard of work, it can also address issues of bias or varying views held by judges within a cohort via misfit statistics.…”

WIP: Adaptive Comparative Judgement as a Tool for Assessing First-Year Engineering Design Projects

“…First-year engineering students are typically required to engage in team and problem-based activities through introductory coursework to support the development of design capabilities [1], [11]. This type of activity is typically assessed using rubrics, portfolios, and criteriongrading tools [2]- [6]. However, there are issues in assessing open-ended and divergent tasks in this manner including; reliability, teacher bias, excessive time investment, and timeliness of feedback [1]- [5].…”