Describing Complex Academic Tasks From Six Graduate Disciplines Using Multidimensional Scaling and Clustering

Abstract

The problem of defining the nature and variety of academic tasks is growing in importance as more complex assessment tasks are introduced in many educational contexts. In this study, multidimensional scaling and cluster analysis were used to describe and categorize tasks from six graduate disciplines including academic psychology, applied psychology, English literature, journalism, physics, and electrical engineering. A sample of task descriptions was constructed through interviews with graduate students from these disciplines. A rating instrument was designed to describe task goals and to evaluate whether the tasks were well or ill structured with respect to various aspects of problem definition and problem solution. Graduate faculty used the rating instrument to characterize a sample of tasks from their discipline. The scales were found reasonably reliable and were useful in identifying and describing task clusters and how such clusters varied both within and across disciplines. A cluster of short‐term problems that were posed by someone other than the student was found in every field although the other characteristics of this cluster of tasks varied with discipline. For example, the short‐term tasks in engineering and physics were well‐structured, requiring the application of established principles, and having objective standards for judging performance. In contrast, a cluster of short‐term tasks in English literature were very ill‐structured because different conceptual approaches could be relevant, there were alternative methods for accomplishing the tasks, many possible solutions existed, and the student had to define an issue or question to consider. In all disciplines except physics, a cluster of complex tasks emerged that was characterized as having multiple objectives that needed to be satisfied. The cluster of complex tasks that was found in physics was not described clearly by the scales. Problem‐finding was an important task characteristic in the social sciences and humanities but not in the physical sciences. The relevance of multidimensional scaling and clustering to test design is discussed.

Disciplinary Cultures and Graduate Education

“…There is also a small body of published research which helps to further de ne selected discipline-speci c aspects of graduate student socialization. One such study proposes a framework for classifying, by discipline, the differing types of academic problems and tasks in which graduate students are engaged (Enright and Oltman, 1996).…”

Toward Assessment of Design Skill in Engineering

“…The current test consists primarily of items requiring the candidate to analyze well-defined engineering systems using standard methods-the common types of questions asked by engineering educators of their students (Donald, 1991;Enright & Oltman, 1996). The current test consists primarily of items requiring the candidate to analyze well-defined engineering systems using standard methods-the common types of questions asked by engineering educators of their students (Donald, 1991;Enright & Oltman, 1996).…”

“…Design is not directly assessed in the current GRE Engineering Subject Test, which probably reflects limits imposed by the multiple-choice format rather than any oversight. The current test consists primarily of items requiring the candidate to analyze well-defined engineering systems using standard methods-the common types of questions asked by engineering educators of their students (Donald, 1991;Enright & Oltman, 1996). However, design is inherently generative and cannot easily be tested with multiple-choice items.…”