Designing Sound Scoring Criteria for Assessing Student Performance

Trevisan, Michael S.; Davis, Denny; Calkins, D. E.; Gentili, Kenneth

doi:10.1002/j.2168-9830.1999.tb00415.x

Cited by 25 publications

(21 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An effective approach is to identify aspects of the product or presentation to be rated (e.g., for grading project or laboratory reports, the aspects might be technical soundness, organization, thoroughness of discussion, and quality of writing), select a weighting factor for each aspect, and construct a rubric-a form on which the evaluator assigns numerical ratings to each specified aspect and then uses the specified weighting factors to compute an overall rating. Trevisan et al [85] offer suggestions regarding the effective design and use of rubrics, including a recommendation that the characteristics of the highest and lowest ratings and the midpoint rating for each feature be spelled out fairly explicitly. If several raters complete forms independently and then reconcile their ratings, the result should be very reliable, and the reliability can be increased even further by giving raters preliminary training on sample products or videotaped presentations.…”

Section: Assessing Learningmentioning

confidence: 99%

Designing and Teaching Courses to Satisfy the ABET Engineering Criteria

Felder

Brent

2003

J of Engineering Edu

626

428

View full text Add to dashboard Cite

Since the new ABET accreditation system was first introduced to American engineering education in the middle 1990s as Engineering Criteria 2000, most discussion in the literature has focused on how to assess Outcomes 3a-3k and relatively little has concerned how to equip students with the skills and attitudes specified in those outcomes. This paper seeks to fill this gap. Its goals are to (1) overview the accreditation process and clarify the confusing array of terms associated with it (objectives, outcomes, outcome indicators, etc.); (2) provide guidance on the formulation of course learning objectives and assessment methods that address Outcomes 3a-3k; (3) identify and describe instructional techniques that should effectively prepare students to achieve those outcomes by the time they graduate; and (4) propose a strategy for integrating programlevel and course-level activities when designing an instructional program to meet the requirements of the ABET engineering criteria.

show abstract

Section: Assessing Learningmentioning

confidence: 99%

Designing and Teaching Courses to Satisfy the ABET Engineering Criteria

Felder

Brent

2003

J of Engineering Edu

626

428

View full text Add to dashboard Cite

show abstract

“…October 2003 they are widely used because they provide a convenient measure with some validity [24,27,28]. Objective measures, which at first glance appear much more direct and definitive, are not simple to obtain and have important disadvantages (i.e., availability, length, cost, and relevance) [27,29]. Problems with observer bias create the most important issue when considering direct measurements because ensuring truly objective data requires careful and complicated experimental design, perhaps even involving "double blind" type studies.…”

Section: Journal Of Engineering Educationmentioning

confidence: 99%

Student Learning of Criterion 3 (a)‐(k) Outcomes with Short Instructional Modules and the Relationship to Bloom's Taxonomy

Pimmel

2003

J of Engineering Edu

View full text Add to dashboard Cite

Engineering accreditation criteria require that engineering graduates demonstrate competency with a set of skills identified in Criterion 3 (a)-(k). Because of a scarcity of instructional material on many of these topics, a team of engineering instructors developed and tested a set of short modules for teaching these skills. Using before and after module surveys, the students indicated their confidence in their ability to do specific tasks derived from the module's learning objectives. Data also were obtained with a control group not receiving the instruction. In comparing pre-and post-module data, 33 percent of the comparisons were significantly different at the 0.05 level. In comparing control and post-module data, the corresponding value was 44 percent. These results indicate that instruction with these short modules produced a significant effect on student learning. I. INTRODUCTIONEngineering accreditation criteria require that engineering programs demonstrate that their students have developed the set of skills defined in ABET Engineering Criterion 3 (a)-(k) [1]. In order for student to acquire these skills, engineering faculty must teach them in an effective way. The literature suggests that this must involve explicit instruction in a workshop or cooperative learning format, opportunities to practice the skill, frequent feedback, and structured discussion activities [2][3][4][5][6]. Engineering instructors, who try to teach processing skills using these methods as a part of a traditional engineering course, confront a number of problems, particularly the scarcity of appropriate instructional material. To begin filling this void, ten engineering faculty members from six departments developed a set of instructional modules for teaching several skills derived from the outcomes identified in Criterion 3(a)-(k). They identified fifteen processing skills in four categories as shown in Table 1. The resulting instructional modules are posted on our Web site and described in several earlier reports [7][8][9][10][11][12][13][14]. The present report describes a systematic study of the effectiveness of these modules in changing the students' perception of their learning. It also looks at how these effects are related to the level of the learning as defined by Bloom's taxonomy [15][16][17].Module developers first established a common set of specifications so all of the modules had certain essential characteristics. Developers designed the modules for a week of classes (i.e., three 50-minute class periods) in upper-level engineering courses. To increase the effectiveness and appeal of the modules, they incorporated active/cooperative learning and the use of Web-based resources [5,6,[18][19][20][21][22][23][24][25]. Each module contained learning objectives, a justification, and student exercises and assignments, along with an instructor's guide.Individual learning objectives for each module were placed into one of the six categories in the Bloom's taxonomy [15][16][17]. Table 2 shows the distribution of learning ...

show abstract

“…Specifically, differences in scoring between or among raters detract from consistency, and therefore, lower reliability estimates. However, work on rater scoring of performance assessments, essays, and constructed response tasks, has shown that high reliability can be obtained 6,7 . Requirements include clear scoring criteria, decision rules, and rater training.…”

Section: Establishing Reliabilitymentioning

confidence: 99%

Enhancing Scoring Reliability In Mid Program Assessment Of Design

Davis¹,

Trevisan²,

McKenzie³

et al.

2001 Annual Conference Proceedings

View full text Add to dashboard Cite

For the past six years, faculty across Washington State have worked to define and measure design competencies for the first two years of engineering and engineering technology degree programs. A three part performance-based assessment to assess student design capabilities at the mid-program level was developed for this purpose. This paper presents a pilot reliability study designed to enhance the consistency of scoring the three-part assessment. Toward this end, three raters participated in a multi-step procedure which included initial scoring of student work, reconciliation of differences among raters, revision of scoring criteria, and the development of decision rules to deal with student work difficult to score within the existing scoring criteria. Intraclass correlation coefficients were computed before and after this process, showing marked improvement of inter-rater reliability. The revised scoring criteria and decision rules offer potential for faculty to produce reliable scores for student design performance on constructed response items and tasks, a prerequisite to sound program decision making.

show abstract

Designing Sound Scoring Criteria for Assessing Student Performance

Cited by 25 publications

References 3 publications

Designing and Teaching Courses to Satisfy the ABET Engineering Criteria

Designing and Teaching Courses to Satisfy the ABET Engineering Criteria

Student Learning of Criterion 3 (a)‐(k) Outcomes with Short Instructional Modules and the Relationship to Bloom's Taxonomy

Enhancing Scoring Reliability In Mid Program Assessment Of Design

Contact Info

Product

Resources

About