Dimensionality and Generalizability of Domain-Independent Performance Assessments

Baker, Eva L.; Abedi, Jamal; Linn, Robert L.; Niemi, David

doi:10.1080/00220671.1996.9941205

Cited by 33 publications

(33 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The findings from this study were consistent with much of the previously published research completed using G-theory to understand the variance inherent in performance assessments (Baker et al, 1995;Baxter et al, 1992;Lane et al, 1996;Ruiz-Primo et al, 1993;Shavelson et al, 1993). As in much of this earlier research, we found a strong effect from persons, tasks, and their interaction; we found little effect from raters.…”

Section: Discussionsupporting

confidence: 92%

Generalizability Theory Applied to Reading Assessments for Students With Significant Cognitive Disabilities

2008

View full text Add to dashboard Cite

Students with significant disabilities must participate in large-scale assessments, often using an alternate assessment judged against alternate achievement standards. The development and administration of this type of assessment must necessarily balance meaningful participation with accurate measurement. In this study, generalizability theory is used to estimate the dependability of reading items and tasks that have been administered using two formats of communication (receptive and expressive). The results reflect a trade-off between meaningful participation and accurate measurement of students with significant cognitive disabilities, particularly when considering the two formats. Significant variance is obtained for persons interacting with tasks, whereas the effect of raters is negligible. Furthermore, these results appear to vary across administrative format.

show abstract

Section: Discussionsupporting

confidence: 92%

Generalizability Theory Applied to Reading Assessments for Students With Significant Cognitive Disabilities

2008

View full text Add to dashboard Cite

show abstract

“…It is unclear whether similar results would have been found had a prompt been assigned that was more restrictive in terms of student choice and that placed greater demands on students in terms of background knowledge. Given the literature on prompt and task effects in writing (Baker, Abedi, Linn, & Niemi, 1995;Chen, Niemi, Wang, Wang, & Mirocha, 2007), it is important that future research attempt to replicate results across different writing tasks.…”

Section: Limitations and Future Researchmentioning

confidence: 99%

Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications

Tetreault

Burstein

Leacock³

2015

View full text Add to dashboard Cite

iii IntroductionWe are excited to be holding the 10th anniversary the BEA workshop. Since starting in 1997, the BEA workshop, now one of the largest workshops at NAACL/ACL, has become one of the leading venues for publishing innovative work that uses NLP to develop educational applications. The consistent interest in and growth of the workshop has clear ties to societal need and related advances in the technology, and the maturity of the NLP/education field. NLP capabilities now support an array of learning domains, including writing, speaking, reading, and mathematics. Within these domains, the community continues to develop and deploy innovative NLP approaches for use in educational settings. In the writing and speech domains, automated writing evaluation (AWE) and speech scoring applications, respectively, are commercially deployed in high-stakes assessment and instructional settings, including Massive Open Online Courses (MOOCs). We also see widely-used commercial applications for plagiarism detection and peer review. Major advances in speech technology, have made it possible to include speech in both assessment and Intelligent Tutoring Systems. There has been a renewed interest in spoken dialog and multi-modal systems for instruction and assessment as well as feedback. We are also seeing explosive growth of mobile applications for game-based applications for instruction and assessment. The current educational and assessment landscape, continues to foster a strong interest and high demand that pushes the state-of-the-art in AWE capabilities to expand the analysis of written responses to writing genres other than those traditionally found in standardized assessments, especially writing tasks requiring use of sources and argumentative discourse.The use of NLP in educational applications has gained visibility outside of the NLP community. First, the Hewlett Foundation reached out to public and private sectors and sponsored two competitions: one for automated essay scoring, and the other for scoring of short answer, fact-based response items. The motivation driving these competitions was to engage the larger scientific community in this enterprise. MOOCs are now beginning to incorporate AWE systems to manage the thousands of constructedresponse assignments collected during a single MOOC course. Learning@Scale is a recent venue for discussing NLP research in education. The NLP-TEA workshop, now in its second year (NLP-TEA2), gives special attention to papers working on Asian languages. The Speech and Language Technology in Education (SLaTE), now in its sixth year, promotes the use of speech and language technology for educational purposes. Another breakthrough for educational applications within the CL community is the presence of a number of shared-task competitions over the last three years. There have been three shared tasks on grammatical error correction with the most recent edition hosted at CoNLL 2014. In 2014 alone, there were four shared tasks for NLP and Education-related areas.As a community, we contin...

show abstract

“…Thirdly, the assessments have to be general enough to accommodate the broad variation in what students choose and produce. Satisfying such diverse requirements demands a great deal of development work in order to create good scoring rules (Arter, 1993;Baker et al, 1995;Novak et al, 1996). In practice, it is often difficult to distinguish between students in a sufficiently reliable and valid manner.…”

Section: Teachers' Assessment Of Students' Research Skills 97mentioning

confidence: 99%

“…However, the potential of this approach is not yet clear (Hambleton & Murphy, 1992;Linn, 1994;Messick, 1994). This is all the more important for examination assignments that count towards qualification (De Groot, 1970;Baker et al, 1995).…”

Section: Introductionmentioning

confidence: 99%

Teachers' assessment of students' research skills

Stokking

Schaaf

Jaspers

et al. 2004

British Educational Res J

View full text Add to dashboard Cite

Today, teachers are expected to develop complex skills, such as research skills, in their students while implementing new views on learning and teaching and using authentic assessment strategies. About these new assessment strategies there is much debate and teachers are vulnerable in using them. We studied upper secondary education natural and social science teachers' practices using two surveys and two rounds of expert panel judgement on teacher‐submitted assessment‐related material and information. Our study shows that there are grounds for concern regarding the clarity of teachers' assessment criteria, the consistency between teachers' goals, assignments, and criteria, and the validity and acceptability of teachers' assessment practices. The extent to which it is justifiable to judge teachers' assessment practice by professional quality criteria is discussed, and suggestions are given as to the main quality criteria for formative and summative assessment and as to ways in which teachers could improve their assessment practices.

show abstract

Dimensionality and Generalizability of Domain-Independent Performance Assessments

Cited by 33 publications

References 10 publications

Generalizability Theory Applied to Reading Assessments for Students With Significant Cognitive Disabilities

Generalizability Theory Applied to Reading Assessments for Students With Significant Cognitive Disabilities

Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications

Teachers' assessment of students' research skills

Contact Info

Product

Resources

About