Rapid Online Assessment of Reading Ability

Yeatman, Jason D.; Tang, Kenny An; Donnelly, Patrick; Yablonski, Maya; Ramamurthy, Mahalakshmi; Karipidis, Iliana I.; Caffarra, Sendy; Takada, Megumi E.; Kanopka, Klint; Ben-Shachar, Michal; Domingue, Benjamin W.

doi:10.1101/2020.07.30.229658

Cited by 3 publications

(13 citation statements)

References 71 publications

(95 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A browser-based LDT was implemented based on the 252 items (which are separated into three test forms consisting of 84 items) that were designed in our previous study and optimized to cover a broad ability range (see Figure 1. ; see more detail in Yeatman et al, 2021). Each list is presented in a block of trials, the order of the lists is randomized, and the stimuli within each list are presented in a random order (but stimuli were not mixed across lists).…”

Section: Methodsmentioning

confidence: 99%

“…Our previous study was the first to use IRT to calibrate item difficulty on a lexical decision task but this initial study only included a relatively small sample of research participants recruited by two universities (Yeatman et al 2021). The goal of Study 1 is to examine the stability of item difficulty parameters across different school samples that vary in terms of socioeconomic status, age, and language-based learning difficulties.…”

Section: Developing Browser-based Computerized Adaptive Testingmentioning

confidence: 99%

“…The previous ROAR study (Yeatman et al, 2021) recruited a total of N = 306 children and adults through a research participant database (https://dyslexia.stanford.edu/). We refer to this sample that was recruited for university-based research studies as the Calibration group.…”

Section: Participantsmentioning

confidence: 99%

“…Accurately assessing students' reading ability and efficiently identifying struggling readers is a shared challenge faced by researchers, clinicians, teachers and school administrators. The Rapid Online Assessment of Reading (ROAR) is a web-based lexical decision task that has shown promise in measuring reading abilities without the need for a proctor, with comparable results to traditional, individually administered reading assessments (Yeatman et al, 2021). The first version of ROAR employed an assessment in which items are randomly ordered.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

ROAR-CAT: Rapid Online Assessment of Reading ability with Computerized Adaptive Testing

Ma,

Richie-Halford,

Burkhardt

et al. 2023

Preprint

View full text Add to dashboard Cite

The Rapid Online Assessment of Reading (ROAR) is a web-based, lexical decision task that measures single word reading abilities in children and adults without a proctor. Here we study whether item response theory (IRT) and computerized adaptive testing (CAT) can be used to create a more efficient online measure of word recognition. To construct an item bank, we first analyzed data taken from four groups of students (N = 1,960) who differed in age, socioeconomic status, and language-based learning disabilities. The majority of item parameters were highly consistent across groups (r=0.78 - 0.94); 6 items that functioned differently across groups were removed, leaving 246 items in the final item bank. Next, we implemented a JavaScript CAT algorithm and conducted a validation experiment with 485 students in grades 1-8 who were randomly assigned to complete trials of all items in the item bank in either a) a random order vs b) an order determined by the CAT algorithm. We found that, to achieve reliability of 0.9, CAT improved test efficiency by 40%: 75 CAT items produced the same standard error of measurement as 125 items in a random order. Subsequent validation in 32 public school classrooms shows 40 CAT items (approximately 3 minutes) can achieve high correlations (r = .89 for 1st grade, r = .73 for 2nd grade) with alternative 15-20 minutes individually proctored reading assessments. Our findings suggest that ROAR-CAT is a promising tool for efficiently and accurately measuring single word reading ability in reading research and educational practice. Furthermore, our development process serves as a model for creating adaptive online assessments that bridge research and practice.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Developing Browser-based Computerized Adaptive Testingmentioning

confidence: 99%

Section: Participantsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ROAR-CAT: Rapid Online Assessment of Reading ability with Computerized Adaptive Testing

Ma,

Richie-Halford,

Burkhardt

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Correlation to reading skills, and score reportsThe ultimate goal of developing an efficient, online PA task is to create a task that a) predicts reading performance and b) provides actionable information about the individual's PA skills. To accomplish the first goal, we correlate ROAR-PA performance to performance on another task of the ROAR test battery: ROAR-Single Word Recognition (SWR)(Yeatman et al, 2021) as well as reading skills assessed with two of the most widely used standardized, individually administered measures of reading, the Woodcock-Johnson Letter-Word-Identification (WJ LW-ID) and Word-attack (WJ-WA) tests (Schrank et al, 2014). Reading scores can only be assessed after children learn how to read, therefore we only acquired reading performance of 111 participants (Age: 5.66 -9.50, μ = 7.67, 𝞼 = 1.13; Sex: 59 F, 52 M) on the WJ and ROAR-SWR.…”

mentioning

confidence: 99%

Rapid Online Assessment of Reading and Phonological Awareness (ROAR-PA)

Gijbels¹,

Burkhardt²,

Anya³

et al. 2023

Preprint

View full text Add to dashboard Cite

Phonological awareness (PA) is at the foundation of reading development: PA is introduced before formal reading instruction, has predictive value for later reading abilities, is a primary target for early intervention, and is considered one of the core mechanisms in developmental dyslexia. Conventional approaches to assessing PA are time-consuming and resource intensive: assessments must be individually administered, require expertise, and scoring verbal responses is challenging and subjective. Therefore, we introduce a rapid, automated, online measure of PA — The Rapid Online Assessment of Reading - Phonological Awareness (ROAR-PA) — that can be widely implemented in classrooms and research studies without a test administrator. We explored whether this gamified, online task, that relies on touchscreen/click responses, can serve as an accurate and reliable measure of PA and as a good predictor of reading development. We found that ROAR-PA is well correlated with standardized measures of PA (CTOPP-2, r = .80) and reading (Woodcock-Johnson, r = .50), for children from Pre-K through fourth grade and achieves exceptional reliability (𝜶 = .96) in a 12-minute automated, online assessment. Furthermore, validation in 50 first and second grade classrooms shows reliable implementation in large, public school classrooms with predictive value of future reading development.

show abstract

Enhancing Online Speech and Language Assessment: Item Development for the Remote Adult Language Experiment (ReAL-E) Tool

Lancaster,

Parks,

Bashford

et al. 2023

Preprint

View full text Add to dashboard Cite

Main objectiveOur main objective is to create an online toolbox that gathers data on productive language skills so that we can recruit large sample sizes in future studies (e.g., hundreds of thousands). Our toolbox, the Remote Adult Language Experiment (ReAL-E), is unique from other online tools that gather language related traits by focusing on productive language responses both spoken and written. We selected seven tasks that tap language related traits and are frequently used in research with adults. In this study, we describe our first stage of development and redesign. We assessed feasibility and user experience in two studies.Study 1We describe how we developed the online tasks and novel items. We then solicited expert review from three content experts. We asked them to rate our adaptions, face validity, and effectiveness. Across tasks, experts highlighted concerns with instructions, task presentation, item difficulty, and potential confounds to future data analysis. We implemented several changes to address their concerns.Study 2We conducted user experience testing using semi-structured interviews and assessed user performance on the tasks to determine if performance aligned with past research. Users indicated that the ReAL-E was easy to navigate, time demands were appropriate, and instructions were sufficiently clear. We identified five areas of improvement to implement in the next round of revisions. Performance on most tasks was within expected ranges, except for timed nonword reading, which had lower performance than anticipated based on prior literature.Main conclusionWe have initial evidence of feasibility for the ReAL-E and several areas of improvement. We are working on developing methods to automatically score these tasks, while continuing to refine the ReAL-E.

show abstract

Rapid Online Assessment of Reading Ability

Cited by 3 publications

References 71 publications

ROAR-CAT: Rapid Online Assessment of Reading ability with Computerized Adaptive Testing

ROAR-CAT: Rapid Online Assessment of Reading ability with Computerized Adaptive Testing

Rapid Online Assessment of Reading and Phonological Awareness (ROAR-PA)

Enhancing Online Speech and Language Assessment: Item Development for the Remote Adult Language Experiment (ReAL-E) Tool

Contact Info

Product

Resources

About