Development and Evaluation of Computer‐administered Analytical Questions for the Graduate Record Examinations General Test

Abstract

Three new computer‐administered items types for the analytical scale of the Graduate Record Examination General Test were developed and evaluated. One item type was a free‐response version of the current analytical reasoning item type. The second item type was a somewhat constrained free‐response version of the pattern identification (or number series) item type in which the student had to state the rule that generated the series. The third item type used the computer to administer yes/no analysis of explanations questions with a limited branching strategy. The computer tests were administered at four ETS regional offices to a sample of students who had previously taken the GRE General Test. Scores from the regular GRE administration and the special computer administration were matched for a sample of 349 students. A number of test administration design issues were identified, including the need to provide adequate practice exercises, design of an interface comfortable for computer‐literate students, and problems with item‐level timing. The pattern identification items were too difficult (or the practice was inadequate), but the other items appeared to function well. There was no evidence that the open‐ended analytical reasoning items were measuring anything beyond what is measured by the current multiple‐choice version of these items.