On the Creation of Representative Samples of Random Quasi-Orders

Abstract

Dependencies between educational test items can be represented as quasi-orders on the item set of a knowledge domain and used for an efficient adaptive assessment of knowledge. One approach to uncovering such dependencies is by exploratory algorithms of item tree analysis (ITA). There are several methods of ITA available. The basic tool to compare such algorithms concerning their quality are large-scale simulation studies that are crucially set up on a large collection of quasi-orders. A serious problem is that all known ITA algorithms are sensitive to the structure of the underlying quasi-order. Thus, it is crucial to base any simulation study that tries to compare the algorithms upon samples of quasi-orders that are representative, meaning each quasi-order is included in a sample with the same probability. Up to now, no method to create representative quasi-orders on larger item sets is known. Non-optimal algorithms for quasi-order generation were used in previous studies, which caused misinterpretations and erroneous conclusions. In this paper, we present a method for creating representative random samples of quasi-orders. The basic idea is to consider random extensions of quasi-orders from lower to higher dimension and to discard extensions that do not satisfy the transitivity property.

Toward a Principled Sampling Theory for Quasi-Orders

“…More generally, orders may be deemed a pivotal contribution to the behavioral and social sciences, amongst others. For a thorough motivation of orders and knowledge or learning space theory, including further references (see Schrepp and Ünlü, 2015). …”

“…In particular, Ünlü and Schrepp (2015) found that utilizing non-representative quasi-order samples yielded biased simulative assessment results with regard to the recovery and coverage qualities associated with the existing item tree analysis algorithms. For further motivation of representative random quasi-orders (see also Schrepp and Ünlü, 2015, Section Introduction).…”

“…Schrepp and Ünlü (2015) introduced an inductive algorithm, which represents the state-of-affairs sampling technique for quasi-orders. In this procedure, trace quasi-orders of lower dimension l are extended, uniformly at random, to dimension l + 1.…”

“…Non-transitive relations are rejected without further analysis. This algorithm improves on two direct methods for drawing representative random quasi-orders (for details, see Schrepp and Ünlü, 2015). However, when the number of items n increases, all of these procedures become computationally too intensive, particularly because the proportion of extensions representing quasi-orders decreases very quickly with n .…”

“…In Section 2, we describe the methods currently available for sampling quasi-orders, including the pertinent inductive uniform extension approach by Schrepp and Ünlü (2015). In Section 3, we introduce the discrete doubly inductive procedure for the construction of potentially all quasi-orders on a finite item set.…”