Biasing Effects of Non-Representative Samples of Quasi-Orders in the Assessment of Recovery Quality of IITA-Type Item Hierarchy Mining

Ünlü, Ali; Schrepp, Martin

doi:10.1007/978-3-319-25226-1_48

“…However, up to now it has not been possible to create truly representative samples for larger item numbers. Thus, previous simulation studies had to live with approximations, which in some cases had a negative impact on the simulation results and caused biased or incorrect conclusions (Ünlü and Schrepp, 2015 , in press ). The only possible workaround was to draw samples of quasi-orders from the prior constructed set of all quasi-orders, which merely worked for small item sets.…”

Section: Discussionmentioning

confidence: 99%

“…Since all known ITA algorithms are sensitive to the structure of the quasi-order, it is important to use a representative set of quasi-orders as the basis for the simulations. Using non-representative quasi-order samples yielded biased or erroneous simulation results regarding the recovery quality of the algorithms (Ünlü and Schrepp, in press ).…”

Section: Introductionmentioning

confidence: 99%

“…In Ünlü and Schrepp ( 2015 , in press ), it was shown that conclusions concerning the performance of the variants of ITA in earlier publications were biased due to the use of non-representative samples of underlying quasi-orders. By repeating simulations with a representative sample drawn from the set of all quasi-orders on six items, the biased results could be corrected, and a clear picture of the performance of the ITA algorithms was reached.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On the Creation of Representative Samples of Random Quasi-Orders

Schrepp

¹

,

Ünlü

²

2015

Self Cite

View full text Add to dashboard Cite

Dependencies between educational test items can be represented as quasi-orders on the item set of a knowledge domain and used for an efficient adaptive assessment of knowledge. One approach to uncovering such dependencies is by exploratory algorithms of item tree analysis (ITA). There are several methods of ITA available. The basic tool to compare such algorithms concerning their quality are large-scale simulation studies that are crucially set up on a large collection of quasi-orders. A serious problem is that all known ITA algorithms are sensitive to the structure of the underlying quasi-order. Thus, it is crucial to base any simulation study that tries to compare the algorithms upon samples of quasi-orders that are representative, meaning each quasi-order is included in a sample with the same probability. Up to now, no method to create representative quasi-orders on larger item sets is known. Non-optimal algorithms for quasi-order generation were used in previous studies, which caused misinterpretations and erroneous conclusions. In this paper, we present a method for creating representative random samples of quasi-orders. The basic idea is to consider random extensions of quasi-orders from lower to higher dimension and to discard extensions that do not satisfy the transitivity property.

show abstract

“…The input of any IITA analysis is a binary data matrix (subjects represented by rows, items represented by columns), and the output is a quasi-order on the item set. The IITA item hierarchy mining techniques are computational, and typically, evaluated and compared based on extensive simulation studies (Ünlü and Schrepp, 2015(Ünlü and Schrepp, , 2016a(Ünlü and Schrepp, , 2017. At the basis of these simulation studies is a large set of randomly generated quasi-orders, each of which is posited to represent the true dependencies underlying the simulated data.…”

Section: Representative Quasi-ordersmentioning

confidence: 99%

“…The representativeness of a randomly generated subset, or sample, of quasi-orders on an item set means that each quasiorder on the item set has equal probability of being selected as part of the sample. Ünlü and Schrepp (2015Ünlü and Schrepp ( , 2016aÜnlü and Schrepp ( , 2017 showed that the use of non-representative samples of quasi-orders led to biased or erroneous conclusions regarding the recovery and coverage qualities of the IITA algorithms in simulation studies. These authors were able to correct the problems induced by non-representative samples with the use of representative random quasi-orders.…”

Section: Representative Quasi-ordersmentioning

confidence: 99%

Location-scale matching for approximate quasi-order sampling

Ünlü¹,

Schrepp²

2019

Preprint

Self Cite

0

View full text Add to dashboard Cite

Quasi-orders are reflexive and transitive binary relations and have many applications. Examples are the dependencies of mastery among the problems of a psychological test, or methods such as item tree or Boolean analysis that mine for quasi-orders in empirical data. Data mining techniques are typically tested based on simulation studies with unbiased samples of randomly generated quasi-orders. In this paper, we develop techniques for the approximately representative sampling of quasi-orders. Polynomial regression curves are fitted for the mean and standard deviation of quasi-order size as a function of item number. The resulting regression graphs are seen to be quadratic and linear functions, respectively. The extrapolated values for the mean and standard deviation are used to propose two quasi-order sampling techniques. The discrete method matches these location and scale measures with a transformed discrete distribution directly obtained from the sample. The continuous method uses the normal density function with matched expectation and variance. The quasi-orders are constructed according to the biased randomized doubly inductive construction, however they are resampled to become approximately representative following the matched discrete and continuous distributions. In simulations, we investigate the usefulness of these methods. The location-scale matching approach can cope with very large item sets. Close to representative samples of random quasi-orders are constructed for item numbers up to n = 400.

show abstract

Location-Scale Matching for Approximate Quasi-Order Sampling

Ünlü

¹

,

Schrepp

²

2019

Self Cite

View full text Add to dashboard Cite

Quasi-orders are reflexive and transitive binary relations and have many applications. Examples are the dependencies of mastery among the problems of a psychological test, or methods such as item tree or Boolean analysis that mine for quasi-orders in empirical data. Data mining techniques are typically tested based on simulation studies with unbiased samples of randomly generated quasi-orders. In this paper, we develop techniques for the approximately representative sampling of quasi-orders. Polynomial regression curves are fitted for the mean and standard deviation of quasi-order size as a function of item number. The resulting regression graphs are seen to be quadratic and linear functions, respectively. The extrapolated values for the mean and standard deviation are used to propose two quasi-order sampling techniques. The discrete method matches these location and scale measures with a transformed discrete distribution directly obtained from the sample. The continuous method uses the normal density function with matched expectation and variance. The quasi-orders are constructed according to the biased randomized doubly inductive construction, however they are resampled to become approximately representative following the matched discrete and continuous distributions. In simulations, we investigate the usefulness of these methods. The location-scale matching approach can cope with very large item sets. Close to representative samples of random quasi-orders are constructed for item numbers up to n = 400.

show abstract

Biasing Effects of Non-Representative Samples of Quasi-Orders in the Assessment of Recovery Quality of IITA-Type Item Hierarchy Mining

Cited by 3 publications

References 7 publications

On the Creation of Representative Samples of Random Quasi-Orders

On the Creation of Representative Samples of Random Quasi-Orders

Location-scale matching for approximate quasi-order sampling

Location-Scale Matching for Approximate Quasi-Order Sampling

Contact Info

Product

Resources

About