Untangling comparison bias in inductive item tree analysis based on representative random quasi-orders

Ünlü, Ali; Schrepp, Martin

doi:10.1016/j.mathsocsci.2015.03.005

Cited by 6 publications

(18 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Algorithms for mining quasi-orders have to be compared based on demanding simulation studies. In particular, Schrepp and Ünlü (2015) and Ünlü and Schrepp (2015) discussed the importance of representative random quasi-order samples needed in extensive simulation studies for the reliable comparison of data mining algorithms used to reconstruct relational dependencies among behavioral test items (cf. Section 1).…”

Section: Discussionmentioning

confidence: 99%

“…As shown in Ünlü and Schrepp (2015), these ad hoc random processes yield non-representative quasi-order samples. In decreasing order of representativeness were the averaged followed by the absolute normal variants, whereas both variants of the uniform method produced the worst results with random samples of overly represented large quasi-orders.…”

Section: State-of-the-art Sampling Techniquesmentioning

confidence: 97%

“…For example, the counts are 9, 535, 241/642, 779, 354/63, 260, 289, 423/8, 977, 053, 873, 043 for 7/8/9/10 items, respectively. In Ünlü and Schrepp (2015), the census-like sampling approach was demonstrated with six items, where we have a total of 209, 527 quasi-orders in the population. In this method, each draw, if feasible, is a quasi-order, although the equal sampling probability for each quasi-order may be very small.…”

Section: State-of-the-art Sampling Techniquesmentioning

confidence: 99%

“…Why sampling quasi-orders is necessary for us was also concretized in Ünlü and Schrepp (2015). In their study, the importance of representative sampling of quasi-orders and the biases and errors induced by non-representative samples were clearly evidenced.…”

Section: Introductionmentioning

confidence: 97%

“…The representativeness of the quasi-orders employed in extensive simulation studies was seen to be an important requirement for the sound comparison of such exploratory data analysis methods as item tree analysis. In particular, Ünlü and Schrepp (2015) found that utilizing non-representative quasi-order samples yielded biased simulative assessment results with regard to the recovery and coverage qualities associated with the existing item tree analysis algorithms. For further motivation of representative random quasi-orders (see also Schrepp and Ünlü, 2015, Section Introduction).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Toward a Principled Sampling Theory for Quasi-Orders

Ünlü

Schrepp

2016

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Quasi-orders, that is, reflexive and transitive binary relations, have numerous applications. In educational theories, the dependencies of mastery among the problems of a test can be modeled by quasi-orders. Methods such as item tree or Boolean analysis that mine for quasi-orders in empirical data are sensitive to the underlying quasi-order structure. These data mining techniques have to be compared based on extensive simulation studies, with unbiased samples of randomly generated quasi-orders at their basis. In this paper, we develop techniques that can provide the required quasi-order samples. We introduce a discrete doubly inductive procedure for incrementally constructing the set of all quasi-orders on a finite item set. A randomization of this deterministic procedure allows us to generate representative samples of random quasi-orders. With an outer level inductive algorithm, we consider the uniform random extensions of the trace quasi-orders to higher dimension. This is combined with an inner level inductive algorithm to correct the extensions that violate the transitivity property. The inner level correction step entails sampling biases. We propose three algorithms for bias correction and investigate them in simulation. It is evident that, on even up to 50 items, the new algorithms create close to representative quasi-order samples within acceptable computing time. Hence, the principled approach is a significant improvement to existing methods that are used to draw quasi-orders uniformly at random but cannot cope with reasonably large item sets.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: State-of-the-art Sampling Techniquesmentioning

confidence: 97%

Section: State-of-the-art Sampling Techniquesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 97%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Toward a Principled Sampling Theory for Quasi-Orders

Ünlü

Schrepp

2016

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

Generalized inductive item tree analysis

Ünlü

Schrepp

2021

Journal of Mathematical Psychology

Self Cite

View full text Add to dashboard Cite

Inductive item tree analysis is an established method of Boolean analysis of questionnaires. By exploratory data analysis, from a binary data matrix, the method extracts logical implications between dichotomous test items based on their positive item scores. For example, assume that we have the problems i and j of a test that can be solved or failed by subjects. With inductive item tree analysis, an implication between the items i and j can be uncovered, which has the interpretation "If a subject is able to solve item i, then this subject is also able to solve item j".Hence, in the current form of the method, (a) solely dichotomous items are considered, and (b) conclusions are drawn from only positive item scores. In this paper, we provide extensions to these restrictions. First, as remedy for (b), we focus on the dichotomous formulation of the inductive item tree analysis algorithm and describe a procedure of how to extend the dichotomous variant to also include negative item scores. Second, to address (a), we further extend our approach to the general case of polytomous items, when more than two answer categories are possible. Thus, we introduce extensions of inductive item tree analysis that can deal with nominal polytomous and ordinal polytomous answer scales. To show their usefulness, the dichotomous and polytomous extensions proposed in this paper are illustrated with empirical data and in a simulation study.

show abstract

Observed hierarchy of student proficiency with period, frequency, and angular frequency

Young

Heckler

2018

Phys. Rev. Phys. Educ. Res.

View full text Add to dashboard Cite

In the context of a generic harmonic oscillator, we investigated students' accuracy in determining the period, frequency, and angular frequency from mathematical and graphical representations. In a series of studies including interviews, free response tests, and multiple choice tests developed in an iterative process, we assessed students in both algebra-based and calculus-based, traditionally instructed university-level introductory physics courses. Using the results, we categorized nine skills necessary for proficiency in determining period, frequency, and angular frequency. Overall results reveal that, postinstruction, proficiency is quite low: only about 20%-40% of students mastered most of the nine skills. Next, we used a semiquantitative, intuitive method to investigate the hierarchical structure of the nine skills. We also employed the more formal item tree analysis method to verify this structure and found that the skills form a multilevel, nonlinear hierarchy, with mastery of some skills being prerequisite for mastery in other skills. Finally, we implemented a targeted, 30-min group-work activity to improve proficiency in these skills and found a 1 standard deviation gain in accuracy. Overall, the results suggest that many students currently lack these essential skills, targeted practice may lead to required mastery, and that the observed hierarchical structure in the skills suggests that instruction should especially attend to the skills lower in the hierarchy.

show abstract

Untangling comparison bias in inductive item tree analysis based on representative random quasi-orders

Cited by 6 publications

References 10 publications

Toward a Principled Sampling Theory for Quasi-Orders

Toward a Principled Sampling Theory for Quasi-Orders

Generalized inductive item tree analysis

Observed hierarchy of student proficiency with period, frequency, and angular frequency

Contact Info

Product

Resources

About