Obtaining the best value for money in adaptive sequential estimation

Kujala, Janne V.

doi:10.1016/j.jmp.2010.09.002

Cited by 8 publications

(12 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Section 4.4, these results are generalized to the situation with random costs of observation associated with each placement as discussed above. The heuristically justified, myopic placement strategy proposed in Kujala [5] turns out to be asymptotically optimal also in the sense of the present paper, supporting the view that this strategy is the most natural generalization of the greedy information maximization strategy to the situation where the costs of observation can vary. We give concrete examples of the optimality results in Section 5 and then end with general discussion in Section 6.…”

Section: Introductionsupporting

confidence: 78%

“…The goal considered in Kujala [5] is maximization of the expected information gain of a sequential experiment that terminates when the total cost overruns a given budget. To achieve this goal, the heuristic of maximizing the expected information gain I t (Θ; Y x ) divided by the expected cost E t (C x ) on each trial is proposed.…”

Section: Varying Cost Of Observationmentioning

confidence: 99%

“…The theoretical framework of this paper is that of Bayesian adaptive estimation with an information based objective function (see, e.g., MacKay [9], Kujala and Lukka [7], Kujala [6]). Following the notation of Kujala [5,6], the basic problem we consider is the estimation of an unobservable random variable Θ : Ω → Obased on a sequence y x1 , . .…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Asymptotic optimality of myopic information-based strategies for Bayesian adaptive estimation

Kujala¹

2016

Bernoulli

Self Cite

View full text Add to dashboard Cite

This paper presents a general asymptotic theory of sequential Bayesian estimation giving results for the strongest, almost sure convergence. We show that under certain smoothness conditions on the probability model, the greedy information gain maximization algorithm for adaptive Bayesian estimation is asymptotically optimal in the sense that the determinant of the posterior covariance in a certain neighborhood of the true parameter value is asymptotically minimal. Using this result, we also obtain an asymptotic expression for the posterior entropy based on a novel definition of almost sure convergence on "most trials" (meaning that the convergence holds on a fraction of trials that converges to one). Then, we extend the results to a recently published framework, which generalizes the usual adaptive estimation setting by allowing different trial placements to be associated with different, random costs of observation. For this setting, the author has proposed the heuristic of maximizing the expected information gain divided by the expected cost of that placement. In this paper, we show that this myopic strategy satisfies an analogous asymptotic optimality result when the convergence of the posterior distribution is considered as a function of the total cost (as opposed to the number of observations).

show abstract

Section: Introductionsupporting

confidence: 78%

Section: Varying Cost Of Observationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Asymptotic optimality of myopic information-based strategies for Bayesian adaptive estimation

Kujala¹

2016

Bernoulli

Self Cite

View full text Add to dashboard Cite

show abstract

“…7. This would be asymptotically equivalent to the ratio of expected gain to expected time cost in Kujala (2010). Further research is warranted to investigate the performance of such a metric, and to compare its performance against the original MIT and MIT-S methods.…”

Section: Summary and Discussionmentioning

confidence: 99%

A simplified version of the maximum information per time unit method in computerized adaptive testing

2016

View full text Add to dashboard Cite

In this article, we propose a simplified version of the maximum information per time unit method (MIT; Fan, Wang, Chang, & Douglas, Journal of Educational and Behavioral Statistics 37: 655-670, 2012), or MIT-S, for computerized adaptive testing. Unlike the original MIT method, the proposed MIT-S method does not require fitting a response time model to the individual-level response time data. It is also computationally efficient. The performance of the MIT-S method was compared against that of the maximum information (MI) method in terms of measurement precision, testing time saving, and item pool usage under various item response theory (IRT) models. The results indicated that when the underlying IRT model is the two-or three-parameter logistic model, the MIT-S method maintains measurement precision and saves testing time. It performs similarly to the MI method in exposure control; both result in highly skewed item exposure distributions, due to heavy reliance on the highly discriminating items. If the underlying model is the one-parameter logistic (1PL) model, the MIT-S method maintains the measurement precision and saves a considerable amount of testing time. However, its heavy reliance on time-saving items leads to a highly skewed item exposure distribution. This weakness can be ameliorated by using randomesque exposure control, which successfully balances the item pool usage. Overall, the MIT-S method with randomesque exposure control is recommended for achieving better testing efficiency while maintaining measurement precision and balanced item pool usage when the underlying IRT model is 1PL.Keywords Maximum information per time unit . Response time . Computerized adaptive testing . Item exposure control . Test efficiency Tailored testing or adaptive testing is known for its "efficiency" over traditional linear testing. The idea is to find the most suitable items from a large bank for each examinee. In the simplest case of educational testing, highly capable test takers should not be asked many easy questions and struggling test takers should not be presented with too many difficult questions. Delivering questions that are too easy may cause boredom while delivering items that are too difficult may cause anxiety or other forms of construct irrelevant meta-cognitive activity. In addition, responses to items that are too easy or too hard provide little information from the perspective of testing efficiency and are not helpful in quickly zeroing in on an examinee's ability.Adaptive testing seeks to avoid these difficulties by delivering assessment tasks that are tailored to each examinee's ability. When the maximum information (MI) method is used for item selection (Weiss, 1982) adaptive testing can achieve the same level of measurement precision with as few as half the number of items required of linear tests. Following this approach, the ability estimate of an examinee is updated every time he or she responds to a question (Lord, 1980). The MI method then identifies the most informative item in the ...

show abstract

“…Recent developments are, for instance, the Ψ-method introduced in Kontsevich and Tyler (1999), the consideration of multidimensional stimulus spaces in Kujala and Lukka (2006), and the framework of adaptive design optimization (ADO) for model discrimination proposed in Cavagnaro et al (2010). Another interesting example is given by Kujala (2010) who considers random cost as a further constraint for experimental observations.…”

Section: Introductionmentioning

confidence: 99%

A predictive approach to nonparametric inference for adaptive sequential sampling of psychophysical experiments

Poppe

Benner

Elze

2012

Journal of Mathematical Psychology

View full text Add to dashboard Cite

We present a predictive account on adaptive sequential sampling of stimulus-response relations in psychophysical experiments. Our discussion applies to experimental situations with ordinal stimuli when there is only weak structural knowledge available such that parametric modeling is no option. By introducing a certain form of partial exchangeability, we successively develop a hierarchical Bayesian model based on a mixture of Pólya urn processes. Suitable utility measures permit us to optimize the overall experimental sampling process. We provide several measures that are either based on simple count statistics or more elaborate information theoretic quantities. The actual computation of information theoretic utilities often turns out to be infeasible. This is not the case with our sampling method, which relies on an efficient algorithm to compute exact solutions of our posterior predictions and utility measures. Finally, we demonstrate the advantages of our framework on a hypothetical sampling problem.

show abstract

Obtaining the best value for money in adaptive sequential estimation

Cited by 8 publications

References 15 publications

Asymptotic optimality of myopic information-based strategies for Bayesian adaptive estimation

Asymptotic optimality of myopic information-based strategies for Bayesian adaptive estimation

A simplified version of the maximum information per time unit method in computerized adaptive testing

A predictive approach to nonparametric inference for adaptive sequential sampling of psychophysical experiments

Contact Info

Product

Resources

About