No abstract

This study aims to compare Sequential Probability Ratio Test (SPRT) and Confidence Interval (CI) classification criteria, Maximum Fisher Information method on the basis of estimated-ability (MFI-EB) and Cut-Point (MFI-CB) item selection methods while ability estimation method is Weighted Likelihood Estimation (WLE) in Computerized Adaptive Classification Testing (CACT), according to the Average Classification Accuracy (ACA), Average Test Length (ATL), and measurement precision under content balancing (Constrained Computerized Adaptive Testing: CCAT and Modified Multinomial Model: MMM) and item exposure control (Sympson-Hetter Method: SH and Item Eligibility Method: IE) when the classification is done based on two, three, or four categories for a unidimensional pool of dichotomous items. Forty-eight conditions are created in Monte Carlo (MC) simulation for the data, generated in R software, including 500 items and 5000 examinees, and the results are calculated over 30 replications. As a result of the study, it was observed that CI performs better in terms of ATL, and SPRT performs better in ACA and correlation, bias, Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE) values, sequentially; MFI-EB is more useful than MFI-CB. It was also seen that MMM is more successful in content balancing, whereas CCAT is better in terms of test efficiency (ATL and ACA), and IE is superior in terms of item exposure control though SH is more beneficial in test efficiency. Besides, increasing the number of classification categories increases ATL but decreases ACA, and it gives better results in terms of the correlation, bias, RMSE, and MAE values.

<p>This content analysis study aims to methodologically evaluate the appropriateness of meta-analyses, conducted on Turkish samples on a variety of topics. Through an exhausting literature review, 80 meta-analyses were gathered together and coded into a detailed Meta-Analysis Evaluation Form.  The form consisted of 59 items (1 = Not Present, 2 = Present and 3 = Not Mentioned) both regarding the study and substantial characteristics. Two researchers coded the studies and the reliability of the coding of five studies indicated no problems with consistencies of the codings (Kappa= .90). According to the results, the most often encountered problem in meta-analyses was reporting both the fixed and random effects analyses without making a priori decision about the model choice. It was found that 60.0% of the meta-analyses investigated by the current study excluded studies conducted abroad which resulted underrepresentation of the literature.  Furthermore, the studies suffered from a small sample size issues. The methodology (how the studies were selected, coding form, reliability of the codings and etc.) was not explained clearly in more than a quarter of the studies. Therefore, it would be hard to claim that they have sufficient level of internal and external validity. It was hoped that researchers may benefit from the results of the current study to conduct better quality meta-analysis in the future.</p><p> </p><p><strong>Özet</strong></p><p>Bu içerik analizi çalışmasının amacı Türkiye'de yapılan meta analiz çalışmalarının metodolojik değerlendirmesinin yapılmasıdır. Meta Analiz Değerlendirme Formu üzerinden Türkiye literatüründeki 80 meta analiz çalışması kodlanmıştır. Değerlendirme formu çalışmaların künyelerini ve meta analiz yönteminin kullanımındaki çeşitlenmeyi içeren 59 (Evet-Hayır-Belirtilmemiş şeklinde cevaplanabilecek) maddeyi kapsamaktadır. İki araştırmacı kodlamaları gerçekleştirmiş ve öncesinde beş çalışmalık bir pilot çalışma üzerinden kodlamaları arasındaki uyum hesaplanmış ve Kappa katsayısı (Kappa= .90) yeterli düzeyde bulunmuştur. Sonuçlara göre meta analiz çalışmalarındaki en belirgin problem herhangi bir tercihte bulunmaksızın sabit ve rasgele etkiler modellerinin birlikte rapor edilmesidir. Çalışmaların %60'ında yurtdışı çalışmalar dahil edilmeksizin Türkiye örneklemindeki çalışmaları kullanarak meta analiz yapılmıştır. Yurtdışı çalışmalara yer veren meta analizlerde ise sayının çok düşük olduğu dolayısıyla örneklemin temsil ediciliğinin düşük olduğu görülmüştür. Meta analizlerde örneklem büyüklüğünün sayıca çok yetersiz olduğu ya da olmadığı görülmüştür. Çalışmaların dörtte birinden fazlasında metodoloji bölümünde çalışmaların nasıl toplandığı, kodlama formu, kodlamaların güvenirliği gibi konular açıklanmamıştır. Bu durum ilgili meta analiz çalışmalarının güvenirlik ve geçerliğini düşürmektedir. Mevcut değerlendirme çalışmasının, gelecekte meta analiz konusunda çalışacak araştırmacılara metodolojik bakımdan daha kaliteli araştırmalar ortaya koymaları hususunda katkı sağlayacağı beklenmektedir.</p>

The purpose of this research was to evaluate the effect of item pool and selection algorithms on computerized classification testing (CCT) performance in terms of some classification evaluation metrics. For this purpose, 1000 examinees’ response patterns using the R package were generated and eight item pools with 150, 300, 450, and 600 items having different distributions were formed. A total of 100 iterations were performed for each research condition. The results indicated that average classification accuracy (ACA) was partially lower, but average test length (ATL) was higher in item pools having a broad distribution. It was determined that the observed differences were more apparent in the item pool with 150 items, and that item selection methods gave similar results in terms of ACA and ATL. The Sympson-Hetter method indicated advantages in terms of test efficiency, while the item eligibility method offered an improvement in terms of item exposure control. The modified multinomial model, on the other hand, was more effective in terms of content balancing.