Measuring recognition performance using computer-based and paper-based methods

Abstract

Using a within-subjects design, we administered to 83 naval pilots and flight officers computerbased and paper-based tests to assess recognition of aircraft silhouettes in order to determine the relative reliabilities and validities of these two measurement modes. Estimates of internal consistencies, equivalences, and discriminative validities were computed for multiple performance measures. It was established that the relative reliabilities and validities derived for these two assessment schemes were contingent on the employed multivariate measurement criteria, that is, percentage correct responses, average response latency, and average degree of confidence in recognition judgments, as well as the statistical criteria used to ascertain the comparative quality of these two modes of testing.

The Equivalence of Paper-and-Pencil and Computer-Based Testing

“…This point about dissimilarity was reinforced in a study by Federico ( 1991 ). In his research on the equivalence of computer-based and paper-and-pencil testing of the ability of 83 naval pilots to recognize aircraft silhouettes, Federico found that there was no significant difference in reliability or internal consistency between these two modes of administration.…”

Using the Item Response Theory (IRT) for Educational Evaluation Through Games

“…The Table 5 presents three related works surveyed, namely: (1) tests with U.S. pilots in the identification of aircraft using the same test in the form of paper and a computer game, by investigating the statistical correlation feasibility for both (Federico, 1991); (2) test with elderlies in three types of devices (mouse, touch screen and tablet-with-stylus) together with testing for levels of cognitive functions MMSE (Mini-Mental State Examination) and social functions, to profile the elderlies and their relationship to the technologies (Muni, 2006); (3) trials with players of massive games, computing the hours of use in relation to their level of sociability, seeking to verify through a psychometric questionnaire which measures the regulation of behavior SSQR (Short Self-Regulation Questionnaire), the level of sociability and/or to detect the level of dependence on virtual life that has the players (Seay, 2007). The main differences between the PPG and related works are:…”

Um Modelo para Geração de Perfis de Usuários Baseado em Técnicas de Psi-cometria

“…A Tabela 5 apresenta os três trabalhos relacionados pesquisados, ou seja: (1) testes com pilotos norteamericanos na identificação de aeronaves, aplicando o mesmo teste na forma de papel e de um jogo computadorizado, apurando através da correlação estatística a viabilidade de ambos [4]; (2) [11]. As principais diferenças encontradas entre os trabalhos relacionados e o PPG, são: (1) os testes com os pilotos provam a viabilidade entre testes via papel e o computadorizado, o PPG utiliza qualquer tipo de teste para aferir um perfil de habilidade e ou comportamento; (2) os testes com idosos apuram a habilidade entre diversos tipos de equipamentos e a saúde mental dos mesmos, o PPG fornece um banco de itens que prospecta, após a aferição do especialista, o perfil dos usuários de forma autônoma, ou seja, sem a necessidade de intervenção do especialista; (3) os testes com jogos massivos tentam aferir a sociabilidade dos jogadores, procurando relacionar as horas gastas no jogo e o seu nível de autoregulação, o PPG utiliza o modelo matemático TRI proposto pela Psicometria para prospectar o perfil dos avaliados oferecendo interface flexível para criação de sistemas capazes de utilizarem o PPG.…”