The usability of personal digital assistants (PDAs) for assessment of practical performance

Abstract

The administration of an objective structured clinical examination (OSCE) using paper checklists presents problems such as illegible handwriting, missing student names and/or numbers and lost checklists. Calculating and entering results is not only time-consuming, but is subject to human errors, and feedback to students is rarely available. To rectify these problems, personal digital assistants (PDAs) and HaPerT software were acquired to replace paper checklists and provide automated results and feedback. This study sought to determine the usability of the PDA assessment system. The usability of the PDA system was evaluated according to effectiveness, efficiency and user satisfaction. Effectiveness was judged by comparing the results of an OSCE conducted in 2003 (paper-based method) and repeated in 2004 and 2005 (PDA method). Efficiency was determined by calculating the amount of time required to organise the logistics for 2 consecutive PDA OSCEs and deliver results and grades, compared with the time required for the paper-based OSCE. User satisfaction was established by using questionnaires to obtain feedback on the assessors' experiences during their first assessments. An independent groups t-test used to compare the means of scores achieved by students in the PDA and paper-based OSCEs, respectively, showed that the difference in effectiveness was not significant. In terms of efficiency, 77% less time was used for the PDA OSCE in 2004 and 93% less in 2005. Assessor feedback on PDA assessment was overwhelmingly positive. Assessment by PDA was found to be just as effective as and more efficient than paper-based assessment in practical examinations, and was highly rated by assessors.

A review of research methodologies used in studies on mobile handheld devices in K-12 and higher education settings

“…Overall, previous empirical results suggested that there was no statistically significant difference in terms of test scores between students who used the PDAs and students who used the paper based assessment method (Segall et al, 2005;Treadwell, 2006). Efficiency may be defined as the resources consumed in order to obtain a goal (Segall et al, 2005).…”

“…In another example, Fujimura and Doi (2006) implemented the use of mobile phones with Internet capability to assess students' degree of comprehension of course content. Other examples of using handhelds as assessment tools can be found in Segall, Doolen and Porter (2005), Treadwell (2006), and Triantafillou, Georgiadou and Economides (2008). Besides using handhelds to conduct summative assessment (e.g.…”

“…In our review of the research, we found four studies that dealt explicitly with this research topic. Of these four, two studies compared the use of PDAs versus paper and pencil in terms of student test scores, efficiency, and student satisfaction (Segall et al, 2005;Treadwell, 2006), one examined only the satisfaction aspect (Ganger & Jackson, 2003), while one study investigated the design and development issues regarding the implementation of a computerised adaptive test (CAT) on PDAs (Triantafillou et al, 2008).…”

Back to the Future: Electronic Marking of Objective Structured Clinical Examinations and Admission Interviews Using an Online Management Information System in Schools of Health Sciences

“…Segall et al [10] compared the usability of paper and pencil method and Personal Digital Assistant (PDA) based quizzes and found the PDA based quiz was more efficient and superior to the traditional based method [10]. Similarly, Treadwell [9] compared the conduction of a paper based OSCEs with an electronic method. The findings indicated that the electronic method was just as effective and more efficient (less time consuming) than the traditional paper based method.…”

“…However, a number of issues have been highlighted with this method including illegible handwriting, missing details (students' names and student numbers) and lost assessment sheets [9]. Furthermore, it is known that manual calculation of results and entering them into a database is time-consuming and is subject to human errors and feedback is rarely provided to students on their performance after paper based assessments [9]. Despite these issues there is a scarcity of literature regarding the use of computer or OSCE software and the assessment of OSCEs.…”