Effects of Apophenia on Multiple-Choice Exam Performance

Abstract

There is a broad literature on the various issues related to effective exam construction applicable to both on-ground and online course delivery. These guidelines tend to support rather close contact between the instructor and the exam. However, to remain competitive, both textbook and course management providers have developed technologies to automate many aspects of exam construction. As test construction becomes automated, the possibility of inadvertently deviating from demonstrated or intuitive guidelines increases. Two experiments were conducted to examine the degree to which apophenia (perceiving patterns in random data) might negatively influence multiple-choice exam performance among college students. Experiment 1 indirectly demonstrated the extent to which certain answer patterns seemed to be tolerated among students (maximum of three repeated answers) in comparison with what might be expected from randomly generated exams from Blackboard. Experiment 2 directly examined the effects of answer patterns on exam performance. Participants' performance declined as the underlying answer patterns became more obvious, and this effect appeared to be particularly strong for the upper level psychology students. The importance and implications of these findings with regard to automated test construction were discussed, and a recommendation is provided.

On a New Algorithm for Removing Repeating Patterns in Similarity Analysis

“…As long as a similarity index concerns copying, repeating patterns should be considered. The similarity caused by a repeating pattern is unlikely a result of copying because the use of a repeating pattern is unusual for normal test-takers (Carlson & Shu, 2007; Paul et al, 2014). Although it is possible that one test-taker may copy from another one who happens to use a repeating pattern, such cases should be rare.…”

“…Haladyna (2004) recommended test developers vary location of the answer key in his guidelines on test construction. Although Attali and Bar-Hillel (2003) showed that test-takers tended to favor options in the middle, which leads to a high likelihood of repeating the same response, Paul, Monda, Olausson, and Reed-Daley (2014) reported that test-takers tended to avoid patterns in responses because they expected the correct answer responses to be random. Carlson and Shu (2007) found that test-takers may perceive a pattern when the same events occurred as few as three times.…”

Position of Correct Option and Distractors Impacts Responses to Multiple-Choice Items: Evidence From a National Test

“…Moreover, it is cautious to present options in a fixed random order for all examinees: Although shuffling options differently for each student is potentially helpful to reduce cheating in tests administered online, it could lead to inequity situations and might be best not to be used, particularly when the test's stakes are high (Bolt et al, 2020). In any case, test instructions should include a warning against the potential misuse of options position to spot CR (Paul et al, 2014).…”

Non-functional biomimicry: utilizing natural patterns in order to provoke attention responses

“…Dunlap further hypothesizes that spiral patterns may also, ultimately, be responsible for the branching patterns of trees, as shown in Figure 3(b). However, this is based on the ability to superimpose spirals over tree patterns and a sceptic would rightly claim that this could be the result of apophenia which, as explained by Paul et al (2011), is the tendency for humans to see patterns in information where no such patterns exist. Undoubtedly, while numerous patterns could be imposed over trees, correlation does not imply causation.…”