Phishing IQ Tests Measure Fear, Not Ability

Abstract

We argue that phishing IQ tests fail to measure susceptibility to phishing attacks. We conducted a study where 40 subjects were asked to answer a selection of questions from existing phishing IQ tests in which we varied the portion (from 25% to 100%) of the questions that corresponded to phishing emails. We did not find any correlation between the actual number of phishing emails and the number of emails that the subjects indicated were phishing. Therefore, the tests did not measure the ability of the subjects. To further confirm this, we exposed all the subjects to existing phishing education after they had taken the test, after which each subject was asked to take a second phishing test, with the same design as the first one, but with different questions. The number of stimuli that were indicated as being phishing in the second test was, again, independent of the actual number of phishing stimuli in the test. However, a substantially larger portion of stimuli was indicated as being phishing in the second test, suggesting that the only measurable effect of the phishing education (from the point of view of the phishing IQ test) was an increased concern-not an increased ability.

Lessons from a real world evaluation of anti-phishing training

“…Interview studies have been conducted to gain insights into users' mental models and decision processes [7,18]. Laboratory experimental studies where participants played a fictitious role and used personal information associated with that role have been used to test users' susceptibility to phishing attacks and evaluate the effectiveness of anti-phishing toolbars and training materials [2,6,14,19,20,21,31]. Laboratory experimental studies where participants used their own credentials have been used to evaluate the effectiveness of mutual authentication tools [30].…”

Teaching Johnny not to fall for phish

“…Others have commented that education is not a feasible solution for phishing and other security attacks because security education "puts the burden on the wrong shoulder" [Nielsen 2004] and security is a secondary goal for users [Evers 2006]. Furthermore, evaluations of some security-related educational materials have found these materials to be ineffective [Anandpara et al 2007;Jackson et al 2007]. In general, we found that existing online anti-phishing training materials tend to make users more cautious about opening and acting upon email, but do not teach people how to determine whether a website or email is fradulent (See Section 4).…”

“…Users are scored based on how well they can identify which emails are legitimate and which are not. However, while this approach raises awareness about phishing, a user study found that it is not an effective training method [Anandpara et al 2007]. Nonetheless, the idea of integrating self tests with other anti-phishing training materials warrants further examination.…”

Delayed password disclosure

“…Typical users know to an even lesser extent that it may be clicked to display a certificate containing information about the purported identity of the website, along with information on the identity of the authority which is vouching for its identity. 1 The result is that when users visit phishing sites, the lack of appropriate cryptographic authentication is not noticed, and users tend to rely on indicators that they have come to trust in the non-digital world, such as the presence of brand logos and other look-and-feel branding intangibles, regardless of how inapplicable these indicators are as authenticators in the digital realm. A further, or more detailed explanation of the general phishing problem is beyond the scope of the current presentation, but interested readers are directed to [28] for comprehensive coverage of the issue.…”