Methodology for a Field Study of Anti-malware Software

Abstract

Abstract. Anti-malware products are typically evaluated using structured, automated tests to allow for comparison with other products and for measuring improved efficiency against specific attacks. We propose that anti-malware testing would benefit from field studies assessing effectiveness in more ecologically valid settings. This paper presents our methodology for conducting a 4-month field study with 50 participants, including discussion of deployment and data collection, encouraging retention of participants, ethical concerns, and our experience to date.

A clinical study of risk factors related to malware infections

“…The details of the methodology have been published elsewhere [14], but we nonetheless provide a brief summary here. The study monitored real-world computer usage through diagnostics and logging tools, monthly interviews and questionnaires, and in-depth investigation of any potential infections.…”

“…Our experimental protocol [14,13] describes in detail the monthly procedure for identifying and classifying suspicious files that were not detected by the AV. This process of identification and classification is based on user reporting of suspicious machine behaviour, the analysis of logs from the monitoring tools, the results of automated queries to on-line sources with respect to processes found on the machine, file and start-up programme databases (obtained automatically by scripts that we wrote), and any other relevant piece of information that the technician conducting the review might deem relevant.…”

“…Section 2 presents the related work. Section 3 describes the study and provides a summary of its methodology, a detailed description of which is also given in a previous methodology publication [14]. In Section 4, we describe and discuss in detail the results of the study in terms of threat detections by AV, missed detections, and identification of potential risk factors related to user characteristics and behaviour.…”