PrivacyScout: Assessing Vulnerability to Shoulder Surfing on Mobile Devices

Abstract

One approach to mitigate shoulder surfing attacks on mobile devices is to detect the presence of a bystander using the phone’s front-facing camera. However, a person’s face in the camera’s field of view does not always indicate an attack. To overcome this limitation, in a novel data collection study (N=16), we analysed the influence of three viewing angles and four distances on the success of shoulder surfing attacks. In contrast to prior works that mainly focused on user authentication, we investigated three common types of content susceptible to shoulder surfing: text, photos, and PIN authentications. We show that the vulnerability of text and photos depends on the observer’s location relative to the device, while PIN authentications are vulnerable independent of the observation location. We then present PrivacyScout – a novel method that predicts the shoulder-surfing risk based on visual features extracted from the observer’s face as captured by the front-facing camera. Finally, evaluations from our data collection study demonstrate our method’s feasibility to assess the risk of a shoulder surfing attack more accurately.

Comparing Dwell time, Pursuits and Gaze Gestures for Gaze Interaction on Handheld Mobile Devices

“…In their review of gaze-enabled handheld mobile devices, Khamis et al [43] argued that this brings a myriad of opportunities, such as gaze-based interaction on the move and inthe-wild analysis of gaze behaviour on mobile devices. Applications of this include improving interaction on mobile devices [16,20,21], security applications [42] including authentication [48] and privacy protection [9,72,83], as well as in-the-wild gaze behaviour analysis [5,6,80]. On the downside, eye tracking on mobile devices comes with a unique set of challenges.…”

An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices

“…Sensor Feature Modality Leverage Application [40] Eye Response ERICA gaze gesture -E interface control [48] external IR cam dwell & gesture -E target selection [145] Tobii X60 dwell -I attention analysis [206] modified prototype gaze gesture -E interface control [193] Tobii EyeX dwell touch I interface control [224] phone camera gaze basic events -E user authentication [125] Facelab 5 gaze basic events -I Website usability test [150] phone camera dwell -I intention inference [120] phone camera gaze gesture touch E user authentication [261] Tobii eyeX gaze basic events touch I gaze adaptive UI [267] phone camera gaze gesture -E gaze input [121] external RGB cam gaze gesture touch E user authentication [227] phone camera gaze basic events -I attention inference [219] Tobii 4C dwell touch E text editing aids [202] Tobii 4C dwell touch E text interface control [167] phone camera dwell voice E map navigation [220] phone camera eye image -I ocular exam [239] phone camera dwell touch E cross-device control [58] phone camera dwell eyelid E interface control [112] phone camera gaze basic events -I attention analysis [254] phone camera gaze gesture touch E gaze-assist input [133] phone camera dwell hand motion E interface control [180] Tobii X2 gaze basic events -I attention analysis [153] phone camera dwell -E target selection [123] Tobii 4C dwell voice I implicit note-taking [122] external RGB cam gaze gesture touch E user authentication [274] phone camera dwell voice E text correction [10] external RGB cam gaze & face -I user privacy [266] phone camera eye image -I holding posture detection [275] external RGB cam gaze basic events * E gaze command definition [103] Tobii X2 gaze basic events -I attention analysis …”

Understanding Shoulder Surfer Behavior and Attack Patterns Using Virtual Reality

“…This finding could be interesting for the design of approaches to mitigate shoulder surfing. For example, recent works [5,32,33] showed that using the front-facing camera, a smartphone could analyze the motion or gaze of bystanders in the visual field of view. A mitigation system could then use a prediction model to identify potential attacks and trigger an intervention.…”