Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on Touchscreens

Abstract

Suggesting multiple target candidates based on touch input is a possible option for high-accuracy target selection on small touchscreen devices. But it can become overwhelming if suggestions are triggered too often. To address this, we propose SATS, a Suggestionbased Accurate Target Selection method, where target selection is formulated as a sequential decision problem. The objective is to maximize the utility: the negative time cost for the entire target selection procedure. The SATS decision process is dictated by a policy generated using reinforcement learning. It automatically decides when to provide suggestions and when to directly select the target. Our user studies show that SATS reduced error rate and selection time over Shift [51], a magnification-based method, and MUCS, a suggestion-based alternative that optimizes the utility for the current selection. SATS also significantly reduced error rate over BayesianCommand [58], which directly selects targets based on posteriors, with only a minor increase in selection time. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Pointing; Touch screens.

Mobile User Interface Adaptation Based on Usability Reward Model and Multi-Agent Reinforcement Learning

“…After taking an action, the agent immediately receives a reward from the environment, set as a negative value of the expected time cost for each action [30]:…”

Modeling Touch-based Menu Selection Performance of Blind Users via Reinforcement Learning

“…Instead of adopting practical menu items such as "Personal Hotspot" in phone settings, the names of animals and fruits are easier for the participants to recognize and distinguish from other items. Following previous works in menu selection [15,47], smartphone APP launching [50], and target selection [3,18,45,64], we generated number of occurrences of each item from the Zipf distribution:…”

An End-to-End Review of Gaze Estimation and its Interactive Applications on Handheld Mobile Devices

“…Sensor Feature Modality Leverage Application [40] Eye Response ERICA gaze gesture -E interface control [48] external IR cam dwell & gesture -E target selection [145] Tobii X60 dwell -I attention analysis [206] modified prototype gaze gesture -E interface control [193] Tobii EyeX dwell touch I interface control [224] phone camera gaze basic events -E user authentication [125] Facelab 5 gaze basic events -I Website usability test [150] phone camera dwell -I intention inference [120] phone camera gaze gesture touch E user authentication [261] Tobii eyeX gaze basic events touch I gaze adaptive UI [267] phone camera gaze gesture -E gaze input [121] external RGB cam gaze gesture touch E user authentication [227] phone camera gaze basic events -I attention inference [219] Tobii 4C dwell touch E text editing aids [202] Tobii 4C dwell touch E text interface control [167] phone camera dwell voice E map navigation [220] phone camera eye image -I ocular exam [239] phone camera dwell touch E cross-device control [58] phone camera dwell eyelid E interface control [112] phone camera gaze basic events -I attention analysis [254] phone camera gaze gesture touch E gaze-assist input [133] phone camera dwell hand motion E interface control [180] Tobii X2 gaze basic events -I attention analysis [153] phone camera dwell -E target selection [123] Tobii 4C dwell voice I implicit note-taking [122] external RGB cam gaze gesture touch E user authentication [274] phone camera dwell voice E text correction [10] external RGB cam gaze & face -I user privacy [266] phone camera eye image -I holding posture detection [275] external RGB cam gaze basic events * E gaze command definition [103] Tobii X2 gaze basic events -I attention analysis …”