Adaptation to visual feedback delays on touchscreens with hand vision

Abstract

Direct touch finger interaction on a smartphone or a tablet is now ubiquitous. However, the latency inherent in digital computation produces an average feedback delay of ~ 75 ms between the action of the hand and its visible effect on digital content. This delay has been shown to affect users' performance, but it is unclear whether users adapt to this delay and whether it influences skill learning. Previous work studied adaptation to feedback delays but only for longer delays, with hidden hand or indirect devices. This paper addresses adaptation to touchscreen delay in two empirical studies involving the tracking of a target moving along an elliptical path. Participants were trained for the task either at the minimal delay the system allows (~ 9 ms) or at a longer delay equivalent to commercialized touch devices latencies (75 ms). After 10 training sessions over a minimum of 2 weeks (Experiment 1), participants adapt to the delay. They also display long-term retention 7 weeks after the last training session. This adaptation generalizes to a similar tracking path (e.g., infinity symbol). We also observed generalization of learning from the longer delay to the minimal-delay condition (Experiment 2). The delay thus does not prevent the learning of tracking skill, which suggests that delay adaptation and tracking skill could be two separate components of learning.

Looking away from a moving target does not disrupt the way in which the movement toward the target is guided

“…Another possible reason for participants being less proficient at adjusting their ongoing movements to the delays in Experiment 2 is that imposing an interception point prevented participants from making spatial corrections ( Brenner & Smeets, 2015 ). Imposing a path as in the present study (and in Cattan et al, 2018 ), physically restricting the movement to a certain path ( Tresilian & Houseman, 2005 ; Tresilian & Lonergan, 2002 ), or even just specifying the position at which participants have to hit a target ( Brenner & Smeets, 2011 ; de la Malla et al, 2017 ) restricts participants to changing their movement speed when they need to adjust their movements, which may be less effective than adjusting the interception point because the latency to do so is longer ( Brenner & Smeets, 2015 ). Irrespective of why the additional requirement influenced performance, it was successful in making participants regularly direct their gaze toward the cursor ( Figure 7 ), even when the cursor was not very difficult to see (before contrast was very low), presumably because doing so helped participants achieve the goal of moving along the line.…”

“…The randomized delays in the current experiments encouraged participants to control the cursor on the basis of visual information provided during the trial, as in our previous study ( Cámara et al, 2018 ). Most studies investigating the effect of temporal delays on performance have used constant delays ( Cattan, Perrier, Bérard, Gerber, & Rochet-Capellan, 2018 ; Cunningham et al, 2001b ; Foulkes & Miall, 2000 ; Miall & Jackson, 2006 ; Rohde et al, 2014 ; Vercher & Gauthier, 1992 ) or delays that changed in a predictable way ( de la Malla et al, 2014 ; Knelange & López-Moliner, 2019 ). In such cases, participants can adjust their actions across consecutive trials (adapt to the delay) in addition to adjusting ongoing movements.…”

How the timing of visual feedback influences goal-directed arm movements: delays and presentation rates

“…Indeed, people are less precise when intercepting moving targets with a cursor that is displayed at the position of their hidden finger (85 Hz frame rate; 60 ms additional delay) than when they do so with the visible finger itself (de la Malla et al 2012). Similarly, when moving their finger across a touchscreen to track a moving target with a cursor, people make larger spatial errors if there is an additional delay of 75 ms than if the additional delay is only 9 ms (Cattan et al 2018). Moreover, increasing the additional delay by tens of milliseconds made movements noticeably slower when guiding a cursor or crosshair to a target on a screen by moving a computer mouse (Ivkovic et al 2015;MacKenzie and Ware 1993;Spjut et al 2019) or 'dragging' an item to a target on a touchscreen (Jota et al 2013).…”