On-Device Few-Shot Personalization for Real-Time Gaze Estimation

He, Junfeng; Pham, Khoi; Valliappan, Nachiappan; Xu, Pingmei; Roberts, Chase; Lagun, Dmitry; Navalpakkam, Vidhya

doi:10.1109/iccvw.2019.00146

Cited by 73 publications

(38 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In terms of DNN-based pupil tracking methods, He et al [ 1 ] proposed the use of on-device few-shot personalization methods for a 2D pupil estimation based on an unsupervised personalization method, which uses only unlabeled facial images to improve the accuracy of the gaze estimation. However, because this method still depends on a heavy end-to-end convolutional neural network (CNN) model, it is difficult to interpret the model and requires a greater reduction in the number of operations for on-device use in real-time.…”

Section: Related Workmentioning

confidence: 99%

“…Similar to [ 19 , 20 ] also proposed a framework for a few-shot adaptive gaze estimation for the learning of person-specific gaze networks by applying very few calibration samples. However, these GAN user-specific gaze adaptation approaches primarily use a transforming GAN and encoder-decoder architectures, which require fine-tuning to adapt the model to a new subject, and thus user-specificity and personalization are computationally intensive, requiring a large amount of calibration data, and cannot be run on low-spec devices [ 1 ].…”

Section: Related Workmentioning

confidence: 99%

“…In contrast, on-device ML processes information inside the smart device itself without going through a remote cloud server, enabling faster work with a low latency, fewer data security concerns, and support for poor network connectivity. Therefore, it allows the accessibility to a large number of applications, such as gaming and HCI [ 1 ]. On-device ML (on-device artificial intelligence (AI) and ML are often used interchangeably; in the paper, we use the term on-device ML) is closely related to interpretable machine learning (IML) and energy-efficient ML.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Energy Efficient Pupil Tracking Based on Rule Distillation of Cascade Regression Forest

Kim

Jeong

2020

Sensors

View full text Add to dashboard Cite

As the demand for human-friendly computing increases, research on pupil tracking to facilitate human–machine interactions (HCIs) is being actively conducted. Several successful pupil tracking approaches have been developed using images and a deep neural network (DNN). However, common DNN-based methods not only require tremendous computing power and energy consumption for learning and prediction; they also have a demerit in that an interpretation is impossible because a black-box model with an unknown prediction process is applied. In this study, we propose a lightweight pupil tracking algorithm for on-device machine learning (ML) using a fast and accurate cascade deep regression forest (RF) instead of a DNN. Pupil estimation is applied in a coarse-to-fine manner in a layer-by-layer RF structure, and each RF is simplified using the proposed rule distillation algorithm for removing unimportant rules constituting the RF. The goal of the proposed algorithm is to produce a more transparent and adoptable model for application to on-device ML systems, while maintaining a precise pupil tracking performance. Our proposed method experimentally achieves an outstanding speed, a reduction in the number of parameters, and a better pupil tracking performance compared to several other state-of-the-art methods using only a CPU.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Energy Efficient Pupil Tracking Based on Rule Distillation of Cascade Regression Forest

Kim

Jeong

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Personalization is an important topic in computer vision and signal processing with many applications such as recommender systems, smart assistance, speaker verification and keyword spotting [1,2,3]. Personalization of a global model may have conflicting objectives in terms of generalization and personalization.…”

Section: Introductionmentioning

confidence: 99%

Prototype-Based Personalized Pruning

Kim

Chang²,

Yun³

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Nowadays, as edge devices such as smartphones become prevalent, there are increasing demands for personalized services. However, traditional personalization methods are not suitable for edge devices because retraining or finetuning is needed with limited personal data. Also, a full model might be too heavy for edge devices with limited resources. Unfortunately, model compression methods which can handle the model complexity issue also require the retraining phase. These multiple training phases generally need huge computational cost during on-device learning which can be a burden to edge devices. In this work, we propose a dynamic personalization method called prototype-based personalized pruning (PPP). PPP considers both ends of personalization and model efficiency. After training a network, PPP can easily prune the network with a prototype representing the characteristics of personal data and it performs well without retraining or finetuning. We verify the usefulness of PPP on a couple of tasks in computer vision and Keyword spotting.

show abstract

“…Deep learning solutions have produced a revolution in the field of computer vision. These learning models are gaining momentum in eye tracking and gaze estimation as clearly demonstrated in the bunch of papers published in the last years [He et al 2019] [Linden et al 2019] [Guo et al 2019]. One of the most important requirements of deep learning models is the availability of large scale annotated datasets.…”

Section: Introductionmentioning

confidence: 99%

Gaze estimation problem tackled through synthetic images

Garde

Larumbe-Bergera

Bossavit

et al. 2020

ACM Symposium on Eye Tracking Research and Applications

View full text Add to dashboard Cite

In this paper, we evaluate a synthetic framework to be used in the field of gaze estimation employing deep learning techniques. The lack of sufficient annotated data could be overcome by the utilization of a synthetic evaluation framework as far as it resembles the behavior of a real scenario. In this work, we use U2Eyes synthetic environment employing I2Head datataset as real benchmark for comparison based on alternative training and testing strategies. The results obtained show comparable average behavior between both frameworks although significantly more robust and stable performance is retrieved by the synthetic images. Additionally, the potential of synthetically pretrained models in order to be applied in user's specific calibration strategies is shown with outstanding performances.

show abstract

On-Device Few-Shot Personalization for Real-Time Gaze Estimation

Cited by 73 publications

References 29 publications

Energy Efficient Pupil Tracking Based on Rule Distillation of Cascade Regression Forest

Energy Efficient Pupil Tracking Based on Rule Distillation of Cascade Regression Forest

Prototype-Based Personalized Pruning

Gaze estimation problem tackled through synthetic images

Contact Info

Product

Resources

About