Efficient deformable 3D face model tracking with limited hardware resources

Goenetxea, Jon; Unzueta, Luis; Dornaika, Fadi; Otaegui, Oihana

doi:10.1007/s11042-019-08515-y

Cited by 8 publications

(8 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, if this is compared to our implementation on an Ubuntu i7 platform (without the application of the rules that increase robustness), our software acceleration method achieves a latency less than 40% of the lowest latency achieved in [ 1 ]. The face alignment applications ([ 8 , 9 , 10 , 12 ]) based on ERTs [ 14 ] achieve a relatively high speed (between 16 and 45 fps) but they concern different applications such as face recognition, pose estimation, etc., and some of them (e.g., [ 9 ]) align a smaller number of landmarks, which is a faster procedure. The yawning detection approaches [ 30 , 32 ] are based on CNNs and operate at a significantly smaller speed.…”

Section: Discussionmentioning

confidence: 99%

“…If one of the relations (10) or ( 11) is not true, then the returned bounding box is assumed invalid and the reference bounding box is used in the current frame potentially extending its size by e.g., 5%. However, if too many successive frames are rejected, the reference bounding box is defined from scratch because the available one is too old to be considered valid.…”

Section: Head Bounding Box Absolute Dimension Restrictions Related To...mentioning

confidence: 99%

“…In [ 9 ], an algorithm is presented that can be executed on an embedded platform (Xilinx FPGA based on ARM A9 processor) that estimates the pose of the hand using 23 landmark points reporting a 30 fps rate. In [ 10 ], J. Goenetxea et al developed a 3D face model tracking application using 68 landmarks achieving a rate of approximately 30 fps. The application presented in [ 10 ] is tested on several commercial devices such as iPhones.…”

Section: Introductionmentioning

confidence: 99%

“…In [ 10 ], J. Goenetxea et al developed a 3D face model tracking application using 68 landmarks achieving a rate of approximately 30 fps. The application presented in [ 10 ] is tested on several commercial devices such as iPhones. A pose estimation system for Unmanned Aerial Vehicles (UAVs) is implemented in [ 11 ].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A High Performance and Robust FPGA Implementation of a Driver State Monitoring Application

Christakos

Petrellis

Mousouliotis

et al. 2023

Sensors

View full text Add to dashboard Cite

A high-performance Driver State Monitoring (DSM) application for the detection of driver drowsiness is presented in this paper. The popular Ensemble of Regression Trees (ERTs) machine learning method has been employed for the alignment of 68 facial landmarks. Open-source implementation of ERTs for facial shape alignment has been ported to different platforms and adapted for the acceleration of the frame processing speed using reconfigurable hardware. Reducing the frame processing latency saves time that can be used to apply frame-to-frame facial shape coherency rules. False face detection and false shape estimations can be ignored for higher robustness and accuracy in the operation of the DSM application without sacrificing the frame processing rate that can reach 65 frames per second. The sensitivity and precision in yawning recognition can reach 93% and 97%, respectively. The implementation of the employed DSM algorithm in reconfigurable hardware is challenging since the kernel arguments require large data transfers and the degree of data reuse in the computational kernel is low. Hence, unconventional hardware acceleration techniques have been employed that can also be useful for the acceleration of several other machine learning applications that require large data transfers to their kernels with low reusability.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Head Bounding Box Absolute Dimension Restrictions Related To...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A High Performance and Robust FPGA Implementation of a Driver State Monitoring Application

Christakos

Petrellis

Mousouliotis

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Dornaika et al [5] used a similar algorithm for recognizing age in facial images. Α 2D landmark detector also based on ERT is presented in [6] for landmark localization balanced between accuracy and high speed. In [7], 68 facial landmarks are aligned, for measuring the reaction of people on advertisements.…”

Section: Introductionmentioning

confidence: 99%

Challenges Towards Hardware Acceleration of the Deformable Shape Tracking Application

Petrellis

Christakos

Zogas

et al. 2021

2021 IFIP/IEEE 29th International Conference on Very Large Scale Integration (VLSI-SoC)

View full text Add to dashboard Cite

In the context of this paper, a shape tracking application based on landmark alignment is transformed to support implementation in Field Programmable Gate Arrays (FPGAs). Towards this direction, several challenges are posed since a) computational intensive operations have to be replaced by faster ones, b) specific loops have to be modified (e.g., unrolled) to support the implementation of operations in parallel with different hardware resources, c) multiple pretrained models have to be compared in terms of speed and accuracy, d) partial loading of the pre-trained models has to be examined in order to fit their parameters in the Block Random Access Memories (BRAMs) of the FPGA for faster access, and e) alternative arithmetic representations have to be evaluated for higher speed and reduced resources. The C++ Deformable Shape Tracking (DEST) implementation of face alignment that is based on an Ensemble of RegressionTrees is employed in our approach. The DEST application uses Eigen library routines to implement algebraic operations which are proved to be quite slow. The achievements of this paper, concern the replacement of appropriate Eigen calls in time critical paths with fast C code that can be directly used to synthesize reconfigurable hardware implementations. The elimination of the computational intensive Eigen calls has already improved the speed of the face alignment application by more than 240 times. In this paper we examine how the modified source code structure of the DEST application can be used to address the challenges described above.

show abstract

Action unit intensity regression for facial MoCap aimed towards digital humans

Vilchis,

Mendez-Ruiz,

Perez-Guerrero

et al. 2024

Multimed Tools Appl

View full text Add to dashboard Cite

Due to the increasing demand for virtual avatars, there has been a recent growth in the research and development of frameworks for realistic digital humans, which create a demand for realistic and adaptable facial motion capture systems. Most frameworks belong to private companies or represent high investments, which is why the creation of democratized solutions is relevant for the growth of digital human content creation. This research work proposes a facial motion capture framework for digital humans with the use of machine learning for facial codification intensity regression. The main focus is to use coded face movement intensities to generate realistic expressions on a digital human. The ablation studies performed on the regression models show that Neural Networks, using Histogram of Oriented Gradients as features, and with person-specific normalization, present overall better performance against other methods in the literature. With an RMSE of 0.052, the proposed framework offers reliable results that can be rendered in the face of a MetaHuman.

show abstract

Efficient deformable 3D face model tracking with limited hardware resources

Cited by 8 publications

References 20 publications

A High Performance and Robust FPGA Implementation of a Driver State Monitoring Application

A High Performance and Robust FPGA Implementation of a Driver State Monitoring Application

Challenges Towards Hardware Acceleration of the Deformable Shape Tracking Application

Action unit intensity regression for facial MoCap aimed towards digital humans

Contact Info

Product

Resources

About