Estimating 3D Finger Angle via Fingerprint Image

Abstract

Touchscreens are the primary input devices for smartphones and tablets. Although widely used, the output of touchscreen controllers is still limited to the two-dimensional position of the contacting finger. Finger angle (or orientation) estimation from touchscreen images has been studied for enriching touch input. However, only pitch and yaw are usually estimated and estimation error is large. One main reason is that touchscreens provide very limited information of finger. With the development of under-screen fingerprint sensing technology, fingerprint images, which contain more information of finger compared with touchscreen images, can be captured when a finger touches the screen. In this paper, we constructed a dataset with fingerprint images and the corresponding ground truth values of finger angle. We contribute with a network architecture and training strategy that harness the strong dependencies among finger angle, finger region, finger type, and fingerprint ridge orientation to produce a top-performing model for finger angle estimation. The experimental results demonstrate the superiority of our method over previous state-of-the-art methods. The mean absolute errors of the three angles are 6.6 degrees for yaw, 7.1 degrees for pitch, and 9.1 degrees for roll, markedly smaller than previously reported errors. Extensive experiments were conducted to examine important factors including image resolution, image size, and finger type. Evaluations on a set of under-screen fingerprints were also performed to explore feasibility in real-world applications. Code and a subset of the data are publicly available.

OneTip Is Enough: A Non-Rigid Input Device for Single-Fingertip Human-Computer Interaction With 6-DOF

“…More recent research has explored the feasibility of estimating fingertip pose from fingerprints. A research team from Tsinghua University developed two 3D angle estimation methods based on deep neural networks [11] and fingerprint matching [27], respectively, and have achieved SOTA performance. These works successfully accurately estimated fingertip 3D angles, where the rolling angles were quantitatively measured for the first time.…”

“…These works successfully accurately estimated fingertip 3D angles, where the rolling angles were quantitatively measured for the first time. However, such methods rely on large data collection to train models with generalization, otherwise they cannot cover all possible combinations of 3D angles [11]. Even if non-learning frameworks are used, a pre-registration library needs to be constructed for fingerprint matching [27].…”

“…However, the output of a touch screen is OneTip Is Enough: A Non-Rigid Input Device for Single-Fingertip Human-Computer Interaction With 6-DOF Mingxuan Li, Yen Hang Zhou, Lunwei Zhang, Tiemin Li, and Yao Jiang T 2 usually a change in the position of a 2-D touch point. Related research has shown that supplementing additional degrees of freedom, such as finger angles, can enhance the interactive experience [11]. It requires researchers to continuously explore various ways to achieve the full 6-DOF input, which is naturally provided by fingertip pose changes.…”

“…Since the output range of the roll and pitch axes is much larger, the above results explain that the accuracy of OneTip's angle estimation for the yaw axis (i.e., normal-to-normal) is lower than that of the other two, which is consistent with the qualitative results.Due to the difference in sensing principles, device designs, and experimental methods, objectively comparing with existing fingertip interaction devices is impossible. From a subjective point of view, compared to state-of-the-art methods using rigid touchscreens (e.g., fingerprint-based 3-D finger angle estimation[11]), OneTip improves the accuracy by sacrificing the effective interaction space (i.e., the range of variations in fingertip pose). Thus, OneTip is more suitable for manipulating virtual objects (like a joystick) rather than navigating and traveling in a virtual environment (like a mouse).…”