Mid-air hand interaction with optical tracking for 3D modelling

Abstract

Compared to common 2D interaction done with mouse and other 2D-tracking devices, 3D hand tracking with low-cost optical cameras can provide more degrees of freedom, as well as natural gestures when shape modeling and assembling are done in virtual spaces. However, though quite precise, the optical tracking devices cannot avoid problems intrinsic to hand interaction, such as hand tremor and jump release, and they also introduce an additional problem of occlusion. This thesis investigates whether interaction during 3D modeling can be improved by using optical sensors so that 3D tasks can be performed in a way similar to interaction in real life and as efficient as when using common 2D-tracking based interaction while still minimizing the intrinsic problems of precise hand manipulations and optical problems. After surveying the relevant works and analyzing technical capabilities of the commonly available optical sensors, two approaches are thoroughly investigated for the natural mid-air hand interaction in precise 3D modeling-they are collision-based and gesture-based interaction. For collision-based methods, a set of virtual interaction techniques is proposed to realistically simulate real-life manipulation and deformation with one and two hands. For gesture-based interaction, a core set of interaction techniques is also devised which allows natural real-life interaction ways to be used. In addition, algorithms are proposed for both collision-based and gesture-based interaction to enhance the precision while minimizing the problems of hand tremor and jump release. However, the results show that virtual interaction designed with collision-based methods is always slower than real-First of all, I would like to thank my supervisor Associate Professor Alexei Sourin for his invaluable instructions and guidance. I am grateful to him for his help throughout my time as a research student at Nanyang Technological University. I also thank Professor Arjan Kuijper for giving me valuable advice and guiding my research at TU Darmstadt. It is credit to their efforts and ideas that my research has reached this point. I also wish to express my gratitude to the School of Computer Science and Engineering and Nanyang Technological University for providing me the opportunity and financial support to further my study and research. Special thanks go to the Joint PhD Degree Programme NTU-TU Darmstadt and Fraunhofer Singapore for supporting my project. I wish to thank my parents for their support. I would like to thank my wife Jin Luying for her selfless concern and encouragement. Praises and thanks to the God, the Almighty, for His showers of blessings throughout my research work to complete the research successfully.