Material Identification System with Sound Simulation Assisted Method in VR/AR Scenarios

Abstract

We propose a new model using transfer learning method in motivation of specifying and identifying material of an object by knocking in VR/AR applications. Different from the traditional contact knocking material/object identification method, we apply the sound simulation method to enlarge the training dataset containing various models and materials in real-world scenarios. Our approach is based on Domain-Adversarial Training of Neural Networks that learns from the pre-collected simulated and corresponding real knocking sound to extract their common features determined by different materials. Given the scanned 3D model and the real knocking sound from users, we present an incremental learning model using the features extracted by pre-trained transfer learning model to generate the final material classifier. We perform an overall evaluation showing that our system achieves around 93.3% accuracy of identifying materials, which is much higher than the accuracy mentioned in previous work. CCS CONCEPTS• Human-centered computing → Interaction design.