Learning Japanese can enhance competitiveness in a globalized economy, and we address the problems of poor open-source Japanese language teaching, cumbersome teaching tasks, and a single teaching model. We propose a hybrid Japanese teaching aid system with multiple information fusion mapping, which can effectively improve the efficiency of Japanese teaching and reduce the tedious human teaching procedures. The system is divided into two branches of Japanese language recognition, namely, the Japanese text recognition branch and the Japanese voice sequence recognition branch. In the Japanese text recognition branch, we integrate attention mechanisms and long short-term memory networks as the basic network for Japanese character text recognition. In addition, we set up separate text feature recognition systems for Japanese computer writing and handwriting to prevent feature overlap problems. For Japanese voice sequence recognition, we used a combination of memory gating unit and encoder, based on the network still extending the structure of the deep neural network and using the residual structure connection in the gating unit to avoid the gradient disappearance problem. At the end of the system, we use a softmax layer to connect the text recognition and voice recognition networks to form a Japanese language teaching aid system. To verify the efficiency of our system, we selected the Japanese text recognition public dataset and voice recognition public dataset for experimental validation. To match the practical application of the system, we created our dataset based on the dataset standard and conducted experimental validation. To compare other Japanese recognition methods, we selected the six most representative Japanese recognition algorithms for experimental comparison. To ensure the balance of the experiments, each algorithm is trained in a separate experimental environment for modeling and tuning parameters. Experimental performance and the experimental results show that our method significantly outperforms the other methods and has better system stability.