A Real-Time Speech Separation Method Based on Camera and Microphone Array Sensors Fusion Approach

Abstract

In the context of assisted human, identifying and enhancing non-stationary speech targets speech in various noise environments, such as a cocktail party, is an important issue for real-time speech separation. Previous studies mostly used microphone signal processing to perform target speech separation and analysis, such as feature recognition through a large amount of training data and supervised machine learning. The method was suitable for stationary noise suppression, but relatively limited for non-stationary noise and difficult to meet the real-time processing requirement. In this study, we propose a real-time speech separation method based on an approach that combines an optical camera and a microphone array. The method was divided into two stages. Stage 1 used computer vision technology with the camera to detect and identify interest targets and evaluate source angles and distance. Stage 2 used beamforming technology with microphone array to enhance and separate the target speech sound. The asynchronous update function was utilized to integrate the beamforming control and speech processing to reduce the effect of the processing delay. The experimental results show that the noise reduction in various stationary and non-stationary noise environments were 6.1 dB and 5.2 dB respectively. The response time of speech processing was less than 10ms, which meets the requirements of a real-time system. The proposed method has high potential to be applied in auxiliary listening systems or machine language processing like intelligent personal assistant.

Human-computer interaction based on speech recognition

“…Intelligent assistant, a smart device that integrates technologies such as speech recognition, natural language processing and machine learning, is gradually entering People's Daily life and bringing unprecedented convenience. They can chat with users, consult, query, but also self-learning and training, after several years of continuous development of artificial intelligence technology, the current intelligent voice recognition assistant can easily achieve voice interaction with users and communication reply, through speech and semantic recognition, voice recognition assistant constantly understand users, calculate user needs [16]. For example, Xiao Ai is able to answer questions, provide information and perform tasks through voice commands.…”

“…In the past 20 years, smart home only looks advanced and convenient, in fact, many simple problems are complicated, and now, the "help" of intelligent algorithms, the general model support of big data statistics, the personalized customization of blockchain and its high consensus, and the new interactive experience brought by pattern recognition, artificial intelligence has brought new breakthroughs for the development of smart home. To solve the "dilemma" of smart home [16]. Application of voice interaction in intelligent furniture was shown in the Figure 2.…”