The progress of global economic integration has forced English learners to have an urgent need to improve their oral English. College students’ oral English ability is currently the worst of the four abilities of listening, speaking, reading, and writing. The main reasons are internal and external. The internal reason is that the pronunciation characteristics of Chinese students are different from those of English. The external cause is that the practice environment and tools of oral English are not ideal, which affects the improvement of learners’ oral English. This study proposes using a deep learning algorithm (DLA) English in the evaluation of oral English quality to improve learners’ oral English level. The quality of oral English can be comprehensively evaluated in terms of pitch, speed of sound, and rhythm. The standard of pronunciation is the foundation of oral English and is the most critical factor. In many DLAs, the input unit of DNN at a certain moment and its upper and lower moment input units have no relationship and are independent of each other, and the timing dependencies of adjacent units are not fully considered. The results are generally not very good on speech recognition tasks. This study proposes a time-delay neural network (TDNN) and a long short-term memory (LSTM) network to calculate the posterior probability of the model state to model context-dependent features in order to solve this problem. The fusion model TDNN-LSTM is applied in the English spoken pronunciation recognition task. To compare the accuracy of oral English pronunciation, several classic DLAs are introduced. The experimental results show that the method described in this study has a number of advantages. Although the performance improvement of this method in terms of recognition accuracy is not large, a certain degree of improvement is also very important for the oral English teaching assistant system.