English as a Foreign Language (EFL) students perform when speaking in public. An increasingly globalized world, effective public speaking is critical, but EFL students struggle to perform it, despite importance of qualities such as eye contact, speech pauses, there is presently no objective examination of such elements. A summative assessment has historically been the predominant form of evaluation in college English speaking assessments. Exam-centric teaching has considerable negative effect on foreign language training. In this research work, English Speaking Assessment Algorithm Based on Deep Learning (ESA-NEGCN-NBOA) is proposed. Initially, input video data are gathered from the multiple video dataset (MVD).The input video data is then pre-processed using Deep Attentional Guided Image Filtering (DAGIF) to remove presence of signal-dependent noise and improve lack of pixels from the regions and enhanced the video data. The data that has been pre-processed is utilized to Feature extraction using New General Double Integral Transform (NGDIT), which extract the significant attributes such as mel-frequency cepstral coefficients, energy, speech rate and pitch. Then NEGCN is proposed to improve students spoken English performance by assessmenting the English speakers. In general, NEGCN doesn’t express some adaption of optimization approaches for determining optimal parameters to promise exact improvement of assessment. Therefore, NBOA is proposed to enhance weight parameter of NEGCN for English speaking assessments, which precisely assess the English speaking. Performance measures such as accuracy, assessment error, evaluation time, pretest and posttest are examined when the proposed ESA-NEGCN-NBOA method is put into practice. The proposed ESA-NEGCN-NBOA method attains21.36%, 23.42% and 19.29% higher accuracy, 23.36%, 18.42% and 28.27% lower evaluation error, 20.36%, 27.42%, 28.17% lesser evaluation time analysed with existing techniques, likes innovative strategy towards oral English assessment utilizing machine learning, data mining, blockchain methods(IST-OEA-ML), machine learning assessment system for spoken English depend on linear predictive coding (AS-SE-LPC-ML), multimodal transfer learning for oral presentation assessment (MM-TL-OPA) respectively.