Education sector in Malaysia had put emphasis on the use of online learning or e-learning with technology and devices as a mediator of communication to replace face-to-face learning during the COVID-19 pandemic. Subsequently, with the improvement of learning technologies, English as a Second Language (ESL) teachers faced various challenges in language teaching. In this regard, this study aimed to investigate the challenges ESL teachers faced in implementing e-learning during COVID-19. Using a qualitative approach, this research was a case study which involved 20 primary ESL teachers. Data collection was done through an in-depth interview to explore teachers’ e-learning experiences and challenges in teaching the English language. Based on the findings, the majority of teachers expressed that the use of e-learning was effective with various limitations such as teachers’ readiness to adopt e-learning, accessibility to mobile phones and Internet connectivity, classroom management in term of low students’ participation and assessment. Thus, the researcher recommended more studies to highlight teachers’ insight regarding the significance of e-learning. ESL teachers, stakeholders, policy makers, and institutions can benefit from the results of the study and come out with practical strategies to utilize online sources for education in the pandemic situation.