Along with the growing practice of teaching English for Specific Purposes (ESP) in non-English departments of tertiary education, it is essential to investigate the challenges faced by ESP teachers. It can be a basis for proposing policies for the improvement of ESP practice. This study was driven by the fact that ESP classes in non-English departments are allocated limited credit hours, and the teachers are generally General English teachers with no experience and training in teaching ESP. Thus, this study attempted to investigate the fundamental challenges faced by ESP teachers in one state and four private higher education institutions. The data of this qualitative study were obtained through interviews with five ESP teachers. The interview questions were mainly concerned with the knowledgeability and competence in teaching related to subject-specific contexts, adequacy of ESP training, needs analysis, and classroom condition. The findings reveal that the evident challenges encountered by ESP teachers were: lacking knowledge on students’ field of study, lacking of ESP training, lacking of proper needs analysis, large classes, and various learners’ English competencies. The findings of this study suggest that policymakers (stakeholders) should pay more attention to the practice of ESP teaching, especially in non-English departments, by reforming policy in order to minimize the problems faced by ESP teachers and to improve the practice of ESP teaching.