Aiming to solve the challenges of low detection accuracy, poor anti-interference ability, and slow detection speed in the traditional tunnel lining defect detection methods, a novel deep learning-based model, named You Only Look Once network v4 enhanced by EfficientNet and depthwise separable convolution (DSC; YOLOv4-ED), is proposed. In the YOLOv4-ED, EfficientNet is used as the backbone to improve the identification accuracy of indistinguishable defect targets in complex tunnel background and light conditions. Furthermore, DSC block is introduced to reduce the storage space of the model and thereby enhance the detection efficiency. The experimental results indicate that the mean average precision, F1 score, Model Size, and FPS of YOLOv4-ED are 81.84%, 81.99%, 49.3 MB, and 43.5 f/s, respectively, which is superior to the comparison models in both detection accuracy and efficiency. Based on robust and cost-effective YOLOv4-ED, a tunnel lining defect detection platform (TLDDP) with the capacity of automated inspection of various lining defects (i.e., water leakage, crack, rebar-exposed) is built. The established TLDDP can realize the high-precision and automatic detection of multiple tunnel lining defects under different lighting and complex background conditions of the practical in-service tunnel.