Recently, extensive studies have been performed for crack detection and segmentation using deep learning and computer vision techniques to accomplish autonomous bridge inspection. These deep network models are frequently trained with a large volume of parameters to ensure good performance. However, the robust applications under real-world situations of actual bridge inspection still face significant challenges. For example, false-positive recognitions of complex background disturbances excluded in the training sets are inevitable to exist. Besides, the real-time requirement for deploying large-volume deep networks in edge computing equipment is still challenging to achieve. This study establishes a lightweight semantic segmentation model for complex concrete crack segmentation in actual bridge inspection. First, the DeepLabv3+ model is adopted as the baseline, and the backbone module is replaced by MobileNetV2 instead of ResNet101. Second, the depthwise separable convolution, atrous convolution pyramid, and inverted residual modules are utilized to reduce convolutional parameters, expand receptive fields, and alleviate gradient vanishing, respectively. Third, the dataset is enhanced with negative disturbance examples, including straight-line-like structural edges and exposed rebars, to improve the model performance against false positives without additional labeling workload. Original images with different resolutions are first collected from actual bridges, and negative samples are further added to the dataset. A total of 4303 patches in 512 × 512 are generated by a sliding window, where 3443, 430, and 430 are randomly selected for training, validation, and test. Ablation experiments demonstrate the necessity and effectiveness of using MobileNetV2 instead of ResNet101 as the backbone and adding negative examples into the dataset. The results show that the mean intersection-over-union (mIoU) for crack segmentation in various real-world scenarios reaches 0.759. The recognition rate of false positives for complex background disturbances is effectually suppressed by introducing straight-line-like structural edges and exposed rebars into the dataset. Furthermore, the average time cost gains a significant reduction of 35.1% using the established lightweight crack segmentation model with only a slight drop on IoU of 0.017.