Automated pipeline safety early warning (PSEW) systems are designed to automatically identify and locate third-party damage events on oil and gas pipelines. They are intended to replace traditional, inefficient manual inspection methods. However, current PSEW methods cannot achieve universality for various complex environments because they are sensitive to the spatiotemporal stability of the signal obtained by its distributed sensors at various locations and times. Our research aimed to improve the accuracy of long-distance oil–gas PSEW systems through machine learning. In this paper, we propose a novel real-time action recognition method for long-distance PSEW systems based on a coherent Rayleigh scattering distributed optical fiber sensor. More specifically, we put forward two complementary feature calculation methods to describe signals and build a new action recognition deep learning network based on those features. Encouraging empirical results on the data collected at a real location confirm that the features can effectively describe signals in an environment with strong noise and weak signals, and the entire approach can identify and locate third-party damage events quickly under various hardware conditions with accuracies of 99.26% (500 Hz) and 97.20% (100 Hz). More generically, our method can be applied to other fields as well.