The more information obtained about the driving environment, the more ensures driving safety. Due to the complex driving environment of farmland roads, targets beside the road sometimes have an important impact on driving safety. To achieve this goal, a novel real-time detection and prediction algorithm of targets was proposed. The whole image was divided into four parts by RCM: driving region, crossroad region, roadside region, and the other region. In addition, a safety policy for every part was enforced by the algorithm, which was based mainly on the combination of the YOLACT and GPM. On this basis, a self-collected data set of 5000 test samples is used for testing. The detection accuracy of the algorithm in the data set could reach up to 90%, and the processing speed to 30.4 fps. In addition, experiments were carried out on actual farmland roads, and the results showed that the proposed algorithm was able to detect, track, and predict targets on the farmland road, and alarm to driver in time before the targets rush into the road. This study provides an important reference for the safe driving of agricultural vehicles.