In the field of video anomaly detection, effective representation for the normal and abnormal video features is a key issue for improving the accuracy of detection models. The feature representation based on global spatio-temporal networks lacks to consider negative impacts for the features between normal segment and abnormal segment, resulting in incorrect detection for some segments in video. To address this issue, a new approach of normal-abnormal negative impacts suppressing via normal feature memory for video anomaly detection is proposed. In this method, feature representation with normal–abnormal negative impacts suppression via normal feature memory pool is modeled, which can be used to suppress normal-abnormal negative impacts in video feature learning. Constraint learning for normal feature memory pool is designed to promote more effective learning for the normal feature memory pool and feature of abnormal video segment. Extensive experiments demonstrate that the proposed method outperforms the relative state-of-the-art methods on the Shanghai-Tech and XD-Violence datasets and achieves competitive results on the UCF-Crime dataset. This indicates that the proposed method can effectively enhances the performance of anomaly detection, thereby validating its effectiveness in practical applications.