The advancement of cellular communication technology has profoundly transformed human life. People can now watch high-definition videos anytime, anywhere, and aim for the implementation of advanced autonomous driving capabilities. However, the sustainability of such an environment is threatened by false base stations. False base stations execute attacks in the Radio Access Network (RAN) of cellular systems, adversely affecting the network or its users. To address this challenge, we propose a behavior rule specification-based false base station detection system, SMDFbs. We derive behavior rules from the normal operations of base stations and convert these rules into a state machine. Based on this state machine, we detect network anomalies and mitigate threats. We conducted experiments detecting false base stations in a 5G RAN simulator, comparing our system with seven machine learning-based detection techniques. The experimental results showed that our proposed system achieved a detection accuracy of 98% and demonstrated lower overhead compared to other algorithms.