Due to the rapid development of communication and sensing technology, a large amount of mobile data is collected so that we can infer the complex movement laws of humans. For cities, some unusual events may endanger public safety. If the early warning of an abnormal event can be issued, it is of great application value to urban construction services. To detect urban anomalies, this paper proposes the Hierarchical Urban Anomaly Detection (HUAD) framework. The first step in this framework is to build rough anomaly characteristics that need to be calculated by some traffic flow consisted of subway and taxi data. In the second step, the alternative abnormal regions were obtained. Then, the long short-term memory (LSTM) network is used to predict the traffic to get the historical anomaly scores. Following that, the refined anomaly characteristics are generated from adjacent regions, adjacent periods and historical anomalies. The final abnormal regions were detected by One-Class Support Vector Machine (OC-SVM). At last, based on real data sets, we analyzes the traffic flow of the target region and adjacent regions from multiple perspectives in view of the large crowd gathering activities, and the effectiveness of the method is verified. INDEX TERMS Spatio-temporal data fusion, traffic flow prediction, urban anomaly detection. I. INTRODUCTION