The Internet of Things establishes the intimacy between the Internet and the physical world. Due to portable size, most IoT devices have limited computing and storage capabilities and are vulnerable to various malicious intrusions. Therefore, it is vital to have efficient approaches to distinguish the true IoT data from fake one, we term such methods as anomaly detection (AD). To detect anomalies accurately and efficiently, in this paper a 3-hierarchy joint local and global anomaly detection framework, HADIoT, is proposed, in which IoT devices generate and transmit sensory data to their local edge servers for local AD after data refinement which includes re-framing, normalization, complexity reduction via Principal Component Analysis, and symbol mapping. High detection accuracy is achieved by jointly local and global ADs. The local AD focuses on the data pattern consistency of individual devices via the Gated Recurrent Unit, and the processed data is then forwarded from edge servers to the cloud server for global AD. The global AD focuses on the analysis of the data pattern correlations between different IoT devices, using the Conditional Random Fields. For the maintenance of cyber-security, the proposed anomaly detection framework HADIoT enables to provide an accurate and faster anomaly detection for IoT applications, compared with existing anomaly detection methods. The performance of the proposed method is also empirically evaluated through simulations, using a real dataset-the Information Security Center of Excellence (ISCX) 2012 dataset. Simulation results demonstrate the effectiveness of the proposed framework in terms of True Positive Rate, False Positive Rate, Precision, Accuracy and F_score, compared with three benchmark schemes.