An address, a textual description of a physical location, plays an important role in location-based services such as on-demand delivery and e-commerce. However, abnormal addresses (i.e., an address without detailed information representing a spatial location) have led to significant costs. In real-world settings like e-commerce, abnormal address detection is not trivial because it needs to be completed in real-time to support massive online queries. In this study, we design FastAddr, a fast abnormal address detection framework, which detects abnormal addresses among millions of addresses in a short time. By investigating and modeling the hierarchical structure of address data, we first design a novel contrastive address augmentation approach to generate training data via learning the entity transition probability matrix. We further design a lightweight multi-head attention model for learning compact address representation by modeling the address characteristics. We conduct a comprehensive three-phase evaluation. (i) We evaluate FastAddr on a real-world dataset and it yields the average F1 of 85.7% in 0.058 milliseconds, which outperforms the state-of-the-art models by 47.4% with similar detection time. (ii) An offline A/B test shows that FastAddr outperforms the previous deployed model significantly. (iii) We also conduct an online A/B test to compare FastAddr with the deployed model, which shows an improvement of F1 by more than 20%. Moreover, a real-world case study demonstrates both the efficiency and effectiveness of FastAddr.