Unmanned Aerial Vehicle (UAV) has gained popularity in visual building inspection due to its unique advantages of high mobility and low cost. However, it has been a challenge to efficiently manage the large number of images collected by the camera-equipped UAV for analysis and support building condition assessment. One potential solution is to manage the UAV data and building data through the platform of Building Information Modelling (BIM). However, little research has been found that successfully combined UAV and BIM for the visual inspection of buildings. This research explores the state-of-the-art UAV and BIM for visual building inspection through a systematic literature review. A mixed quantitative-qualitative analysis is conducted to provide insights into the application of UAV and BIM in visual building inspection based on identified academic publications (i.e., 48 articles on UAV, 31 articles on BIM, and 4 articles on integrating UAV and BIM). Furthermore, challenges and possible research opportunities are highlighted to guide future research: (1) integrating UAV and BIM to automate the visual building inspection process, including the data collection and data management; (2) considering the safety concern induced by the complex surrounding environment for the BIM-based UAV flight path planning; (3) developing an efficient way for managing UAV images in BIM. In addition, a conceptual framework for integrating UAV and BIM towards automated visual building inspection is proposed to serve as a roadmap.