Off-site construction has been increasingly employed due to its advantages, for instance, improved quality control, reduced skills labour, faster construction time, decreased material wastage and safe working environment. As the most cutting-edge off-site construction, modular buildings have been utilised for residential building, student accommodation, and hotel projects. However, because of existing and underlying constraints, the adoption of modular buildings is still relatively low. To reveal factors hindering the development of high-rise modular buildings, a comprehensive literature review, coupled with a focus group study, were conducted. A questionnaire survey inquiring about all stakeholders was implemented to quantify constraints. The results were further examined according to a real-life case study. This paper manifested that “Lack of coordination and communication among stakeholders”, “Higher cost”, “Lack of government support”, “Lack of experience and expertise”, “Lack of building codes and standards”, “Poor supply chain integration”, and “Complexity of connection” are the top barriers curbing the uptake of modular buildings. The findings should provide a valuable reference for stakeholders adopting modular buildings, whilst mitigating risks amid modular construction. Future research is expected to exploit building information modelling and design for manufacture and assembly to alleviate these existing constraints and promote the performance of modular construction as well.