With the rapid advancement of sensing and computing technology and the wide adoption of mobile computing, the construction industry has faced a rise in the amount of information and data generated during the lifecycle of the construction project. To deal with a large variety of project data and information to support efficient and effective decision making, visualization, information modeling, and simulation (VIMS) has become critical in the development of capital facilities and infrastructures. The objective of this paper is to identify and investigate grand challenges in VIMS for the construction industry, to assist the academic and industry communities in establishing a future research agenda to solve VIMS challenges. In particular, 17 VIMS grand challenges were identified by an expert task force in the VIMS committee of the ASCE Computing and Information Technology Division, and then VIMS experts in the civil and construction areas from both academia and industry participated in a survey to assess the identified challenges, examine the relative importance of the identified challenges, and investigate current practices and future directions of VIMS. The survey results indicate that several knowledge gaps regarding VIMS challenges between academia and industry still exist, and it is the contention of this research that these particular gaps need to be addressed in future research. These research directions apply to technical issues and sociological/cultural/organizational issues in VIMS challenges. The major contribution of this paper is its claim that the provision of shared views on VIMS challenges lays a firm foundation in which collaborative actions between academia and industry can take place, which will, in turn, advance VIMS for the construction industry. By incorporating the whole project lifecycle, not only project execution but also planning, operation, and maintenance, the analysis provides meaningful hints for VIMS challenges in architecture, engineering, and facility management industries.