Construction sites experience low productivity due to particular characteristics such as unique designs in each project, sporadic arrival of projects, and complexity of the process. Another contributing factor to low productivity is poor communication among workers, supervisors, and a site’s centralized knowledge hub. Research shows that introducing advanced artificial intelligence (AI) technology in construction can tackle these problems. In this paper, we analyzed human factors considerations–user, task, environment, and technology and identified their characteristics and challenges to design an interactive AI system to facilitate communication between workers and other stakeholders. Based on the analysis, we propose a voice-based intelligent virtual agent (VIVA) as a multi-purpose AI system on construction sites with a further research agenda. We hope that this effort can guide the design of construction-specific AI systems and that this worker-AI teaming can improve overall work processes, enhance productivity, and promote safety in construction.