Despite the economic importance of the construction industry worldwide, it suffers from low productivity compared to other sectors, due to weak industrialisation, fragmented supply chain and poor collaboration. Recent national initiatives are promoting the adoption of information and automation technologies to increase efficiency, quality, safety, and reduce costs in construction. Emerging technologies will also provide highly integrated, connected and scalable new methods of construction. Nevertheless, the benefits and risks of automation in construction remain largely unknown due to the lack of standards and management tools to assess them from a holistic perspective. The aim of this research is to develop a Balanced Scorecard (BSC) as an evaluation framework for automation in construction. A BSC is a strategic management system that links performance measurement to business strategy using a holistic set of performance assessment criteria. BSCs expand evaluation beyond financial criteria to include environmental and social considerations. The proposed BSC under development in this research uses a hierarchic system of multidimensional indicators (e.g. resource consumption, GHG emissions, costs, productivity, etc.) relevant to automation in construction, at operational, organisational and societal levels. The validity, priority and accessibility of the indicators were explored via a workshop with 20 participants from the construction industry. The workshop outcomes provided a means to focus attention on relevant key performance indicators (KPIs) for decision-making regarding construction processes. Based on the outcomes of this study, the final BSC will help construction organisations to achieve their sustainability goals and address low productivity, because automation solutions can be seen through a holistic, and pragmatic lens, thus are more likely to be included in, and contribute to, construction operations in the future.