This paper presents our effort in developing an agile robotic system for general in situ construction work of large-scale structures. The robotic system comprises of a lightweight industrial robot mounted on a specially designed robot platform capable of being quickly integrated into available scaffold structures. The current prototype weighs less than 50 kg, making it portable and easily carried around by two persons. Furthermore, by making use of the scaffold structures commonly found around the construction site as its foundation, the robotic system requires very little additional setup and would be capable of entering work areas in the construction site which were previously only accessible by human. The key design components which make up the agile robotic system are discussed in detail in this paper. Optimization of some design variables was also performed to reduce the anticipated deflection of the supporting scaffold pipes based on a derived analytical beam flexion model. Finally, actual experiments were carried out to verify the theoretical model and the viability of the proposed agile robotic system.