Vision-based systems are promising tools for displacement measurement in civil structures, possessing advantages over traditional displacement sensors in instrumentation cost, installation efforts and measurement capacity in terms of frequency range and spatial resolution. Approximately one hundred papers to date have appeared on this subject, investigating topics like system development and improvement, the viability on field applications and the potential for structural condition assessment. The main contribution of this paper is to present a literature review of vision-based displacement measurement, from the perspectives of methodologies and applications. Video-processing procedures in this paper are summarised as a three-component framework: camera calibration, target tracking and structural displacement calculation. Methods for each component are presented in principle, with discussions about the relative advantages and limitations. Applications in the two most active fields, bridge deformation and cable vibration measurement, are examined followed by a summary of field challenges observed in monitoring tests. Important gaps requiring further investigation are presented, e.g. robust tracking methods, non-contact sensing and measurement accuracy evaluation in field conditions.