The seismic vibration measurement of reinforced concrete (RC) structures is an important aspect of post‐disaster reconstruction. Unfortunately, the existing method of installing sensors on walls and columns is prohibitively expensive, and the measurement signals are prone to loss during seismic vibration. In this paper, a marker‐free measurement method based on the Lucas‐Kanade (L‐K) optical flow algorithm for vibration measurements of RC structures under seismic vibration has been developed. In addition, to improve the accuracy of L‐K optical flow in dealing with the RC structure under seismic vibration, a new approach for selecting the feature point that is used to solve the restrictions of manual marking, as well as an adaptive matched window strategy that uses different sizes of matched window along the motion to replace the fixed matched window in the original L‐K algorithm, are proposed. The precision of the proposed method is first validated by comparing it to the original L‐K optical flow algorithm. Then, compare its precision and efficiency to the existing vision‐based methods, including feature matching (FM) and digital image correlation (DIC). Finally, the behaviors of RC structures during seismic vibration, such as inter‐story drift, frequency, and strain, are investigated. The results show that, when compared to the original L‐K optical flow algorithm and other vision‐based methods, the proposed method can accurately and efficiently measure motion without spraying speckle or high contrast markers on the structure's surface, and it enables the rapid diagnosis of structures affected by earthquakes.