Summary Computer vision‐based crack detection is an effective technique for evaluating the structural safety of concrete building structures. Currently, the existing crack‐detection methods based on machine learning often require pre‐training or/and re‐training the model, which is an experiential and complex task. In this study, based on sparse representation, we cast the crack damage detection problem by determining the outlier in the sparse correlation coefficients between the selected crack and the testing image regions. Specifically, by dividing one concrete image to be detected, we can obtain multiple testing image regions. Then, the spatial variation features of these image region contents are computed via discrete cosine transformation and are further used as the dictionary set. Considering that only a fraction of the dictionary set belongs to the cracks, the correlation coefficients of the selected known crack regions in the dictionary set should be sparse. Furthermore, a fast iterative shrinkage‐thresholding algorithm (FISTA) was utilized to obtain the optimum sparse correlation coefficients. Finally, for the dictionary set, the atoms (i.e., regions) that have larger values in the sparse correlation coefficients are treated as outliers (i.e., cracks), and the 3δ principle is exploited to identify these outliers. Experiments on a practical concrete image set show that the proposed algorithm is more accurate and efficient than traditional crack‐detection methods.