Explicit representation of uncertainties is essential to improve the reliability of seismic assessments of earthquake-damaged buildings, particularly when dealing with unreinforced masonry buildings. Modern inspection techniques use images for detecting and quantifying the damage to a structure. Based on the principle of falsification, this paper evaluates how the use of information of damage that is obtained from images taken on earthquake-damaged buildings reduces the uncertainty when predicting the seismic response under a future earthquake. New model falsification criteria use information on the residual state of a building, such as shear cracks, residual roof displacements, and observation of out-of-plane failure. To demonstrate the effectiveness of these criteria in reducing the uncertainty in response predictions, results from a four-story unreinforced masonry building stiffened with reinforced concrete walls, which was experimentally tested under a sequence of ground motions, are assessed. Three commonly used modeling approaches (single degree of freedom (DOF) systems, multi DOF systems with four DOFs, and equivalent frame models) are used, where uncertainties in model parameters and model bias are included and propagated through the analysis. Out of the models used, and in the absence of any additional source of information, the proposed falsification criteria are most effective in connection with the equivalent frame model because this model can simulate the response at the element-level, while the simpler models can only represent the global response or the response at the storey-level. The results show that when using only the information on the presence of shear cracks, which might be the first and only source of information after an earthquake, the effectiveness of model falsification is increased, thus reducing the uncertainty in model parameter values and seismic response predictions through the use of image-based inspection.