Numerous single image deraining algorithms have been recently proposed. However, these algorithms are mainly evaluated using certain type of synthetic images, assuming a specific rain model, plus a few real images. It is thus unclear how these algorithms would perform on rainy images acquired "in the wild" and how we could gauge the progress in the field. This paper aims to bridge this gap. We present a comprehensive study and evaluation of existing single image deraining algorithms, using a new largescale benchmark consisting of both synthetic and realworld rainy images of various rain types. This dataset highlights diverse rain models (rain streak, rain drop, rain and mist), as well as a rich variety of evaluation criteria (fulland no-reference objective, subjective, and task-specific) Our evaluation and analysis indicate the performance gap between synthetic rainy images and real-world images and allow us to better identify the strengths and limitations of each method as well as future research directions.