We developed a super-resolution (SR) benchmark to analyze SR capabilities to upscale compressed videos. The dataset for the benchmark was collected using video codecs of 5 different compression standards. We assessed 17 state-of-the-art SR models using our benchmark and evaluated their ability to preserve scene context and their robustness to compression artifacts. To get an accurate perceptual ranking of SR models, we conducted a crowd-sourced side-by-side comparison of SR results. We also analyzed the results of the benchmark and developed an objective quality assessment metric based on existing best-performing objective metrics. Our metric outperforms other video quality metrics by Spearman correlation with subjective scores for the task of upscaling compressed videos.