Coming with the proliferation of mobile devices, the issue of displaying videos with varying resolutions and aspect ratios has become emerging. To this end, state-of-the-art works typically resort to video retargeting or cropping, i.e. adaptively adjusting the display regions and contents within each video frame to fit for the mobile device screens. However, most existing approaches retain on evaluating the objective visual statistics to select salient parts to be cropped within each frame, with time-consuming constraints to ensure the temporal consistency, which is indeed unsuitable to cope with the massive videos uploaded daily from user-contributed platforms. In this paper, we propose an efficient yet robust video cropping algorithm which merits in two-fold: First, a perceptual Region of Interest selection approach is proposed by employing a novel Interesting Region Score, which is a linear combination of Rate of Focused Attention, Total Saliency Score, and Bias from Center Penalty. Second, we adopt fast curve fitting to seek for an optimal cropping sequence over a set of consecutive video frames within a shot, which avoids the complex optimization with temporal constraints among adjacent frames. Experiments on video sequences from Backkom cartoon and Iron Man II with comparisons against several alternative approaches demonstrate the efficiency and robustness of the proposed algorithm.