In this paper, we propose a vision-based target detection scheme based on video saliency to find the rescue targets automatically. A new definition of video saliency is first proposed to comprehend the target properties based on structure similarity among adjacent video frames. Then, a cascaded target detection is devised with the local image feature and regional video saliency. We treat the salient objects in aerial images as a basic feature and filter the false candidates based on the region-based video saliency. The propose method can improve the search and rescue of targets, and reduce the economical losses in marine accidents.