This paper proposes a novel Shift with Importance Sampling (SIS) scheme to improve the efficiency in pedestrian detection but maintain its high accuracy. For fast and efficient object detection, the cascade-Adaboost structure is the commonly-used approach in the literature. However, its detection performance is quite lower due to non-robust features and a fully-scanning on image especially when deformable part models are adopted. Firstly, various SURF points are first detected and then clustered via the K-Means scheme to produce potential candidates. Each pedestrian candidate is verified by a SVM-classifier based on HOG features. However, each SURP point will not exactly locate in the center of each detected pedestrian and lead to the failure of detection. To speed up the detection efficiency, we propose a novel Shift with Importance Sampling technique (SIS) to quickly shift into the correct location of each pedestrian with minimum tries and tests. The time complexity is reduced from 2 ( ) O n to (log ) O n . After that, the particle filter is adopted to track targets if they are missed. Experimental results show the superiority of our SIS method in pedestrian detection.