Although an automatic parking system has been installed in many vehicles recently, it is still hard for the system to confirm by itself whether a vacant parking area truly exists or not. In this paper, we introduced a robust vision-based vacancy parking area detecting method for both indoor and outdoor environments. The main contribution of this paper is given as follows. First, an automatic image stitching method is proposed. Secondly, the problem of environment illuminating change and line color difference is considered and solved. Thirdly, the proposed algorithm is insensitive to the shadow and scene diversity, which means the detecting result satisfies most of the environment. Finally, a vehicle model is considered for tracking and reconfirming the detecting results to eliminate most of the false positives.