WLAN-based indoor positioning algorithm has the characteristics of simple layout and low price, and it has gradually become a hotspot in both academia and industry. However, due to the poor stability of Wi-Fi signals, received signal strength (RSS) fingerprints of some adjacent reference positions are difficult to evaluate similarity when utilizing traditional distance-based calculation methods. By clustering these RSS fingerprints into one region, the commonly utilized KNN algorithm in the past cannot achieve accurate positioning in the region. For this, we introduce a concept of the insensitive region of the RSS fingerprint and a new algorithm named DBSCAN-KRF. This algorithm can delete noise sample and detect insensitive region, then, different methods are selected to achieve indoor positioning by judging the region of the estimated fingerprint sample, the KNN algorithm is selected when the region is sensitive, and random forest algorithm is selected when the region is insensitive. The experimental results show that the DBSCAN-KRF algorithm is superior while compared with other alternative indoor positioning algorithms. INDEX TERMS WLAN indoor position, control and optimization, machine learning, DBSCAN-KRF algorithm, fingerprint data.