Understanding which factors affect the distribution pattern of extremely small populations is essential to the protection and propagation of rare and endangered plant species. In this study, we established 108 plots covering the entire Hainan Island, and measured the appearance frequency and species richness of plant species with extremely small populations, as well as the ecological environments and human disturbances during 2012–2013. We explored how the ecological environments and human activities affected the distribution pattern of these extremely small populations. Results showed that the extremely small populations underwent human disturbances and threats, and they were often found in fragmental habitats. The leading factors changing the appearance frequency of extremely small populations differed among plant species, and the direct factors making them susceptible to extinction were human disturbances. The peak richness of extremely small populations always occurred at the medium level across environmental gradients, and their species richness always decreased with increasing human disturbances. However, the appearance frequencies of three orchid species increased with the increasing human disturbances. Our study thus indicate that knowledge on how the external factors, such as the ecological environment, land use type, roads, human activity, etc., affect the distribution of the extremely small populations should be taken for the better protecting them in the future.