Invasive alien plants have invaded various habitats, posing a threat to biodiversity. Several hypotheses have been proposed to explain the mechanisms of invasion, but few studies have considered the characteristics of the invaded communities and the effects of human interference in the invasion. In this study, we compared the invasibility of three different habitats: abandoned land, eucalyptus plantations, and natural secondary forests. We explored the effects of species diversity, phylogenetic diversity, and disturbance factors on the invasibility of different habitats. The results showed that the invasibility of abandoned land was the highest and the invasibility of the natural secondary forest was the lowest. Phylogenetic indicators affected the invasibility of abandoned land and eucalyptus plantations, and disturbance factors affected the invasibility of all three habitats, while the characteristics of the invaded communities had a weak impact. Our research provided supporting evidence for Darwin’s naturalization hypothesis and his disturbance hypothesis but found no relationship between biotic resistance and invasibility. This study indicated that the differences among habitats should be considered when we prove Darwin’s naturalization hypothesis in nature reserves.