Invasions by exotic plant species and their impacts on invaded communities are a highly topical field of research because it provides a basis for the management of neophyte populations. However, for many invasive neophyte species in Central Europe little is known about their impacts on invaded plant communities. Moreover, it has hardly been considered whether effects vary among habitat conditions. Here, we selected each ten sites with different habitat conditions invaded by Erigeron annuus, Fallopia japonica, Impatiens glandulifera and Solidago canadensis which were listed as invasive in Switzerland. At each site, we established four 4 m × 1 m subplots covering a gradient from very low to very high cover of the particular neophyte species to investigate the effect of increasing neophyte cover on the species richness, Shannon diversity and evenness of the invaded plant communities. Moreover, we measured soil pH and characterized habitat conditions using Ellenberg indicator values to light and soil fertility. Whereas increasing cover of I. glandulifera had no effect on the diversity of invaded plant communities, an increasing cover of E. annuus negatively affected Shannon diversity. An increasing cover of F. japonica combined with a decreasing soil pH negatively affected the Shannon diversity of invaded plant communities. Similarly, an increasing cover of S. canadensis in combination with decreasing soil fertility negatively affected the Shannon diversity and evenness of invaded communities. Our results indicate that significant effects of increasing neophyte cover are mostly coupled to particular habitat conditions and then rather suppress than eliminate native plant species in invaded communities. We therefore suggest including abiotic variables in further impact studies on biotic invasions. Furthermore, adapting measures to the respective environmental context can be a useful tool in priority setting for the management of invasive neophyte populations and the restoration of invaded habitats.