Avermectins are pharmaceutical drugs widely used mainly in livestock to combat both ectoparasites and endoparasites. Drugs belonging to this family include ivermectin, abamectin, doramectin, selamectin, eprinomectin, and emamectin benzoate, and they share similar chemical characteristics. When administered to livestock, between 80 and 98% of the drug is estimated to leave the body without being metabolized in feces, thus reaching the soil. For this reason, concern for avermectin contamination in soil is increasing, and researchers are focused on estimating the effects on non-target organisms, such as plants and soil invertebrates. This review aimed to compile and discuss updated data of avermectin toxicity on non-target organisms to better comprehend its effect on the environment. Effects on plants are scarcely studied, since they were not believed to absorb these drugs. However, recent studies suggest that plants can be negatively affected. Regarding soil invertebrates, negative effects such as increased mortality and reduced reproduction are best known to dung-beetles. Recently, some studies have also suggested that earthworms, springtails, and enchytraeids can be adversely affected by avermectin exposure. Since ivermectin was the first avermectin marketed, most of the data refers to this product. According to new data on scientific literature, avermectins can now be considered harmful to non-target organisms, and its prudent use is recommended in order to reduce negative effects on the environment. For future investigations, inclusion of avermectins other than ivermectin, as well as field and “omics” studies is suggested.