Gaining a comprehensive understanding of how weed communities respond to both environmental and human-induced factors is of paramount importance in developing effective and ecologically sound weed control strategies. The objectives of the current research were to (1) assess the effect of the main weed management practices used in Greek olive groves on weed species’ diversity; (2) explore the filtering effect of management, site, and soil variables in determining weed species’ composition; and (3) shed light on the association between weed species’ composition and the diversity of the understory vegetation of olive groves. To accomplish these objectives, winter weed species’ coverage was assessed in 116 olive groves, both conventional and organic, distributed across three provinces in southern Greece. The investigation encompassed 29 explanatory variables, categorized into three groups: soil (22), management practices (6), and site conditions (1). It was confirmed that glyphosate use may lower biodiversity and species richness; however, this trend was not universal. In fact, the negative influence of the presence of Oxalis pes-caprae L. on species richness and diversity far outweighed the effect of spraying glyphosate. Redundancy analysis (RDA) revealed that among the 29 variables used to describe the ecological niche, eight (i.e., Mn, Mg, chemical spraying, mowing, rotary tiller, grazing, irrigation, and elevation) were significant and explained 21.5% of the total variation in weed species’ data. Interestingly, the soil Mn concentration was identified as the most influential one, highlighting the importance of soil micronutrients in determining weed species’ composition. The variation partitioning procedure demonstrated that the effect of the management variables on weed species’ composition accounted for 2.2 times the variance of soil variables and 4.5 times the variance of elevation. The present findings might help to enhance optimal management in olive groves that can sustain the biodiversity of flora and, in turn, provide various ecosystem services to agro-ecosystems.