Olive anthracnose outbreaks caused by the Colletotrichum species complex in the Mediterranean region decrease both fruit yield and olive oil production while also drastically degrading olive oil quality. The presence of various Colletotrichum species able to produce disease symptoms in olive fruits significantly deteriorates the efforts for an efficient crop protection strategy. In this report, the major olive productive area of Peloponnese was screened for Colletotrichum species capable of generating anthracnose symptoms. Olive fruits of 12 different olive cultivars were collected from 60 groves distributed analogously in the Peloponnese. Thirty-two fungal strains isolated from asymptomatic olive drupes were identified morphologically as Colletotrichum spp. and were multilocus genetically analyzed. The 32 isolates were grouped into two primary lineages resembling the previously characterized Colletotrichum acutatum and Colletotrichum nymphaeae based on the conducted genetic analysis for five genetic loci. The virulence of 16 Colletotrichum spp. strains were evaluated in a detached fruit assay of 10 Greek olive cultivars. The results clearly suggested that fungal isolates belonging to both C. acutatum and C. nymphaeae exhibited different levels of pathogenicity in a cultivar-dependent manner. Thus, cultivars examined in terms of the % Disease Index (%DI) were divided into highly tolerant, tolerant, and susceptible, and those analyzed regarding the % Disease Severity Index (%DSI) were divided into tolerant and susceptible. Our results suggest that the Greek cultivars of Athinolia and Megaritiki are highly tolerant to the vast majority of Colletotrichum strains isolated from Peloponnesian groves and consist of a significant genetic material for the future design of crop protection programs against anthracnose breakouts.