Microbial fermentation is crucial to the development of flavors in cacao-based products, but how the abiotic environment and pod surface microbiomes can contribute to these different sensory profiles is underdeveloped. To better understand the connections between environment, plant variety, their associated pod phyllosphere microbiomes and the fermentation microbiome, we characterized the fungi and bacteria on pod surfaces and early fermentations of two cacao varieties across wet and dry sites on the island of O‘ahu, Hawai‘i. We showed that site (and the environmental conditions linked to those sites) significantly influenced the microbiomes of both the pod phyllosphere and fermentations, but did not significantly explain the differences between cacao varieties. Certain taxa of bacteria and fungi seemingly prefer the wet Kualoa Ranch site compared to the drier Urban Garden Center Site. Rare members of the pod phyllosphere communities became highly enriched during fermentation, where basidiomycetous yeasts (Filobasidium sp., Hannella zeae, Rhodotorula mucilaginosa, R. paludigena, and Sporobolomyces carnicolor) became dominant at the wet site, and both ascomycetous and basidiomycetous yeasts (Hyphopichia burtonii, Meyerozyma sp., Yamadazyma phyllophila, Filobasidium sp., Kwoniella heveanensis, R. mucilaginosa, Sporobolomyces sp., and S. carnicolor) became dominant at the dry site. The agricultural production site where cacao plants are grown should be considered carefully since it is likely important in structuring microbes that are present on pod surfaces and the microbes (mostly yeasts) that will be carried forwards into fermentation. These yeasts have potentials to contribute to the development of the unique fruity or floral flavors in chocolates produced in Hawai‘i.