Litsea cubeba (Lour.) Pers. is an important woody spice tree in southern China, and its fruit is a rich source of valuable essential oil. We surveyed and sampled L. cubeba germplasm resources from 36 provenances in nine Chinese provinces, and detected rich phenotypic diversity. The survey results showed that plants of SC-KJ, SC-HJ, and SC-LS provenance presented higher leaf area (LA); YN-SM and YN-XC plants had larger thousand-grain fresh weight (TFW); and HN-DX plants had the highest essential oil content (EOC). To explain the large differences in the phenotypes of L. cubeba among different habitats, we used Pearson’s correlation analysis, multiple stepwise regression path analysis, and redundancy analysis to evaluate the phenotypic diversity of L. cubeba. It was found that compared to other traits, leaf and fruit traits had more significant geographical distributions, and that leaf phenotypes were correlated to fruit phenotypes. The results showed that elevation, latitude, longitude, total soil porosity (SP), soil bulk density (SBD), and average annual rainfall (AAR, mm) contributed significantly to the phenotypic diversity of L. cubeba. Geographical factors explained a higher percentage of variation in phenotypic diversity than did soil factors and climate factors. Plants of SC-KJ and HN-DX provenances could be important resources for domestication and breeding to develop new high-yielding varieties of this woody aromatic plant. This study describes significant phenotypic differences in L. cubeba related to adaptation to different environments, and provides a theoretical basis for the development of a breeding strategy and for optimizing L. cubeba cultivation.