Ginkgo biloba L. has significant health benefits and considerable economic value, but harvesting the fruit is highly labor-intensive. Mechanical vibration harvesting has been shown effective in harvesting various fruit types. In the study of vibration harvesting, the research on the vibration characteristics of fruit trees focuses on the natural frequency (resonance frequency), model, and damping coefficient, which are the main factors affecting the vibration characteristics of trees. But field harvesting experiments have shown that the tree structure may have an impact on the vibration characteristics of the fruit tree and the efficiency of mechanical harvesting. In addition, the research on the damping coefficient of fruit trees is mainly low-frequency damping, and the relevant results cannot be applied to the actual vibration harvesting frequency range. Applying a natural frequency with low damping coefficient to excite a tree can reduce additional energy dissipation. This study explored the influence of ginkgo crown structure on the vibration characteristics and the law of damping changes with frequency. After counting 273 ginkgo trees, two typical ginkgo crown structures, monopodial branching and sympodial branching, were selected to be analyzed for vibration spectrum and damping coefficient. The vibration models for different crown-shaped ginkgo trees were simulated to analyze the vibration state at different frequencies. For sympodial branching ginkgo trees, the consistency of natural frequencies at different branches was better than monopodial branching ginkgo trees. The finite element model analysis shows that monopodial branching ginkgo trees have mainly partial vibrations at different branches when vibrating at high frequencies. The high-frequency vibrations in sympodial branching reflect the better overall vibration of the canopy. The damping coefficients for the two crown types decreased with the increase in frequency. The monopodial branching damping coefficient was 0.0148–0.0298, and the sympodial branching damping coefficient was slightly smaller at 0.0139–0.0248. Based on the test results, the sympodial branching ginkgo tree has better vibration characteristics. The results indicate that controlling the crown structure of fruit trees to be sympodial branching by pruning may help improve the overall vibration characteristics of fruit trees.