The branches of fruit trees provide support for the growth of leaves, buds, flowers, fruits, and other organs. The number and length of branches guarantee the normal growth, flowering, and fruiting of fruit trees and are thus important indicators of tree growth and yield. However, due to their low height and the high number of branches, the precise management of fruit trees lacks a theoretical basis and data support. In this paper, we introduce a method for extracting topological and structural information on fruit tree branches based on LiDAR (Light Detection and Ranging) point clouds and proved its feasibility for the study of fruit tree branches. The results show that based on Terrestrial Laser Scanning (TLS), the relative errors of branch length and number are 7.43% and 12% for first-order branches, and 16.75% and 9.67% for second-order branches. The accuracy of total branch information can reach 15.34% and 2.89%. We also evaluated the potential of backpack-LiDAR by comparing field measurements and quantitative structural models (QSMs) evaluations of 10 sample trees. This comparison shows that in addition to the first-order branch information, the information about other orders of branches is underestimated to varying degrees. The root means square error (RMSE) of the length and number of the first-order branches were 3.91 and 1.30 m, and the relative root means square error (NRMSE) was 14.62% and 11.96%, respectively. Our work represents the first automated classification of fruit tree branches, which can be used in support of precise fruit tree pruning, quantitative forecast of yield, evaluation of fruit tree growth, and the modern management of orchards.