Background and Objective. Three-dimensional (3D) ultrasound (US) is needed to provide sonographers with a more intuitive panoramic view of the complex anatomical structure, especially in the musculoskeletal system. In actual scanning, sonographers may perform fast scanning using a 1D array probe at random angles to gain rapid feedback, which will lead to a large US image interval and missing regions in the reconstructed volume. Methods. A 3D residual network (3D-ResNet) modified by a 3D global residual branch (3D-GRB) and two 3D local residual branches (3D-LRBs) was proposed to retain the details and reconstruct high-quality 3D US volumes with high efficiency using only sparse 2D US images in this study. The feasibility and performance of the proposed algorithm were evaluated on the ex and in vivo sets. Results. High-quality 3D US volumes in the fingers, radial and ulnar bones, and metacarpophalangeal joints were obtained by the 3D-ResNet, respectively. Their axial, coronal, and sagittal slices exhibited rich texture and speckle details. Compared with kernel regression, voxel nearest-neighborhood, squared distance weighted methods, and 3D convolution neural network in the ablation study, the mean peak-signal-to-noise ratio and mean structure similarity of the 3D-ResNet were up to 28.53 ± 1.29 dB and 0.98 ± 0.01, respectively, and the corresponding mean absolute error dropped to 0.023 ± 0.003 with a better resolution gain of 1.22 ± 0.19 and shorter reconstruction time. Significance. These results illustrated that the proposed algorithm could rapidly reconstruct high-quality 3D US volumes in the musculoskeletal system in cases of a large amount of data loss. It suggested that the proposed algorithm had the potential to provide rapid feedback and precise analysis of stereoscopic details in the complex and meticulous musculoskeletal system scanning, while less limiting the scanning speed and pose variations of the 1D array probe.