The human visual pathway starts from the retina, passes through the retinogeniculate visual pathway, the optic radiation, and finally connects to the primary visual cortex. Diffusion MRI tractography is the only technology that can noninvasively reconstruct the visual pathway. However, complete and accurate visual pathway reconstruction is challenging because of the skull base environment and complex fiber geometries. Specifically, the optic nerve within the complex skull base environment can cause abnormal diffusion signals. The crossing and fanning fibers at the optic chiasm, and a sharp turn of Meyer's loop at the optic radiation, contribute to complex fiber geometries of the visual pathway. A fiber trajectory distribution (FTD) function‐based tractography method of our previous work and several high sensitivity tractography methods can reveal these complex fiber geometries, but are accompanied by false‐positive fibers. Thus, the related studies of the visual pathway mostly applied the expert region of interest selection strategy. However, interobserver variability is an issue in reconstructing an accurate visual pathway. In this paper, we propose a unified global tractography framework to automatically reconstruct the visual pathway. We first extend the FTD function to a high‐order streamline differential equation for global trajectory estimation. At the global level, the tractography process is simplified as the estimation of global trajectory distribution coefficients by minimizing the cost between trajectory distribution and the selected directions under the prior guidance by introducing the tractography template as anatomic priors. Furthermore, we use a deep learning‐based method and tractography template prior information to automatically generate the mask for tractography. The experimental results demonstrate that our proposed method can successfully reconstruct the visual pathway with high accuracy.