The most common technique for the numerical implementation of peridynamic theory is the uniform discretization together with constant horizon. However, unlike the nonuniform discretization and varying horizons, it is not a natural and intrinsic component of the adaptive refinement analysis and multiscale modeling. Besides, it encounters discretization difficulty in analyzing irregular structures. Therefore, to analyze problems with nonuniform discretization and varying horizons and solve the resulting problems of ghost forces and spurious wave reflection, the dual-horizon peridynamics based on uniform discretization is extended to the nonuniform discretization based on Voronoi diagrams, for which we call it Voronoi-based peridynamics. We redefine the damage definition as well. Next, an adaptive refinement analysis method based on the proposed Voronoi-based peridynamics and its numerical implementation is introduced. Finally, the traditional bond-based peridynamics and the Voronoi-based peridynamics with or without refinement are used to simulate 4 benchmark problems. The examples of 2-D quasi-static elastic deformation, 2-D wave propagation, 2-D dynamic crack growth, and 3-D simulation of the Kalthoff-Winkler experiment demonstrate the efficiency and effectivity of the proposed Voronoi-based peridynamics. Further, the adaptive refinement analysis can be used to obtain reasonable crack path and crack propagation speed with reduced computational burden.