Asphalt pavement, which is mainly made up of the asphalt mixture, exhibits complicated mechanical behaviors under the combined effects of moving vehicle loads and external service environments. Multi-scale numerical simulation can well characterize behaviors of asphalt materials and asphalt pavement, and the essential research progress is systematically summarized from an entire view. This paper reviews extensive research works concerning aspects of the design, characterization, and prediction of performance for asphalt materials and asphalt pavement based on multi-scale numerical simulation. Firstly, full-scale performance modeling on asphalt pavement is discussed from aspects of structural dynamic response, structural and material evaluation, and wheel–pavement interaction. The correlation between asphalt material properties and pavement performance is also analyzed, and so is the hydroplaning phenomenon. Macro- and mesoscale simulations on the mechanical property characterization of the asphalt mixture and its components are then investigated, while virtual proportion design for the asphalt mixture is introduced. Features of two-dimensional and three-dimensional microscale modeling on the asphalt mixture are summarized, followed by molecular dynamics simulation on asphalt binders, aggregates, and their interface, while nanoscale behavior modeling on asphalt binders is presented. Finally, aspects that need more attention concerning this study’s topic are discussed, and several suggestions for future investigations are also presented.