Over the last few years, autonomous vehicles (AVs) have witnessed tremendous worldwide interest. Although AVs have been extensively studied in the literature regarding their benefits, implications, and public acceptance, research on the physical infrastructure requirements for autonomous vehicles is still in the infancy stage. For the road infrastructure, AVs can be very promising; however, AVs might introduce new risks and challenges. This paper investigates the impact of AVs on the physical infrastructure with the objective of revealing the infrastructure changes and challenges in the era of AVs. In AVs, the human factor, which is the major factor that influences the geometric design, will not be a concern anymore so the geometric design requirements can be relaxed. On the other hand, the decrease in the wheel wander, because of the lane-keeping system, and the increase in the lane capacity, because of the elimination of the human factor, will bring an accelerated rutting potential and will quickly deteriorate the pavement condition. Additionally, the existing structural design methods for bridges are not safe to support autonomous truck platoons. For parking lots, AVs have the potential to significantly increase the capacity of parking lots using the blocking strategy. However, the implementation of this parking strategy faces multiple issues such as the inconsistent marking system. Finally, AVs will need new infrastructure facilities such as safe harbor areas.