To standardize definitions and guide the design, regulation, and policy related to automated transportation, the Society of Automotive Engineers (SAE) has established a taxonomy consisting of six levels of vehicle automation. The SAE taxonomy defines each level based on the capabilities of the automated system. It does not fully consider the infrastructure support required for each level. This can be considered a critical gap in the practice because the existing taxonomy does not account for the fact that the operational design domain (ODD) of any system must describe the specific conditions, including infrastructure, under which the system can function. In this paper, we argue that the ambient road infrastructure plays a critical role in characterizing the capabilities of autonomous vehicles (AVs) including mapping, perception, and motion planning, and therefore, the current taxonomy needs enhancement. To throw more light and stimulate discussion on this issue, this paper reviews, analyzes, and proposes a supplement to the existing SAE levels of automation from a road infrastructure perspective, considering the infrastructure support required for automated driving at each level of automation. Specifically, we focus on Level 4 because it is expected to be the most likely level of automation that will be deployed soon. Through an analysis of driving scenarios and state-of-the-art infrastructure technologies, we propose five sub-levels for Level 4 automated driving systems: Level 4-A (Dedicated Guideway Level), Level 4-B (Expressway Level), Level 4-C (Well-Structured Road Level), Level 4-D (Limited-Structured road Level), and Level 4-E (Disorganized Area Level). These sublevels reflect a progression from highly structured environments with robust infrastructure support to less structured environments with limited or no infrastructure support. The proposed supplement to the SAE taxonomy is expected to benefit both potential AV consumers and manufacturers through defining clear expectations of AV performance in different environments and infrastructure settings. In addition, transportation agencies may gain insights from this research towards their planning regarding future infrastructure improvements needed to support the emerging era of driving automation.