Writing in the early third century AD, Julius Africanus claimed to have built a library “in the Pantheon” in Rome, the exact location of which remains elusive. In considering the competing possibilities for the site of the library, this paper argues that the building we commonly refer to as the Pantheon does not correspond to the ancient understanding of what the Pantheum was. The case is made that it was not a single building, but instead comprised a larger complex, of which the still-standing structure was only one part. This interpretation allows for a number of details associated with the Pantheon to be rethought within a wider context and alternative proposals advanced regarding the forecourt in front of porch, the “arch” in the centre of this space, the location of the now lost caryatids and bronze columns, the little understood Severan restoration, and the meanings of the much-discussed inscriptions on the façade.