The Royal Botanic Garden of Madrid was founded in 1755 in the outskirts of the city, in the place known as Migas Calientes. Joseph Quer, its first Director, previously maintained three other additional gardens where he cultivated the plants that were later used in the final plantation of Migas Calientes. All those gardens disappeared and until now its location has been considered doubtful. In this article, based on historic documentation and cartography, we provide the precise location of the four aforementioned gardens.