The aim of this article is to identify the factors affecting user perception of transfers in order to reduce the penalty associated with transfers. Policy makers can thus concentrate on the most relevant factors to achieve a more effective investment of public funds. We carried out an extensive review of the literature and identified a total of 45 factors. A qualitative study was then conducted in two Spanish cities, Vitoria and Madrid, consisting of various focus groups stratified by age. The results indicate that a combination of factors influence the perception of transfers. Time spent during transfers, both walking and waiting, is the main penalty associated with transfers. However, the provision of real-time passenger information helps reduce the perceived waiting time. Differences were found between cities and age groups. Two new factors were identified, mainly in Madrid: mental effort and activity disruption. We, therefore, advise policy makers to concentrate their efforts on providing real-time information to passengers, improving the frequency and reliability of services, and designing transfer areas to ensure continuity between transport modes and between information and communication systems.