COVID-19 has dramatically struck each section of our society: health, economy, employment, and mobility. This work presents a data-driven characterization of the impact of COVID-19 pandemic on public and private mobility in a mid-size city in Spain (Fuenlabrada). Our analysis used real data collected from the public transport smart card system and a Bluetooth traffic monitoring network, from February to September 2020, thus covering relevant phases of the pandemic. Our results show that, at the peak of the pandemic, public and private mobility dramatically decreased to 95% and 86% of their pre-COVID-19 values, after which the latter experienced a faster recovery. In addition, our analysis of daily patterns evidenced a clear change in the behavior of users towards mobility during the different phases of the pandemic. Based on these findings, we developed short-term predictors of future public transport demand to provide operators and mobility managers with accurate information to optimize their service and avoid crowded areas. Our prediction model achieved a high performance for pre- and post-state-of-alarm phases. Consequently, this work contributes to enlarging the knowledge about the impact of pandemic on mobility, providing a deep analysis about how it affected each transport mode in a mid-size city.