Transition metal oxides that exhibit a metal-to-insulator transition (MIT) as a function of oxygen vacancy concentration are promising systems to realize energy-efficient platforms for neuromorphic computing. However, the current lack of understanding of the microscopic mechanism driving the MIT hinders the realization of effective and stable devices. Here we investigate defective cobaltites and we unravel the structural, electronic, and magnetic changes responsible for the MIT when oxygen vacancies are introduced in the material. We show that, contrary to accepted views, cooperative structural distortions instead of local bonding changes are responsible for the MIT, and we describe the subtle interdependence of structural and magnetic transitions. Finally, we present a model, based on first principles, to predict the required electric bias to drive the transition, showing good agreement with available measurements and providing a paradigm to establish design rules for low-energy cost devices.