We here report a study of the correlation between the evolution of resistive switching and the oxygen vacancy configuration in La₀.₅Ca₀.₅MnO₃ (LCMO) based memristive devices. By taking advantage of LCMO located at a phase boundary of the metal-to-insulator transition, we observe the development of the high resistive states, depending upon not only the electrical pulse magnitude but also the switching cycles. We discuss the experimental results by an oxygen migration model that involves both single isolated and clustered oxygen vacancies, which are later verified using aberration-corrected scanning transmission electron microscopy.