Scanning Thermal Microscopy is emerging as a powerful Atomic Force Microscope based platform for mapping dynamic temperature distributions on the nanoscale. To date, however, spatial imaging of temperature changes in electrocaloric materials using this technique has been very limited. We build on the prior works of Kar-Narayan et al. (Appl. Phys. Lett. 102, 032903, 2013) and Shan et al. (Nano Energy, 67, 104203, 2020) to show that Scanning Thermal Microscopy can be used to spatially map electrocaloric temperature changes on microscopic length scales, here demonstrated in a commercially obtained multilayer ceramic capacitor. In our approach, the electrocaloric response is measured at discrete locations with point-to-point separation as small as 125 nm, allowing for reconstruction of spatial maps of heating and cooling, as well as their temporal evolution. This technique offers a means to investigate electrocaloric responses at sub-micron length scales, which cannot easily be accessed by the more commonly used infrared thermal imaging approaches.