The electrocaloric (EC) effect is the adiabatic temperature change of a material in a varying external electrical field which is promising for novel cooling devices. While the fundamental understanding of the caloric response of defect-free materials is well progressed, there are important gaps in knowledge on reversibility and time-stability of the response. Particularly, it is not settled how the time-dependent elements of microstructure which are always present in real materials act on the field- induced temperature changes. Ab initio based molecular dynamics simulations allow us to isolate and understand the effects arising from domain walls and defect dipoles and study their interplay. We show that domain walls in cycling fields do not improve the response in either one ferroelectric phase or at the ferroelectric phase transition but may result in irreversible heat losses. The presence of defect dipoles may be beneficial for the EC response for proper field protocols, and interestingly this benefit is not too sensitive to the defect configuration.