The occurrence of the inverse (or negative) electrocaloric effect, where the isothermal application of an electric field leads to an increase in entropy and the removal of the field decreases the entropy of the system under consideration, is discussed and analyzed. Inverse electrocaloric effects have been reported to occur in several cases, for example, at transitions between ferroelectric phases with different polarization directions, in materials with certain polar defect configurations, and in antiferroelectrics. This counterintuitive relationship between entropy and applied field is intriguing and thus of general scientific interest. The combined application of normal and inverse effects has also been suggested as a means to achieve larger temperature differences between hot and cold reservoirs in future cooling devices. A good general understanding and the possibility to engineer inverse caloric effects in terms of temperature spans, required fields, and operating temperatures are thus of fundamental as well as technological importance. Here, the known cases of inverse electrocaloric effects are reviewed, their physical origins are discussed, and the different cases are compared to identify common aspects as well as potential differences. In all cases the inverse electrocaloric effect is related to the presence of competing phases or states that are close in energy and can easily be transformed with the applied field.