During the last decade, valence change memory (VCM) has been extensively studied due to its promising features, such as a high endurance and fast switching times. The information is stored in a high resistive state (logcial '0', HRS) and a low resistive state (logcial '1', LRS). It can also be operated in two different writing schemes, namely a unipolar switching mode (LRS and HRS are written at the same voltage polarity) and a bipolar switching mode (LRS and HRS are written at opposite voltage polarities). VCM, however, still suffers from a large variability during writing operations and also faults occur, which are not yet fully understood and, therefore, require a better understanding of the underlying fault mechanisms. In this study, a new intrinsic failure mechanism is identified, which prohibits RESET times (transition from LRS to HRS) faster than 400 ps and possibly also limits the endurance. We demonstrate this RESET speed limitation by measuring the RESET kinetics of two valence change memory devices (namely Pt/TaOx/Ta and Pt/ZrOx/Ta) in the time regime from 50 ns to 50 ps, corresponding to the fastest writing time reported for VCM. Faster RESET times were achieved by increasing the applied pulse voltage. Above a voltage threshold it was, however, no longer possible to reset both types of devices. Instead a unipolar SET (transition from HRS to LRS) event occurred, preventing faster RESET times. The occurrence of the unipolar SET is attributed to an oxygen exchange at the interface to the Pt electrode, which can be suppressed by introducing an oxygen blocking layer at this interface, which also allowed for 50 ps fast RESET times.