Nanoscale devices -such as all-optical modulators and electro-optical transducers -can be implemented in heterostructures that integrate plasmonic nanostructures with functional active materials.Here we demonstrate all-optical control of a nanoscale memory effect in such a heterostructure by coupling the localized surface plasmon resonance (LSPR) of gold nanodisk arrays to a phase-changing material (PCM), vanadium dioxide (VO 2 ). By latching the VO 2 in a distinct correlated metallic state during the insulator-to-metal transition (IMT), while concurrently exciting the hybrid nanostructure with one or more ultraviolet optical pulses, the entire phase space of this correlated state can be accessed optically to modulate the plasmon response. We find that the LSPR modulation depends strongly but linearly on the initial latched state, suggesting that the memory effect encoded in the plasmon resonance wavelength is linked to the strongly correlated electron states of the VO 2 . The continuous, linear variation of the electronic and optical properties of these model heterostructures opens the way to multiple design strategies for hybrid devices with novel optoelectronic functionalities, which can be controlled by an applied electric or optical field, strain, injected charge or temperature.