We introduce off-axis chromatic scanning confocal electron microscopy, a technique for fast mapping of inelastically scattered electrons in a scanning transmission electron microscope without a spectrometer. The off-axis confocal mode enables the inelastically scattered electrons to be chromatically dispersed both parallel and perpendicular to the optic axis. This enables electrons with different energy losses to be separated and detected in the image plane, enabling efficient energy filtering in a confocal mode with an integrating detector. We describe the experimental configuration and demonstrate the method with nanoscale core-loss chemical mapping of silver (M4,5) in an aluminium-silver alloy and atomic scale imaging of the low intensity core-loss La (M4,5@840 eV) signal in LaB6. Scan rates up to 2 orders of magnitude faster than conventional methods were used, enabling a corresponding reduction in radiation dose and increase in the field of view. If coupled with the enhanced depth and lateral resolution of the incoherent confocal configuration, this offers an approach for nanoscale three-dimensional chemical mapping.