Remanent magnetization in geological samples may record the past intensity and direction of planetary magnetic fields. Traditionally, this magnetization is analyzed through measurements of the net magnetic moment of bulk millimeter to centimeter sized samples. However, geological samples are often mineralogically and texturally heterogeneous at submillimeter scales, with only a fraction of the ferromagnetic grains carrying the remanent magnetization of interest. Therefore, characterizing this magnetization in such cases requires a technique capable of imaging magnetic fields at fine spatial scales and with high sensitivity. To address this challenge, we developed a new instrument, based on nitrogen‐vacancy centers in diamond, which enables direct imaging of magnetic fields due to both remanent and induced magnetization, as well as optical imaging, of room‐temperature geological samples with spatial resolution approaching the optical diffraction limit. We describe the operating principles of this device, which we call the quantum diamond microscope (QDM), and report its optimized image‐area‐normalized magnetic field sensitivity (20 µT⋅µm/Hz1/2), spatial resolution (5 µm), and field of view (4 mm), as well as trade‐offs between these parameters. We also perform an absolute magnetic field calibration for the device in different modes of operation, including three‐axis (vector) and single‐axis (projective) magnetic field imaging. Finally, we use the QDM to obtain magnetic images of several terrestrial and meteoritic rock samples, demonstrating its ability to resolve spatially distinct populations of ferromagnetic carriers.