Boulders form from a variety of geological processes, which their size, shape, and orientation may help us better understand. Furthermore, they represent potential hazards to spacecraft landing that need to be characterized. However, mapping individual boulders across vast areas is extremely labor‐intensive, often limiting the extent over which they are characterized and the statistical robustness of obtained boulder morphometrics. To automate boulder characterization, we use an instance segmentation neural network, Mask R‐CNN, to detect and outline boulders in high‐resolution satellite and aerial images. Our neural network, BoulderNet, was trained from a data set of >33,000 boulders in >750 image tiles from Earth, the Moon, and Mars. BoulderNet not only correctly detects the majority of boulders in images but also identifies the outline of boulders with high fidelity, achieving average precision and recall values of 72% and 64% relative to manually digitized boulders from the test data set, when only detections with intersection‐over‐union ratios >50% are considered valid. These values are similar to those obtained from human mappers. On Earth, equivalent boulder diameters, aspect ratios, and orientations extracted from predictions were benchmarked against ground measurements and yield values within ±15%, ±0.20, and ±20° of their ground‐truth values, respectively. BoulderNet achieves better boulder detection and characterization performance relative to existing methods, providing a versatile open‐source tool to characterize entire boulder fields on planetary surfaces.