Point defects are a universal feature of crystalline materials. Their identification is often addressed by combining experimental measurements with theoretical models. The standard approach of simulating defects is, however, prone to missing the ground state atomic configurations associated with energy-lowering reconstructions from the idealised crystallographic environment. Missed ground states compromise the accuracy of calculated properties. To address this issue, we report an approach to efficiently navigate the defect configurational landscape using targeted bond distortions and rattling. Application of our workflow to a range of materials (CdTe, GaAs, Sb2S3, Sb2Se3, CeO2, In2O3, ZnO, anatase-TiO2) reveals symmetry breaking in each host crystal that is not found via conventional local minimisation techniques. The point defect distortions are classified by the associated physico-chemical factors. We demonstrate the