The infratentorial regions are vulnerable to develop brain metastases (BMs). However, the associations between the infratentorial localization of BMs and clinical characteristics remained unclear. We retrospectively studied 1102 patients with 4365 BM lesions. Voxel-wise mapping of MRI was applied to construct the tumor frequency heatmaps after normalization and segmentation. The analysis of differential involvement (ADIFFI) was further used to obtain statistically significant clusters. Kaplan-Meier method and Cox regression were used to analyze the prognosis. The parietal, insular and left occipital lobes, and cerebellum were vulnerable to BMs with high relative metastatic risks. Infratentorial areas were site-specifically affected by the lung, breast, and colorectal cancer BMs, but inversely avoided by melanoma BMs. Significant infratentorial clusters were associated with young age, male sex, lung neuroendocrine and squamous cell carcinomas, high expression of Ki-67 of primaries and BMs, and patients with poorer prognosis. Inferior OS was observed in patients with ≥3 BMs and those who received whole-brain radiotherapy alone. Infratentorial involvement of BMs was an independent risk factor of poor prognosis for patients who received surgery (p = 0.023, hazard ratio = 1.473, 95% confidence interval = 1.055–2.058). The current study may add valuable clinical recognition of BMs and provide references for BMs diagnosis, treatment evaluation, and prognostic prediction.