Adverse pregnancy outcomes and their complications cause increased maternal and neonatal morbidity and mortality and contribute considerably to the global burden of disease. In the last two decades, numerous narrative and systematic reviews have emerged assessing non-essential, potentially harmful, trace element exposure as a potential risk factor. This narrative review summarizes the recent literature covering associations between exposure to cadmium, lead, arsenic, and mercury and pregnancy outcomes and highlights common limitations of existing evidence that may hinder decision-making within public health. Several initial scoping searches informed our review, and we searched PubMed (latest date July 2022) for the literature published within the last five years reporting on cadmium, lead, arsenic, or mercury and pre-eclampsia, preterm birth, or prenatal growth. Pre-eclampsia may be associated with cadmium and strongly associated with lead exposure, and exposure to these metals may increase risk of preterm birth. Many reviews have observed cadmium to be negatively associated with birth weight. Additionally, lead and arsenic exposure may be negatively associated with birth weight, with arsenic exposure also adversely affecting birth length and head circumference. These findings should be interpreted with caution due to the limitations of the reviews summarized in this paper, including high heterogeneity due to different exposure assessment methods, study designs, and timing of sampling. Other common limitations were the low quality of the included studies, differences in confounding variables, the low number of studies, and small sample sizes.