The interest in and establishment of urban gardens and food forests has been growing in recent years. Food production in urban environments has its challenges, however, particularly regarding the safety of edible crops grown in environments that contain potential contaminants. While some studies exist on urban agriculture food safety for annual crops, the literature is limited on relative risks in urban food forests that include fruit-and nut-producing trees and shrubs. This review provides an overview of the potential capacity of woody species (trees and shrubs) to accumulate polycyclic aromatic hydrocarbons (PAHs), heavy metals (HMs), and metalloids considering the safety of food production in the urban environment. The general trends found in a review of existing literature indicate (a) a lower risk for woody species to accumulate HMs and PAHs compared with vegetables, and (b) less accumulation in the fruit of woody species compared with other plant parts in those same species. However, these trends are not always consistent, and the accumulation of contaminants depends on variety of factors such as the concentration of a given element in the environment, type of contaminant, type of species, and species variety. This study highlights the critical need for more research on the safety of growing edible fruits and nuts on trees and shrubs in urban environments that may contain contaminants.