Plant molecular farming (PMF) has been promoted since the 1990s as a rapid, cost-effective and (most of all) safe alternative to the cultivation of bacteria or animal cells for the production of biopharmaceutical proteins. Numerous plant species have been investigated for the production of a broad range of protein-based drug candidates. The inherent safety of these products is frequently highlighted as an advantage of PMF because plant viruses do not replicate in humans and vice versa. However, a more nuanced analysis of this principle is required when considering other pathogens because toxic compounds pose a risk even in the absence of replication. Similarly, it is necessary to assess the risks associated with the host system (e.g., the presence of toxic secondary metabolites) and the production approach (e.g., transient expression based on bacterial infiltration substantially increases the endotoxin load). This review considers the most relevant host systems in terms of their toxicity profile, including the presence of secondary metabolites, and the risks arising from the persistence of these substances after downstream processing and product purification. Similarly, we discuss a range of plant pathogens and disease vectors that can influence product safety, for example, due to the release of toxins. The ability of downstream unit operations to remove contaminants and process-related toxic impurities such as endotoxins is also addressed. This overview of plant-based production, focusing on product safety aspects, provides recommendations that will allow stakeholders to choose the most appropriate strategies for process development.