Regulators increasingly rely on real‐world evidence generated from routine‐care health data to evaluate novel therapies. Particularly, external control arms are increasingly used to supplement and contextualize efficacy and safety claims of single arm clinical trials for rare disease therapies. However, there are a number of methodological issues that may affect the validity of results derived from such comparisons. In this mini‐review, we briefly summarize frequently used approaches and outline some of the most important criticisms and paths forward.