As clinical studies with artificial pancreas systems for automated blood glucose control in patients with type 1 diabetes move to unsupervised real-life settings, product development will be a focus of companies over the coming years. Directions or requirements regarding safety in the design of an artificial pancreas are, however, lacking. This review aims to provide an overview and discussion of safety and design requirements of the artificial pancreas. We performed a structured literature search based on three search components—type 1 diabetes, artificial pancreas, and safety or design—and extended the discussion with our own experiences in developing artificial pancreas systems. The main hazards of the artificial pancreas are over- and under-dosing of insulin and, in case of a bi-hormonal system, of glucagon or other hormones. For each component of an artificial pancreas and for the complete system we identified safety issues related to these hazards and proposed control measures. Prerequisites that enable the control algorithms to provide safe closed-loop control are accurate and reliable input of glucose values, assured hormone delivery and an efficient user interface. In addition, the system configuration has important implications for safety, as close cooperation and data exchange between the different components is essential.