Within the brachytherapy community, many phantoms are constructed in-house, and less commercial development is observed as compared to the field of external beam. Computational or virtual phantom design has seen considerable growth; however, physical phantoms are beneficial for brachytherapy, in which quality is dependent on physical processes, such as accuracy of source placement. Focusing on the design of physical phantoms, this review paper presents a summary of brachytherapy specific phantoms in published journal articles over the last twenty years (January 1, 2000-December 31, 2019). The papers were analyzed and tabulated by their primary clinical purpose, which was deduced from their associated publications. A substantial body of work has been published on phantom designs from the brachytherapy community, but a standardized method of reporting technical aspects of the phantoms is lacking. In-house phantom development demonstrates an increasing interest in magnetic resonance (MR) tissue mimicking materials, which is not yet reflected in commercial phantoms available for brachytherapy. The evaluation of phantom design provides insight into the way, in which brachytherapy practice has changed over time, and demonstrates the customised and broad nature of treatments offered.