Apples and apple-based products are among the most popular foods around the world for their delightful flavors and health benefits. However, the commonly found mold, Penicillium expansum invades wounded apples, causing the blue mold decay and ensuing the production of patulin, a mycotoxin that negatively affects human health. Patulin contamination in apple products has been a worldwide problem without a satisfactory solution yet. A comprehensive understanding of the factors and challenges associated with patulin accumulation in apples is essential for finding such a solution. This review will discuss the effects of the pathogenicity of Penicillium species, quality traits of apple cultivars, and environmental conditions on the severity of apple blue mold and patulin contamination. Moreover, beyond the complicated interactions of the three aforementioned factors, patulin control is also challenged by the lack of reliable detection methods in food matrices, as well as unclear degradation mechanisms and limited knowledge about the toxicities of the metabolites resulting from the degradations. As apple-based products are mainly produced with stored apples, pre- and post-harvest strategies are equally important for patulin mitigation. Before storage, disease-resistance breeding, orchard-management, and elicitor(s) application help control the patulin level by improving the storage qualities of apples and lowering fruit rot severity. From storage to processing, patulin mitigation strategies could benefit from the optimization of apple storage conditions, the elimination of rotten apples, and the safe and effective detoxification or biodegradation of patulin.