Chestnuts are multipurpose trees significant for the economy and wildlife. These trees are currently found around the globe, demonstrating their genetic adaptation to different environmental conditions. Several biotic and abiotic stresses have challenged these species, contributing to the decline of European chestnut production and the functional extinction of the American chestnut. Several efforts started over the last century to understand the cellular, molecular, and genetic interactions behind all chestnut biotic and abiotic interactions. Most efforts have been toward breeding for the primary diseases, chestnut blight and ink disease caused by the pathogens, Cryphonectria parasitica and Phytophthora cinnamomi, respectively. In Europe and North America, researchers have been using the Asian chestnut species, which co-evolved with the pathogens, to introgress resistance genes into the susceptible species. Breeding woody trees has several limitations which can be mostly related to the long life cycles of these species and the big genome landscapes. Consequently, it takes decades to improve traits of interest, such as resistance to pathogens. Currently, the availability of genome sequences and next-generation sequencing techniques may provide new tools to help overcome most of the problems tree breeding is still facing. This review summarizes European and American chestnut’s main biotic stresses and discusses breeding and biotechnological efforts developed over the last decades, having ink disease and chestnut blight as the main focus. Climate change is a rising concern, and in this context, the adaptation of chestnuts to adverse environmental conditions is of extreme importance for chestnut production. Therefore, we also discuss the abiotic challenges on European chestnuts, where the response to abiotic stress at the genetic and molecular level has been explored.