Data for loin and backfat depth, as well as carcass growth of 126,051 three-way crossbred pigs raised between 2015 and 2019, were combined with climate records of air temperature, relative humidity, and temperature–humidity index. Environmental covariates with the largest impact on the studied traits were incorporated in a random regression model that also included genomic information. Genetic control of tolerance to heat stress and the presence of genotype by environment interaction were detected. Its magnitude was more substantial for loin depth and carcass growth, but all the traits studied showed a different impact of heat stress and different magnitude of genotype by environment interaction. For backfat depth, heritability was larger under comfortable conditions (no heat stress), as compared to heat stress conditions. Genetic correlations between extreme values of environmental conditions were lower (∼0.5 to negative) for growth and loin depth. Based on the solutions obtained from the model, sires were ranked on their breeding value for general performance and tolerance to heat stress. Antagonism between overall performance and tolerance to heat stress was moderate. Still, the models tested can provide valuable information to identify genetic material that is resilient and can perform equally when environmental conditions change. Overall, the results obtained from this study suggest the existence of genotype by environment interaction for carcass traits, as a possible genetic contributor to heat tolerance in swine.