Deep inspiration breath-hold (DIBH) is widely used to reduce the cardiac dose in left-sided breast cancer radiotherapy. This study aimed to develop a deep learning chest X-ray model for cardiac dose prediction to select patients with a potentially high risk of cardiac irradiation and need for DIBH radiotherapy. We used 103 pairs of anteroposterior and lateral chest X-ray data of left-sided breast cancer patients (training cohort: n = 59, validation cohort: n = 19, test cohort: n = 25). All patients underwent breast-conserving surgery followed by DIBH radiotherapy: the treatment plan consisted of three-dimensional, two opposing tangential radiation fields. The prescription dose of the planning target volume was 42.56 Gy in 16 fractions. A convolutional neural network-based regression model was developed to predict the mean heart dose (∆MHD) reduction between free-breathing (MHDFB) and DIBH. The model performance is evaluated as a binary classifier by setting the cutoff value of ∆MHD > 1 Gy. The patient characteristics were as follows: the median (IQR) age was 52 (47–61) years, MHDFB was 1.75 (1.14–2.47) Gy, and ∆MHD was 1.00 (0.52–1.64) Gy. The classification performance of the developed model showed a sensitivity of 85.7%, specificity of 90.9%, a positive predictive value of 92.3%, a negative predictive value of 83.3%, and a diagnostic accuracy of 88.0%. The AUC value of the ROC curve was 0.864. The proposed model could predict ∆MHD in breast radiotherapy, suggesting the potential of a classifier in which patients are more desirable for DIBH.