Background Automatic coronary angiography (CAG) assessment may help in faster screening and diagnosis of patients. Current CNN-based vessel-segmentation suffers from sampling imbalance, candidate frame selection, and overfitting; few have shown adequate performance for CAG stenosis classification. We aimed to provide an end-to-end workflow that may solve these problems. Methods A deep learning-based end-to-end workflow was employed as follows: 1) Candidate frame selection from CAG videograms with CNN+LSTM network, 2) Stenosis classification with Inception-v3 using 2 or 3 categories (<25%, >25%, and/or total occlusion) with and without redundancy training, and 3) Stenosis localization with two methods of class activation map (CAM) and anchor-based feature pyramid network (FPN). Overall 13744 frames from 230 studies were used for the stenosis classification training and 4-fold cross-validation for image-, artery-, and per-patient-level. For the stenosis localization training and 4-fold cross-validation, 690 images with >25% stenosis were used. Results Our model achieved an accuracy of 0.85, sensitivity of 0.96, and AUC of 0.86 in per-patient level stenosis classification. Redundancy training was effective to improve classification performance. Stenosis position localization was adequate with better quantitative results in anchor-based FPN model, achieving global-sensitivity for LCA and RCA of 0.68 and 0.70 with mean square error (MSE) values of 39.3 and 37.6 pixels respectively, in the 520 × 520 pixel image. Conclusion A fully-automatic end-to-end deep learning-based workflow that eliminates the vessel extraction and segmentation step was feasible in coronary artery stenosis classification and localization on CAG images.