Intravascular Ultrasound (IVUS) is a medical imaging modality widely used for the detection and treatment of coronary heart disease. The detection of vascular structures is extremely important for accurate treatment procedures. Manual detection of lumen and calcification is very time-consuming and requires technical experience. Ultrasound imaging suffers from the generation of artifacts which obstructs the clear delineation among structures. Considering, the need, to provide special attention to crucial areas, convolutional block attention modules (CBAM) is integrated into an encoder-decoder-based U-Net architecture along with Atrous Spatial Pyramid Pooling (ASPP) to detect vessel components: lumen, calcification and shadow borders. The attention modules prove effective in dealing with areas of special attention by assigning additional weights to crucial channels and preserving spatial features. The IVUS data of 12 patients undergoing the treatment is taken for this study. The novelty of the model design is such that it is able to detect the lumen area in the presence/absence of calcification and bifurcation artifacts too. Also, the model efficiently detects the calcification area even in case of severely complex lesions with shadows behind them. The main contribution of the work is that IVUS images of varying degrees of calcification till 360° are also considered in this work, which is usually neglected in previous studies. The experimental results of 1097 IVUS images of 12 patients resulted in meanIoU (0.7894 ± 0.011), Dice Coefficient (0.8763 ± 0.070), precision (0.8768 ± 0.069) and recall (0.8774 ± 0.071) of the proposed model CADNet which show the model’s effectiveness relative to other state-of-the art methods.