Deep learning is expanding and continues to evolve its capabilities toward more accuracy, speed, and cost-effectiveness. The core ingredients for getting its promising results are appropriate data, sufficient computational resources, and best use of a particular algorithm. The application of these algorithms in medical image analysis tasks has achieved outstanding results compared to classical machine learning approaches. Localizing the area-of-interest is a challenging task that has vital importance in computer aided diagnosis. Generally, radiologists interpret the radiographs based on their knowledge and experience. However, sometimes, they can overlook or misinterpret the findings due to various reasons, e.g., workload or judgmental error. This leads to the need for specialized AI tools that assist radiologists in highlighting abnormalities if exist. To develop a deep learning driven localizer, certain alternatives are available within architectures, datasets, performance metrics, and approaches. Informed decision for selection within the given alternative can lead to batter outcome within lesser resources. This paper lists the required components along-with explainable AI for developing an abnormality localizer for X-ray images in detail. Moreover, strong-supervised vs weak-supervised approaches have been majorly discussed in the light of limited annotated data availability. Likewise, other correlated challenges have been presented along-with recommendations based on a relevant literature review and similar studies. This review is helpful in streamlining the development of an AI based localizer for X-ray images while extendable for other radiological reports.